For historical reasons, part of our CI process generates several autoreoad labels and tags many files (potentially hundreds of thousands) with each label. Some of these labels are removed very soon after the tag (or labelsync, I forget which). I've run across multiple issues that look related to this. The level of resultant discrepancies (outlined below) varies between master, local replicas, and remote replicas (>120mS pings).
- After running this way for several years, the number of dirs/files living in the unload depot varies widely across the systems (currently ranging from 70,000 to > 90,000 unload files).
- Some of the dirs (ex: /p4/1/depots/unloaded/label/42,d/label:thing1-fred.ckp,d/ ) contain the expected file named 1.0 . Others were empty. Others contained a temp file (I believe it was tmp.nn.nnnn or something simlar). A few contained both a 1.0 and a tmp file!
- Some labels no longer had a directory, much less a file, in the unload depot.
- Some dirs and files in the unload depot no longer had a label associated.
I also don't understand why the directory would be missing when the label is there.
Has anyone else noticed this?
For the record, we're still at 2017.1 . We have a workaround for the 2018 showstopper we hit, but are hopefully going to 2019 soon.
 We're reworking that process.
 Some of the temp files are deleted within a few seconds of creation. On busy servers, I can see the journal entry to delete a label hitting the replica while a large file was still being transferred. This could leave a directory or file behind without a label.