Jump to content


Can metadata backups become inconsistent with versioned files?

backup

  • Please log in to reply
1 reply to this topic

#1 davidair

davidair

    Member

  • Members
  • PipPip
  • 21 posts

Posted 16 July 2020 - 03:40 AM

Hello folks,

I am curious if metadata backups can become inconsistent with versioned files backup. Our process is as follows (on a nightly basis):

- Take a checkpoint and back up the checkpoint file
- rsync the versioned files (to Google Cloud Storage)

Normally this approach works well because any changes that happen after the checkpoint should be additive (i.e. new files get created in the physical depot locations) so there is no harm backing them up.
However, there appear to be two situations when this is not the case:

1. A shelved file gets deleted (possibly but not necessarily during a changelist submission)
2. An existing versioned file gets updated (it looks like Perforce will, under some circumstances, not create a new revision, but instead update the file, such as /opt/depot/somefile,d/1.1234.gz)

If this happens during the small time window after we've backed up the checkpoint but not yet the versioned files, we will end up in an inconsistent state, with files being reported as MISSING or BAD by p4 verify.
In the case of Swarm code reviews, this will cause Swarm to fail loading the diffs (throwing a 500 internally).

I am expecting these cases to eventually become consistent (i.e. with the next checkpoint) but I was wondering if this was a known issue/limitation and if there was a good strategy around it.

#2 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 214 posts
  • LocationAustin. Texas. Y'all.

Posted 20 July 2020 - 08:42 PM

It is definitely a potential problem. Our solution is a backup server (read-only replica). At a low-traffic time, we run a script that stops the replica, generates a checkpoint, backs everything up (using parallel threads), and restarts the replica. We run test restores and verifies. We have 800GB worth of metadata and 19TB versioned files (last time I checked, about 80M files). The total process takes a bit over 7.5 hours currently, but only the two largest depots take that long. The rest is about 5.5 hours.

I forgot; we aren't currently do checkpoints on that server due to lack of space on the smallish SSDs. We will when we upgrade hardware.





Also tagged with one or more of these keywords: backup

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users