Jump to content


Recovery of Dropbox Perforce Repository

migration recovery backup

  • Please log in to reply
5 replies to this topic

#1 WetSavannaAnimal

WetSavannaAnimal

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 28 February 2017 - 12:01 PM

I am a user of Perforce to keep track of my own software projects.

I keep the P4ROOT on my Dropbox so that, theoretically, all the versioned files and metadata should be backed up always.

I also make checkpoints as described here. There were some minor changes to my repository between the last checkpoint and a crash of my Windows partition after the crash.

I use the simple P4V and P4Admin applications to drive Perforce and tried for the last day to to get the P4Admin application to find the restored repository. Here is how I proceeded:

1. I am running P4V and P4Admin under Windows 10, 64 Bit OS, and my versioned files and metadata live in my Dropbox;
2. On rebuilding Windows and reinstalling Perforce Helix and P4V, I downloaded my repository from Dropbox;
3. I then tried to get P4Admin to see the former repository by editing the depot in P4Admin so that its root reflected my former repository's location. No repository was recognized and the depot appeared empty in P4Admin;
4. However, P4Admin does understand that there is something there, as when I tried to delete it I got a warning that "Other Perforce branches/ repos" are under P4ROOT;
5. With a WIndows command line, I then switched to the former repository's root director, stopped the Perforce server with "p4 admin stop" and then manually started a server with root at the repository's location with "p4d -r ."
6. With P4V I then tried to connect to the server that I had manually started. The connexion was made successfully, but, again, the depot came up showing empty, and my former files were not there.
7. I then made a backup copy of the repository, deleted all the db.*, *.lbr and server.locks directory and rebuilt the metadata from my latest checkpoint as described here. Then I repeated steps 3 through 6 above with the restored metadata and  still the depots show up as empty in the P4V and P4Admin tools.

Is there anything I am doing wrong here? How does one go about telling a newly installed P4Admin to point to a former, restored repository?

Many thanks in advance

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 979 posts

Posted 28 February 2017 - 03:36 PM

Quote

3. I then tried to get P4Admin to see the former repository by editing the depot in P4Admin so that its root reflected my former repository's location. No repository was recognized and the depot appeared empty in P4Admin;
4. However, P4Admin does understand that there is something there, as when I tried to delete it I got a warning that "Other Perforce branches/ repos" are under P4ROOT;
5. With a WIndows command line, I then switched to the former repository's root director, stopped the Perforce server with "p4 admin stop" and then manually started a server with root at the repository's location with "p4d -r ."

Your step #5 should have worked, but the fact that you did #3 and #4 first might have messed something up.  Note that when you change the depot's Map (I assume this is what you're talking about since IIRC you can't change P4ROOT from P4Admin) you're changing the location of the archive files relative to the metadata (i.e. P4ROOT) -- if you had the whole depot structure in Dropbox you should not have needed/wanted to do this since everything would still be in the same relative locations and messing with that setting would just make your depot inaccessible.

You should have still been able to see the files after #3 (you'd have just gotten errors trying to sync them), but you said something about deleting in #4 and it's not clear which directory you meant -- if that was the P4ROOT directory then that'd have wiped out your metadata, which WOULD make everything show up empty.

Quote

7. I then made a backup copy of the repository, deleted all the db.*, *.lbr and server.locks directory and rebuilt the metadata from my latest checkpoint as described here. Then I repeated steps 3 through 6 above with the restored metadata and  still the depots show up as empty in the P4V and P4Admin tools.

If you made your backup after having deleted everything from P4ROOT you'd still end up with an empty server...

So let's go back to the start -- are you able to have a working server from anywhere at this point?  I.e. on the original machine?  If you deleted the server files out of your Dropbox you might need to start by rolling back that operation to recover them.  Once you have a working Perforce server, maybe you could give some context to your description by indicating what your server Root ("p4 info") and depot Map ("p4 depot -o") fields point to in the working configuration.

#3 WetSavannaAnimal

WetSavannaAnimal

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 01 March 2017 - 10:45 AM

Dear Sambwise

The original machine crashed; Perforce was working up till then.

I didn't delete anything in step 4: I was simply saying that when I went to delete the depot, I got warnings that showed Perforce was aware of other repositories.

And you are right: I was confusing the depot's map and P4ROOT. The reason I had to do this is that I had to tell P4Admin where the versioned files were. I used p4 info to check where P4ROOT is, and p4 depot -o to verify that the map pointed to the versioned files.

I tried to reconstruct the metadata in P4ROOT from the slightly out of date checkpoint I had using my step number 7 I described yesterday: I wasn't putting the metadata in P4ROOT, so I was hopeful I might have some luck.

However, after all this, I still have an empty repository. I'm guessing that somehow my checkpoint didn't include the right info, although I thought I had done a p4 -q verify on it.

I have been able to reconstruct the latest version of my project from a backup of my workspace, which is the main thing I need, but it seems I have lost my versioning history. This has really shattered my confidence in Perforce: I have used it for over 15 years and loved it, although mostly as a client and only began administrating when I left my former employer to work for myself. For the moment, I am going to keep my project in Visual Studio Team Services until I can work out what the hell I've screwed up with Perforce usage. I need to simulate a crash and make sure I can recover in full before I rely on it again, but that will have to be later when I have more time. I simply thought that backing the metadata up with a checkpoint and the versioned files were all that one needed to get back on track after a crash.

Anyhow, many thanks for your help.

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 979 posts

Posted 01 March 2017 - 11:14 PM

"p4 verify" doesn't do anything to your checkpoint -- if you want to double check it I'd just open it up in a text editor and eyeball it (there should be a LOT of ASCII data in there, including the names of all the files you've ever submitted to Perforce).  If your checkpoint is empty or very small, you probably checkpointed the wrong directory.

All of the metadata for the Perforce server is contained in the db.* files (which are what the checkpoint is a backup mechanism for).  If those were in your Dropbox on the old machine they're probably still in Dropbox somewhere as long as you didn't delete them too long ago -- if your checkpoint is bad I'd try to figure out ASAP where the original db.* files went so you can get them back.  Empty db.* files are something like 16K in size; if your P4ROOT directory contains a bunch of empty db.* files it means they were deleted (or were never there) and the server initialized P4ROOT with new ones on startup (or when you recovered from an empty checkpoint).

To get a functioning server, you need to start a p4d instance (or Perforce service if you're on Windows) with P4ROOT set to the directory where your db.* files live.  If your P4ROOT was in Dropbox, and the depot Maps were all subdirectories of P4ROOT rather than absolute locations elsewhere, all you should have needed to do is:

net stop Perforce
p4 set -S Perforce P4ROOT=C:\Users\me\Dropbox\p4root
net start Perforce


The only thing you should have needed to tell P4Admin is the address of the server (e.g. localhost:1666).  All the other information is in the db.* files, which are in the P4ROOT directory, which was in your Dropbox -- right?

#5 WetSavannaAnimal

WetSavannaAnimal

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 02 March 2017 - 12:34 AM

Dear Sambwise

This is all making a great deal more sense now. My original server was not in Dropbox (as you rightly point out, I was confusing mappings with P4ROOT), only the versioned files were and my checkpoints were defective - somehow I must have been checkpointing the wrong directory.

Not to worry: I have what I need software wise, and I've only been administering my own Perforce for a few months. But it would be great to have confidence that I know what I doing if this happens again. So here are my findings: I'd appreciate if you would confirm that I have my understanding right.

I have now created a new Perforce server - as you say, there doesn't seem to be an equivalent to "p4 set -S Perforce P4ROOT=C:\Users\me\Dropbox\p4root" in P4Admin (I have never really learnt the Perforce line commands until now, which seems to be the way "real" Perforce people do things). This is in "C:\Program Files\Perforce\Server", confirmed with the "p4 info" command.

I have the versioned files in my Dropbox, which I have set with a MAPPING in P4Admin. The depot is called "HypatiaSimulator" and the command "p4 depot -o HypatiaSimulator" shows that the HypatiaSimulator depot is mapped to "C:/Users/RodVance/Dropbox/SoftwareRepository/.."

This morning I took a checkpoint of the new repository from the running server after I had checked in all my files again with "p4d -jc" and the checkpoint file looked totally different from what I had ever seen. This tells me I was somehow screwing up in the checkpointing process: the new checkpoint is much longer and contains very sensible text information that seems to describe the depot mapping (where the versioned files are), workspace information and the adding of and checking in of files to the repository (see the end of my post).

So now I would like to know how I would go about restoring. From all of this research, armed with the backed up checkpoint and the versioned files in my dropbox, I would do the following to recover from a disk crash:

1. Install Helix server and P4V
2. Go to P4ROOT and put the latest backed up checkpoint there
3. On a command line run "p4 admin stop"
4. Delete all the server's "db.*",  "*.lbr" files and "server.locks" from a command line
5. Unpack the checkpoint with "p4d -r <checkpoint>" to create the new database
6. Restart Windows so that the Perforce server is again running in its root with the new files.

If I understand correctly, the Perforce server will now know about all of the versioned files (as long as they continue to live in the same place in my Dropbox) AND the workspaces because all that information was in the checkpoint and now presumably in the newly rebuilt database.

Is that all correct? Many thanks for your thoughts.

One last question: It would seem sensible to keep the server off the Dropbox and store checkpoints instead, no? I say this because Dropbox "fights" some applications as they try to lock files for various things because, of course, Dropbox needs to lock and ensure its transactions are atomic too (I find I can't digitally sign documents from MS-Word when they are on Dropbox for this reason). So there would be a significant risk of a fight between the Perforce server and Dropbox, which are trying to do the same things in their own ways to the server files. In any case, if I want to use P4Admin, then I can't store the server on the Dropbox anyway.

____________________________________________________

Notes On The Checkpoint

In particular, I can see in the checkpoint text a line that would seem to record the mapping:

"@pv@ 1 @db.depot@ @HypatiaSimulator@ 0 @@ @C:/Users/RodVance/Dropbox/SoftwareRepository/...@"

and then I see thousands of lines that seem very sensible and seem to be recording the assignment of Workspace (which I called "HypatiaMaser")  files to checksums, version info and then confirmation that they are received, such as

"@pv@ 3 @db.have@ @//HypatiaMaster/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ 1 0 1463727147"

(here HypatiaMaster is a workspace I have created for myself in P4V) presumably showing that the files in my workspace have been added to the repository and then thousands of corresponding lines like:

"@pv@ 9 @db.rev@ @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ 1 0 0 1 1488374325 1463727147 2DC57DE99F2092B6CECEF38C7A59C7DF 20130 0 0 @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ @1.1@ 0"

presumably assigning a checksum and version to each file and then

"@pv@ 0 @db.revcx@ 1 @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ 1 0"

@pv@ 9 @db.revhx@ @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ 1 0 0 1 1488374325 1463727147 2DC57DE99F2092B6CECEF38C7A59C7DF 20130 0 0 @//HypatiaSimulator/Hypatia/SimulationEngine/HypatiaMath/Quaternion.hpp@ @1.1@ 0"

persumably confirming the checking in to the repository.

#6 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 979 posts

Posted 02 March 2017 - 01:08 AM

Yep, that backup/recovery description sounds about right.

Putting the db files in Dropbox seems like it should work in principle as long as you're not trying to run the server in more than one place at once, but locking contention is a good point -- with a small server it'd probably work all right, but the Dropbox client would be reading and uploading the files each time they changed, so as the db files got bigger you'd start seeing some slowdown as Perforce waited on Dropbox to finish uploading the last batch of updates in real time.  The archive files are more spread out and less frequently accessed/modified so contention isn't as big of an issue there.

Checkpointing to the Dropbox folder seems like a good way to go.  You can have the checkpoints get created there directly by running something like:

p4 admin checkpoint C:\Users\me\Dropbox\checkpoints






Also tagged with one or more of these keywords: migration, recovery, backup

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users