Jump to content


Speeding up checkpoints

checkpoint backup admin p4d

  • Please log in to reply
6 replies to this topic

#1 G Barthelemy

G Barthelemy

    Advanced Member

  • Members
  • PipPipPip
  • 65 posts
  • LocationUnited Kingdom

Posted 03 May 2014 - 06:00 PM

With a 500GB database, checkpoints are taking a long time to complete even on over specified hardware. The production server is replicated, so this is not affecting users as I take checkpoints from the replica. I compress (-Z), and noticed that it's using just 1 of many CPU cores, and that is getting maxed out while we have plenty of spare capacity idling on the host. There is plenty of memory and spare I/O bandwidth too, and it would seem that the contention is on the CPU. On the other hand, I noticed that uncompressed checkpoints were only marginally faster in my case, but didn't pay attention whether they were CPU bound too. Anyway, we need to compress them for some reason.

It should be much more efficient if this could be parallelised. I heard during a conference a few years ago of a well known multinational corporation specialising in Internet-related services and products, whose creative Perforce admins came up with a method for dumping checkpoints fast. My understanding is that it went roughly like this: duplicate their humongous database several times (using block level snapshots on a filer I guess, or perhaps by splitting mirrors or some clever file system trick, as it seems to me that just copying into multiple folders would nullify any time gains) and by deleting db files from each volume or using some other clever arrangement, have the capability of quickly isolating groups of database tables on multiple folders/volumes (I'd imagine db.label table to be on its own, and so would db.have in my case :rolleyes: ). Then run a checkpoint per volume, all in parallel, and then stitch the partial checkpoints together to obtain a proper checkpoint file, using some script not necessarily approved by Perforce :P. Somewhat daunting...

But then I noticed that p4d -jd can be passed a table name to produce a dump of just that table. So could a new type of checkpoint be envisaged, whereby each table is dumped separately which would allow the use of multiple threads in parallel and thus a significant reduction in time ? My guess is that the largest table would dictate the duration of the overall "checkpoint", but in my case that could be a third of the current duration. There are around 50 tables, give or take, so this database checkpoint method would produce 50 files. Each would restore its own table.

Would that work ? Has anyone considered or actually tried this, or a similar method in production ?

Cheers,
Guillaume

#2 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 05 May 2014 - 01:45 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


> Would that work ? Has anyone considered or actually tried this, or a similar
method in production ?

I have a vague recollection that Google (Dan Bloch) talked about this method several years ago at one of the conference.
Maybe you can search the conference materials for it.
It was more than three years ago.

What I remember is that they would create a temporary directory, copy there one or a few db.* files and create a checkpoint using the root as that directory.
Split up db files across those temp directories and create several checkpoints - one for each - in parallel.

--Michael Mirman
508-647-7555
MathWorks, Inc.
3 Apple Hill Dr, Natick, MA 01760

-----Original Message-----
From: perforce-user-bounces@perforce.com [mailto:perforce-user-bounces@perforce.com] On Behalf Of G Barthelemy
Sent: Saturday, May 03, 2014 2:05 PM
To: perforce-user@perforce.com
Subject: [p4] Speeding up checkpoints


Posted on behalf of forum user 'G Barthelemy'.

With a 500GB database, checkpoints are taking a long time to complete even on
over specified hardware. The production server is replicated, so this is not
affecting users as I take checkpoints from the replica. I compress (-Z), and
noticed that it's using just 1 of many CPU cores, and that is getting maxed
out while we have plenty of spare capacity idling on the host. There is plenty
of memory and spare I/O bandwidth too, and it would seem that the contention is
on the CPU. On the other hand, I noticed that uncompressed checkpoints were only
marginally faster in my case, but didn't pay attention whether they were CPU
bound too. Anyway, we need to compress them for some reason.

It should be much more efficient if this could be parallelised. I heard during a
conference a few years ago of a well known multinational corporation
specialising in Internet-related services and products, whose creative Perforce
admins came up with a method for dumping checkpoints fast. My understanding is
that it went roughly like this: duplicate their humongous database several times
(using block level snapshots on a filer I guess, or perhaps by splitting mirrors
or some clever file system trick, as it seems to me that just copying into
multiple folders would nullify any time gains) and by deleting db files from
each volume or using some other clever arrangement, have the capability of
quickly isolating groups of database tables on multiple folders/volumes (I'd
imagine db.label table to be on its own, and so would db.have in my case
:rolleyes:  ). Then run a checkpoint per volume, all in parallel, and then
stitch the partial checkpoints together to obtain a proper checkpoint file,
using some script not necessarily approved by Perforce  :P . Somewhat
daunting...

But then I noticed that  p4d -jd  can be passed a table name to produce a dump
of just that table. So could a new type of checkpoint be envisaged, whereby each
table is dumped separately which would allow the use of multiple threads in
parallel and thus a significant reduction in time ? My guess is that the largest
table would dictate the duration of the overall "checkpoint", but in
my case that could be a third of the current duration. There are around 50
tables, give or take, so this database checkpoint method would produce 50 files.
Each would restore its own table.

Would that work ? Has anyone considered or actually tried this, or a similar
method in production ?

Cheers,
Guillaume



--
Please click here to see the post in its original format:
  http://forums.perfor...-up-checkpoints
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user


#3 Robert Cowham

Robert Cowham

    Advanced Member

  • PCP
  • 271 posts
  • LocationLondon, UK

Posted 06 May 2014 - 08:31 AM

It is worth adding your name to a SIR requesting this feature via support. Maybe even p4ideax too.

Meanwhile, it is possible to get the gzipping done via a pipe to pigz utility (http://zlib.net/pigz/) - there is also a Windows build available - which does the zipping in parallel.

For one customer on Windows, a text checkpoint operation took about 30% of the time of the compressed option - so this depends on hardware/OS etc.

The usual approach is to do offline checkpoint to avoid issues with the time required. If you then restore from the offline checkpoint say once a week and replace your production db.* files (on the same filesystem to avoid copying), you also get fragmentation free database which improves performance and reduces size.

See option 3:

http://answers.perfo...ine-Checkpoints

Let me know if you want more detailed steps.

Regards
Robert
Co-Author of "Learning Perforce SCM", PACKT Publishing, 25 September 2013, ISBN 9781849687645

"It's wonderful to see a new book about Perforce, especially one written by Robert Cowham and Neal Firth. No one can teach Perforce better than these seasoned subject matter experts"
  • Laura Wingerd, author of Practical Perforce, former VP of Product Technology at Perforce

#4 Robert Cowham

Robert Cowham

    Advanced Member

  • PCP
  • 271 posts
  • LocationLondon, UK

Posted 06 May 2014 - 08:54 AM

p.s. The Google method is outlined:

http://static.google...chive/39983.pdf

Quote

Finally, we optimize our checkpoints and restores by processing each database file individually.
This is straightforward to do by creating a subdirectories and populating each with a link to a
database file. This means, since we have lots of CPUs and lots of bandwidth, that we can
checkpoint and restore multiple database files in parallel, so our checkpoints and restores take
five and a half hours instead of the seventeen they would if we just used p4d –jd and –jr.

Co-Author of "Learning Perforce SCM", PACKT Publishing, 25 September 2013, ISBN 9781849687645

"It's wonderful to see a new book about Perforce, especially one written by Robert Cowham and Neal Firth. No one can teach Perforce better than these seasoned subject matter experts"
  • Laura Wingerd, author of Practical Perforce, former VP of Product Technology at Perforce

#5 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 06 May 2014 - 09:35 AM

Originally posted to the perforce-user mailing list by: Michael Mirman


Quote

  This is straightforward to do by creating a subdirectories and populating each with a link to a
  database file.

This piece didn’t work for me very well.
What I saw was: when p4d creates a checkpoint, it locks the database (naturally).
I *think* flock() follows the symlink and locks the actual file, but I haven't really investigated.
At least what I saw was: when multiple db* files were linked and parallel checkpointing started, all normal operations to the server hung until the needed db* files were unlocked.

Then, we switched to do checkpointing on a replica. It has been working perfectly for years.
Google's problem was that the non-parallel checkpoint would take some enormous amount of time.
For us, it's just a few hours. Having one replica to be offline for 2-3 hours at night is not an issue.
If/when the time grows significantly, we'll need to switch to parallel checkpointing, too.

--Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

-----Original Message-----
From: perforce-user-bounces@perforce.com [mailto:perforce-user-bounces@perforce.com] On Behalf Of Robert Cowham
Sent: Tuesday, May 06, 2014 4:55 AM
To: perforce-user@perforce.com
Subject: Re: [p4] Speeding up checkpoints


Posted on behalf of forum user 'Robert Cowham'.

p.s. The Google method is outlined:

http://static.google...chive/39983.pdf



Quote

Quote

  Finally, we optimize our checkpoints and restores by processing each database file individually.
  This is straightforward to do by creating a subdirectories and populating each with a link to a
  database file. This means, since we have lots of CPUs and lots of bandwidth, that we can
  checkpoint and restore multiple database files in parallel, so our checkpoints and restores take
  five and a half hours instead of the seventeen they would if we just used p4d –jd and –jr.




--
Please click here to see the post in its original format:
  http://forums.perfor...-up-checkpoints
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#6 G Barthelemy

G Barthelemy

    Advanced Member

  • Members
  • PipPipPip
  • 65 posts
  • LocationUnited Kingdom

Posted 07 May 2014 - 03:58 PM

Thanks for the pointers. I did not realise that those papers were online.

I have a problem with the concept of copying database files to several directories to checkpoint them individually in parallel: simply copying the files can take a long time, especially for the big ones (db.have, db.label). Add to this the time to checkpoint the largest file, and in the end you may not gain that much overall.
OK, perhaps the issue I have is that the read only replica where I take checkpoints from has a disk I/O that should be improved, but at the moment the concern is that the time it takes is overtaking our RPO. And Michael, even if your replica checkpoint takes 3 hours to generate, what if you lose the master storage 2 hours and 59 minutes after the checkpoint started on the replica ? You would not have any journal to complement your checkpoint, or would you ?

But anyway it would seem to me that dispatching files to various directories is no longer required, and "p4d -jd <checkpoint_file> <table>" would be the thing to go for. I guess I'll query with support about this.

The alternative would be to checkpoint from a file system snapshot on the replica. That concept is fine when you say it quick but in reality it's a little bit more convoluted than that, as when in a replicated topology, it's usually the replica that schedules the checkpoint and the rotation of the journal on the master launches it. So at face value a checkpoint obtained from a file system snapshot of a replica would be fine in itself, but there would not be any journal that could be replayed into a database restored from that checkpoint. Or maybe make of the database captured in the snapshot an actual replica, with a short life-span.

Guillaume

#7 Mark H

Mark H

    Member

  • Members
  • PipPip
  • 10 posts
  • LocationLiverpool, UK

Posted 20 May 2014 - 02:16 PM

I highly recommend moving to a Linux environment and adding as much RAM as the server can physically hold.
(This is assuming you're hosting the service on W*ndows)

In one of our game studios over a week end I moved the Perforce service from a W*ndows server to Linux. The new server has  around 400GB of RAM and locally attatched SSD's to host the database.
The checkpoint was reduced from approximately 3 hours to around 38 minutes.





Also tagged with one or more of these keywords: checkpoint, backup, admin, p4d

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users