Jump to content


Can I break the creating checkpoint process


  • Please log in to reply
6 replies to this topic

#1 ytt1515234

ytt1515234

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 18 July 2020 - 07:08 AM

Hi,

I'm trying to create a checkpoint which the size of journal file is too large (around 300GB), it looks like will take a long time. But if our team want to use perforce emergency, can I break the checkpoint process? Or if someone could tell me how long will it takes when the journal file is large. Thanks a lot.

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1149 posts

Posted 18 July 2020 - 07:12 AM

The time to create a checkpoint isn't necessarily a function of the size of the journal file; most of what's in the journal won't end up in the checkpoint (e.g. db.have entries that have since been overwritten).

Since a checkpoint is primarily a read operation, it should be safe to kill it without damaging the db, but the checkpoint it's writing will be incomplete and is not suitable for recovery, and the journal will not be rotated.  The next time you take a checkpoint it'll start over from the beginning.

#3 ytt1515234

ytt1515234

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 18 July 2020 - 08:06 AM

View PostSambwise, on 18 July 2020 - 07:12 AM, said:

The time to create a checkpoint isn't necessarily a function of the size of the journal file; most of what's in the journal won't end up in the checkpoint (e.g. db.have entries that have since been overwritten).

Since a checkpoint is primarily a read operation, it should be safe to kill it without damaging the db, but the checkpoint it's writing will be incomplete and is not suitable for recovery, and the journal will not be rotated.  The next time you take a checkpoint it'll start over from the beginning.


thank you! so it's safety to break the process, that's a good news!
from your answer, can I say that size of db files will affect the time spent on creating checkpoint? The db files save records from journal, and the checkpoint read records from db files, is this the workflow?

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1149 posts

Posted 18 July 2020 - 02:57 PM

View Postytt1515234, on 18 July 2020 - 08:06 AM, said:



thank you! so it's safety to break the process, that's a good news!
from your answer, can I say that size of db files will affect the time spent on creating checkpoint? The db files save records from journal, and the checkpoint read records from db files, is this the workflow?

Yes, the size of the db files is a much better gauge for how long the checkpoint will take!

If you want to try to gauge how close the checkpoint is to completion, you can peek at the checkpoint file and see which tables have been dumped so far.  The tables will be checkpointed in the locking order (documented here: https://www.perforce...current/schema/) and the number of rows in each table will roughly correspond to their size on disk.  Typically db.have, db.rev, and db.integed are the largest ones.

#5 ytt1515234

ytt1515234

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 20 July 2020 - 03:44 AM

 Sambwise, on 18 July 2020 - 02:57 PM, said:

Yes, the size of the db files is a much better gauge for how long the checkpoint will take!

If you want to try to gauge how close the checkpoint is to completion, you can peek at the checkpoint file and see which tables have been dumped so far.  The tables will be checkpointed in the locking order (documented here: https://www.perforce...current/schema/) and the number of rows in each table will roughly correspond to their size on disk.  Typically db.have, db.rev, and db.integed are the largest ones.

thanks a lot, you answer is very helpful!

#6 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 224 posts
  • LocationSan Diego, CA

Posted 21 July 2020 - 04:12 PM

I'd also consider keeping an offline copy of your db so you can dump checkpoints off of that without taking the server down. Generally speaking, your db is only going to grow over time and at some point taking it down to run a checkpoint will become impractical.

It might be overkill and a deep dive into process you'll never need, but parts of the SDP are useful even if you don't run it yourself. It might be a fun (seriously!) exercise to read about it, learn what it does, and implement the parts (offline db/checkpoints, journal rotation, db compress/swap) that suit your needs. Lots of good stuff in there:

https://swarm.worksh...e-software-sdp/
-Matt Janulewicz
Currently unemployed, looking for work in Boise, ID!

#7 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 204 posts
  • LocationAustin. Texas. Y'all.

Posted 21 July 2020 - 09:28 PM

I second Matt's suggestion. We are up to an 800GB checkpoint, which would have our master down for a couple of hours every night. Instead it's only down when we rotate the checkpoints, which takes a second or two once a month (we don't do that weekly). Make sure when you upgrade servers to get plenty of extra disk for things like checkpoints, journals, and logs to grow, as well as the metadata an depot files.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users