Jump to content


Edges, Backups and Shelf Promotion

backup shelves

  • Please log in to reply
5 replies to this topic

#1 davidair

davidair

    Member

  • Members
  • PipPip
  • 21 posts

Posted 25 March 2020 - 03:21 PM

What are the best practices around backing up edge replicas?

We have a setup where a master commit server is replicated to two edges and everyone works on the edge servers. In terms of backing up the metadata, we believe we can use coordinated checkpoints to back up both the master commit and edge servers.

The big question is backing up archived files. Normally, backing up only the main commit server's archived files should be enough, but it doesn't cover edge shelves.

We were thinking of forcing all shelves to go be promoted by setting the dm.shelve.promote server configurable to a value of 1 (docs). However, the article goes on saying "Generally, it is a bad idea to enable automatic promotion because it causes a lot of unnecessary file transfers for shelved files that are not meant to be shared."

If this is a bad idea, how should we back up the edges? Is there a way of figuring out which archived files on the edge server come from local shelves, as opposed to the regular depot files? It feels like it would be a very bad idea to end up with three full copies of archived files being backed up (just from the storage cost perspective).

#2 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 230 posts
  • LocationSan Diego, CA

Posted 26 March 2020 - 01:58 AM

At the last place I worked, we had edges all over the world with the commit server living in San Francisco. We enabled/required global shelves for just this reason and it wasn't that big of a deal. We didn't have _too_ much latency between most of our locations and San Francisco, but people were used to whatever speed they were getting on submits anyway, so the slower offices didn't notice/complain about 'slow shelving'. It is what it is and it worked out fine.

You should look into offline databases (second, non-production version of your databases, living on each server) and back those up, as well as the checkpoints/journals. The SDP is a good thing to look into.

One other note that hopefully you already know is that aside from some db files, edge servers have other unique things that need backing up (namely any //spec/... or //unload/... depots.)

Once you switch over to global shelves, also remember to promote the shelves that are local to the edge servers. It's fun/easy to check this by using P4V to list shelved changes, then make sure they have the little globe icon on there. :)
-Matt Janulewicz
Currently unemployed, looking for work in Boise, ID!

#3 davidair

davidair

    Member

  • Members
  • PipPip
  • 21 posts

Posted 06 April 2020 - 02:28 PM

Thank you Matt!

A couple of follow-up questions:

1. Thanks for flagging the special Spec and Unload depots. Is there a comprehensive "what else I forgot to backup from the edge" list?
2. After a coordinated checkpoint happens on the edge (as a result of ```p4 admin checkpoint```), is it worth it to explicitly backup individual tables (by dumping them first per this article) instead of the complete checkpoint? Does the linked article provide an authoritative list of which tables should be backed up?

#4 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 230 posts
  • LocationSan Diego, CA

Posted 07 April 2020 - 04:58 AM

I'm not sure if there are any other unique depots on edge servers, //unload/ and //spec/ were the only two non-standard ones we had. Maybe archive depots? Not sure.

Dumping and backing up each db file seems excessive in my opinion, but I'm coming off of a job where we had:

1. An offline db on every server, updated once a day after journal rotation.
2. Weekly checkpoints and intervening daily journals on every server.
3. Two standby replicas of the commit server.
4. A backup replica server that had live replicas of every server, plus checkpoints, journals and offline databases for each other server/instance.
5. A second server like #4 offsite.
6. Daily/monthly snapshots (ZFS) on servers 4 and 5.
7. An instance in AWS where snapshots from #5 were sent daily.

So in all, for an edge server, we had a local offline db and checkpoints, two replicas of that, one cloud copy of that, and daily/monthly snapshots of everything. By my count I think that's 4 copies of everything plus snapshots. We deemed that paranoid enough but others might have a different opinion.

We also tended to trust checkpoints more than anything and were okay with a recovery time of a database being 'however long it takes to unwind a checkpoint' (3-4 hours in our case.)
-Matt Janulewicz
Currently unemployed, looking for work in Boise, ID!

#5 davidair

davidair

    Member

  • Members
  • PipPip
  • 21 posts

Posted 07 April 2020 - 02:37 PM

View PostMatt Janulewicz, on 26 March 2020 - 01:58 AM, said:

Once you switch over to global shelves, also remember to promote the shelves that are local to the edge servers. It's fun/easy to check this by using P4V to list shelved changes, then make sure they have the little globe icon on there. :)

Is it possible to promote someone else's shelf? I tried this:

```
p4 -c SOMEONE_ELSES_HOST_CLIENT shelve -p -c 42
```

But got an error:

Client 'SOMEONE_ELSES_HOST_CLIENT' can only be used from host 'SOMEONE_ELSES_HOST'.

How can an admin promote existing local shelves after setting the dm.shelve.promote flag?

#6 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 214 posts
  • LocationAustin. Texas. Y'all.

Posted 07 April 2020 - 03:49 PM

That client is locked to a host. You would need (as that user or as a superuser) to edit the client to remove the Host: restriction.





Also tagged with one or more of these keywords: backup, shelves

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users