Jump to content


Helix in the cloud


  • Please log in to reply
2 replies to this topic

#1 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 128 posts

Posted 13 August 2019 - 05:58 PM

Has anyone moved a large-ish Perforce installation (let's say, millions of files and/or terabytes of files, and/or at least 500GB or metadata) to the cloud?

If so, which cloud platform? How along ago? What has your experience been?

This is purely curiosity; sooner or later, it's bound to come up. When it does, I'd like to know if and how others are coping.

If you'd rather not say anything publicly, please feel free to message me directly (though I doubt I am the only person interested in the answers).

Thanks,
Miles

#2 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 176 posts
  • LocationSan Francisco, CA

Posted 18 August 2019 - 09:54 AM

About a year ago we put one edge server in the cloud (AWS) to facilitate dynamic(ly sized) builds on the cheap. As opposed to building out a local VM cluster that was big enough to handle our biggest surges in builds but would sit idle most of the time.

Our builds that go into the cloud are triggered in such a way that they spin up dozens of VMs at once (all Linux for now) and hammer the edge server like crazy for some number of minutes then everything dies down. The main growing pain we had was figuring out which AWS machine type was best for our needs. We had to keep inching it up until it could take on 150-200 syncs at once.

We initially didn't 'seed' the edge server with any library files, we left it empty and let the projects syncing from it dictate how much storage we needed. Including the database we only have about 2 TB of data in there at the moment. (db is a tad under 500 GB.)

The reason we've set it up in such a strict way is because with AWS you get charged for every byte you copy _out_, but nothing for what you put in. So an edge server that only services other VM's in the same cloud is super cost-effective. Linux hosts are so cheap, free in a lot of cases, that the setup we have lends itself very nicely to doing large, dynamically-sized builds that are self-contained in AWS.

We're currently kicking around the idea of having a warm standby in AWS that could aid us in disaster recovery, especially as a commit server if our data center goes down for a lengthy amount of time. It would be easy enough to put an edge server in there that's intended to be used by humans outside of the cloud, but it would cost us an arm and a leg, and maybe part of a second arm, given the amount of data we download from our main edge server in a day. We haven't done a cost analysis on that but there have been times when someone accidentally connected a build server outside AWS to that edge server and blew through our entire monthly AWS budget in a day or two.

So anyway, that's a long-winded way of saying that most of the considerations on our end are financial. Technology wise, the thing has been performing really well for about a year and we haven't had any real problems with it.
-Matt Janulewicz
Staff SCM Engineer, Perforce Administrator
Dolby Laboratories, Inc.
1275 Market St.
San Francisco, CA 94103, USA
majanu@dolby.com

#3 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 128 posts

Posted 19 August 2019 - 08:33 PM

Thanks, Matt. As we would use this only in the cloud, we will firewall it to avoid such problems if we go cloud-diving.
We already have DR sites for our hotbeds of Helix traffic.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users