Perforce Performance with Lots Binary Files
Posted 13 September 2011 - 02:27 PM
I'm part of a team that has been tasked to improve the utilization and the performance of our Perforce Servers at our company. We are still at the early stage of our work (i.e. planning, brainstorming, metrics analysis, etc). Nevertheless, we did some initial research online and we came across a series of whitepapers and presentations from people like Geoff Mendal, Dan Bloch, Michael Shields, Tom Brazil, etc. They were all very informative read but, at this moment, we do have one question.
Our company uses much much more binary files than text files. Considering that, for text file, perforce stores diff whereas for binary file it stores the whole, I am uncertain if it affects any recommendations (generally speaking) in these papers? If it does, is there anything that our administrators should keep in mind during the study?
Thanks in advance,
Posted 13 September 2011 - 03:15 PM
performance of our Perforce Servers at our company. We are still at the
stage of our work (i.e. planning, brainstorming, metrics analysis, etc).
Nevertheless, we did some initial research online and we came across a
whitepapers and presentations from people like Geoff Mendal, Dan Bloch,
Shields, Tom Brazil, etc. They were all very informative read but, at this
moment, we do have one question.
In very broad terms, a p4d instance that serves a large number of
small text files needs a lot of RAM for the active metadata and a fast
disk subsystem to hold the database images. For a p4d instance that
serves a small number of large files the performance bottlenecks may
show up in the disks that hold the file data and in network bandwidth.
If many large files are submitted at once you'll find that you have
to provide space for temporary copies of the files.
If some of your team's developers are in remote offices with limited
network connectivity, give the Perforce proxy (p4p) a try. It works
well for large, often-read files in bandwidth-limited use cases.
Chuck Karish firstname.lastname@example.org (650) 283-7864
perforce-user mailing list - email@example.com
Posted 13 September 2011 - 04:55 PM
1. Fast disks for the repository - high RPM, lots of RAM disk cache, with a RAID 10 array with lots of spindles - anything to improve disk speed.
2. Fast local network.
3. P4 proxies to all remote sites. Remote access was the only place where performance was an issue. We had to watch out how much data we let get pushed down the pipe and when we let the p4 syncs for new files happen at the remote site (we did a cache filling script that made sure that the proxy cache got updates in small chucks spread out over the day).
The biggest problem we had was not performance but disk space on the repository drives - binary files consume lots of disk space. We made extensive use of the +S flag to limit the number of revisions stored.
Posted 13 September 2011 - 07:19 PM
Some good info on building for scale can be found in these whitepapers:
Scaling Perforce Servers and Storage for Film Assets
Scaling Source Control for the Next Generation of Game Development
And these whitepapers detail our system architecture:
Templar System Architecture
Linkatron (filesystem access to p4 repository)
Posted 14 September 2011 - 01:33 PM
We'll start reading through all these materials.
Posted 15 November 2011 - 04:42 AM
Where possible try to keep you binary assets small in size (*10 if you have remote sites using a proxy).
Be careful +Sn can have a negative implications if you do any integrating. It saves space but also force non-lazy copies so an integrate from a source with +Sn essentially has to duplicate all the files at the destination on disk.
Posted 30 October 2014 - 07:40 PM
Posted 30 October 2014 - 08:03 PM
Posted 09 June 2017 - 08:51 PM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users