Jump to content


Perforce Performance with Lots Binary Files


  • Please log in to reply
9 replies to this topic

#1 Jandler

Jandler

    Member

  • Members
  • PipPip
  • 10 posts

Posted 13 September 2011 - 02:27 PM

Hello everyone,



I'm part of a team that has been tasked to improve the utilization and the performance of our Perforce Servers at our company. We are still at the early stage of our work (i.e. planning, brainstorming, metrics analysis, etc). Nevertheless, we did some initial research online and we came across a series of whitepapers and presentations from people like Geoff Mendal, Dan Bloch, Michael Shields, Tom Brazil, etc. They were all very informative read but, at this moment, we do have one question.



Our company uses much much more binary files than text files. Considering that, for text file, perforce stores diff whereas for binary file it stores the whole, I am uncertain if it affects any recommendations (generally speaking) in these papers? If it does, is there anything that our administrators should keep in mind during the study?



Thanks in advance,



Jandler

#2 Chuck Karish

Chuck Karish

    Member

  • Members
  • PipPip
  • 22 posts

Posted 13 September 2011 - 03:15 PM

On Tue, Sep 13, 2011 at 7:30 AM, Jandler
<perforce-user-forum@forums.perforce.com> wrote:

Quote

I'm part of a team that has been tasked to improve the utilization and the
performance of our Perforce Servers at our company. We are still at the
early
stage of our work (i.e. planning, brainstorming, metrics analysis, etc).
Nevertheless, we did some initial research online and we came across a
series of
whitepapers and presentations from people like Geoff Mendal, Dan Bloch,
Michael
Shields, Tom Brazil, etc. They were all very informative read but, at this
moment, we do have one question.

In very broad terms, a p4d instance that serves a large number of
small text files needs a lot of RAM for the active metadata and a fast
disk subsystem to hold the database images.  For a p4d instance that
serves a small number of large files the performance bottlenecks may
show up in the disks that hold the file data and in network bandwidth.
  If many large files are submitted at once you'll find that you have
to provide space for temporary copies of the files.

If some of your team's developers are in remote offices with limited
network connectivity, give the Perforce proxy (p4p) a try.  It works
well for large, often-read files in bandwidth-limited use cases.

--
Chuck Karish   chuck.karish@gmail.com   (650) 283-7864

_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user

#3 Andrew reynolds

Andrew reynolds

    Member

  • Members
  • PipPip
  • 24 posts
  • LocationSan Jose, CA

Posted 13 September 2011 - 04:55 PM

I did p4 admin at a site that had lots of binary assets checked in.  P4d worked fine and over all we didn't have a real performance problem.  Some of the things we did to keep it that way were:

1.  Fast disks for the repository - high RPM, lots of RAM disk cache, with a RAID 10 array with lots of spindles - anything to improve disk speed.
2.  Fast local network.
3.  P4 proxies to all remote sites.  Remote access was the only place where performance was an issue.  We had to watch out how much data we let get pushed down the pipe and when we let the p4 syncs for new files happen at the remote site (we did a cache filling script that made sure that the proxy cache got updates in small chucks spread out over the day).

The biggest problem we had was not performance but disk space on the repository drives - binary files consume lots of disk space.  We made extensive use of the +S flag to limit the number of revisions stored.
--Andrew

#4 Mike Sundy

Mike Sundy

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 13 September 2011 - 07:19 PM

Hi.  I've done p4 administration for Electronic Arts and Pixar, both of which store large amounts of binary data in Perforce.  I agree with Andrew that a large part of tuning for performance for binary assets is focusing on the repository storage technology.  We recently upgraded one of our 7.5 TB depots to faster storage technology.  Perforce was handing the load fine, but the back-end storage was withering under heavy renderfarm load, which slowed p4 access to files.

Some good info on building for scale can be found in these whitepapers:
Scaling Perforce Servers and Storage for Film Assets
Scaling Source Control for the Next Generation of Game Development

And these whitepapers detail our system architecture:
Templar System Architecture
Linkatron (filesystem access to p4 repository)

-Mike

#5 Jandler

Jandler

    Member

  • Members
  • PipPip
  • 10 posts

Posted 14 September 2011 - 01:33 PM

Great!

We'll start reading through all these materials.

Thanks everyone.

#6 Gareth

Gareth

    Member

  • Members
  • PipPip
  • 26 posts

Posted 15 November 2011 - 04:42 AM

Set the no compress option for all files where appropriate (i.e. don't compress already compressed data formats like jpg, mpeg etc)

Where possible try to keep you binary assets small in size (*10 if you have remote sites using a proxy).

Be careful +Sn can have a negative implications if you do any integrating. It saves space but also force non-lazy copies so an integrate from a source with +Sn essentially has to duplicate all the files at the destination on disk.

#7 roadkills_r_us

roadkills_r_us

    Member

  • Members
  • PipPip
  • 13 posts

Posted 30 October 2014 - 07:40 PM

Do we know whether Pixar is going to release a version of this?

#8 P4Matt

P4Matt

    Advanced Member

  • Staff Moderators
  • 1383 posts

Posted 30 October 2014 - 08:03 PM

Pixar announced in June that generously they will be open sourcing Linkatron as well as hopefully some other helpful tools. Linkatron is currently being cleaned up for public release and approved by legal; you know much lawyers love source code. =) Keep an eye on the Perforce blog for news. I will write myself a note to post here when it is live as well.

#9 roadkills_r_us

roadkills_r_us

    Member

  • Members
  • PipPip
  • 13 posts

Posted 08 June 2017 - 08:46 PM

What happened with this?

#10 p4rfong

p4rfong

    Advanced Member

  • Staff Moderators
  • 142 posts

Posted 09 June 2017 - 08:51 PM

We have made a lot of performance improvements since the last post.  Consider a forwarding replica seen in http://answers.perfo...es/KB/1260.  If the forwarding replica in the remote location keeps up with the master, the forwarding replica will have the same metadata and versioned files as the master.  Forwarding replicas process read-only commands and send read-write requests to the master server.  Users will retrieve binary files from the nearby replica (instead of the master) thereby not only improving performance to the user but also offloading the master server.  In addition, the forwarding replica can serve as a disaster recovery server and an offline backup server.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users