Jump to content


Perforce Performance with Lots Binary Files


  • Please log in to reply
16 replies to this topic

#1 Jandler

Jandler

    Member

  • Members
  • PipPip
  • 10 posts

Posted 13 September 2011 - 02:27 PM

Hello everyone,



I'm part of a team that has been tasked to improve the utilization and the performance of our Perforce Servers at our company. We are still at the early stage of our work (i.e. planning, brainstorming, metrics analysis, etc). Nevertheless, we did some initial research online and we came across a series of whitepapers and presentations from people like Geoff Mendal, Dan Bloch, Michael Shields, Tom Brazil, etc. They were all very informative read but, at this moment, we do have one question.



Our company uses much much more binary files than text files. Considering that, for text file, perforce stores diff whereas for binary file it stores the whole, I am uncertain if it affects any recommendations (generally speaking) in these papers? If it does, is there anything that our administrators should keep in mind during the study?



Thanks in advance,



Jandler

#2 Chuck Karish

Chuck Karish

    Member

  • Members
  • PipPip
  • 22 posts

Posted 13 September 2011 - 03:15 PM

On Tue, Sep 13, 2011 at 7:30 AM, Jandler
<perforce-user-forum@forums.perforce.com> wrote:

Quote

I'm part of a team that has been tasked to improve the utilization and the
performance of our Perforce Servers at our company. We are still at the
early
stage of our work (i.e. planning, brainstorming, metrics analysis, etc).
Nevertheless, we did some initial research online and we came across a
series of
whitepapers and presentations from people like Geoff Mendal, Dan Bloch,
Michael
Shields, Tom Brazil, etc. They were all very informative read but, at this
moment, we do have one question.

In very broad terms, a p4d instance that serves a large number of
small text files needs a lot of RAM for the active metadata and a fast
disk subsystem to hold the database images.  For a p4d instance that
serves a small number of large files the performance bottlenecks may
show up in the disks that hold the file data and in network bandwidth.
  If many large files are submitted at once you'll find that you have
to provide space for temporary copies of the files.

If some of your team's developers are in remote offices with limited
network connectivity, give the Perforce proxy (p4p) a try.  It works
well for large, often-read files in bandwidth-limited use cases.

--
Chuck Karish   chuck.karish@gmail.com   (650) 283-7864

_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user

#3 Andrew reynolds

Andrew reynolds

    Member

  • Members
  • PipPip
  • 24 posts
  • LocationSan Jose, CA

Posted 13 September 2011 - 04:55 PM

I did p4 admin at a site that had lots of binary assets checked in.  P4d worked fine and over all we didn't have a real performance problem.  Some of the things we did to keep it that way were:

1.  Fast disks for the repository - high RPM, lots of RAM disk cache, with a RAID 10 array with lots of spindles - anything to improve disk speed.
2.  Fast local network.
3.  P4 proxies to all remote sites.  Remote access was the only place where performance was an issue.  We had to watch out how much data we let get pushed down the pipe and when we let the p4 syncs for new files happen at the remote site (we did a cache filling script that made sure that the proxy cache got updates in small chucks spread out over the day).

The biggest problem we had was not performance but disk space on the repository drives - binary files consume lots of disk space.  We made extensive use of the +S flag to limit the number of revisions stored.
--Andrew

#4 Mike Sundy

Mike Sundy

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 13 September 2011 - 07:19 PM

Hi.  I've done p4 administration for Electronic Arts and Pixar, both of which store large amounts of binary data in Perforce.  I agree with Andrew that a large part of tuning for performance for binary assets is focusing on the repository storage technology.  We recently upgraded one of our 7.5 TB depots to faster storage technology.  Perforce was handing the load fine, but the back-end storage was withering under heavy renderfarm load, which slowed p4 access to files.

Some good info on building for scale can be found in these whitepapers:
Scaling Perforce Servers and Storage for Film Assets
Scaling Source Control for the Next Generation of Game Development

And these whitepapers detail our system architecture:
Templar System Architecture
Linkatron (filesystem access to p4 repository)

-Mike

#5 Jandler

Jandler

    Member

  • Members
  • PipPip
  • 10 posts

Posted 14 September 2011 - 01:33 PM

Great!

We'll start reading through all these materials.

Thanks everyone.

#6 Gareth

Gareth

    Member

  • Members
  • PipPip
  • 26 posts

Posted 15 November 2011 - 04:42 AM

Set the no compress option for all files where appropriate (i.e. don't compress already compressed data formats like jpg, mpeg etc)

Where possible try to keep you binary assets small in size (*10 if you have remote sites using a proxy).

Be careful +Sn can have a negative implications if you do any integrating. It saves space but also force non-lazy copies so an integrate from a source with +Sn essentially has to duplicate all the files at the destination on disk.

#7 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 43 posts

Posted 30 October 2014 - 07:40 PM

Do we know whether Pixar is going to release a version of this?

#8 P4Matt

P4Matt

    Advanced Member

  • Members
  • PipPipPip
  • 1383 posts

Posted 30 October 2014 - 08:03 PM

Pixar announced in June that generously they will be open sourcing Linkatron as well as hopefully some other helpful tools. Linkatron is currently being cleaned up for public release and approved by legal; you know much lawyers love source code. =) Keep an eye on the Perforce blog for news. I will write myself a note to post here when it is live as well.

#9 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 43 posts

Posted 08 June 2017 - 08:46 PM

What happened with this?

#10 p4rfong

p4rfong

    Advanced Member

  • Staff Moderators
  • 197 posts

Posted 09 June 2017 - 08:51 PM

We have made a lot of performance improvements since the last post.  Consider a forwarding replica seen in http://answers.perfo...es/KB/1260.  If the forwarding replica in the remote location keeps up with the master, the forwarding replica will have the same metadata and versioned files as the master.  Forwarding replicas process read-only commands and send read-write requests to the master server.  Users will retrieve binary files from the nearby replica (instead of the master) thereby not only improving performance to the user but also offloading the master server.  In addition, the forwarding replica can serve as a disaster recovery server and an offline backup server.

#11 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 43 posts

Posted 09 October 2017 - 08:30 PM

I'm sorry; I definitely should have clarified. I meant to ask,

"What happened with Pixar making the Linkatron code available?"

P4Matt hasn't said anything about a drop yet. 8^)

[We already make extensive use of forwarding replicas.]

#12 p4rfong

p4rfong

    Advanced Member

  • Staff Moderators
  • 197 posts

Posted 09 October 2017 - 09:56 PM

Unfortunately, we do not have any further details on Pixar's linkatron code being made public.  I would not expect it to become available.

#13 p4steph

p4steph

    Member

  • Members
  • PipPip
  • 19 posts

Posted 09 October 2017 - 10:35 PM








I have information on this If you're interested ping me at steph@swervelogic.com
Steph




Get Outlook for iOS





On Mon, Oct 9, 2017 at 3:00 PM -0700, "p4rfong" <perforce-user-forum@forums..perforce.com> wrote:










Posted on behalf of forum user 'p4rfong'.

Unfortunately, we do not have any further details on Pixar's linkatron code
being made public.  I would not expect it to become available.



--
Please click here to see the post in its original format:
  http://forums.perfor...ts-binary-files
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user





_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user


#14 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 490 posts

Posted 10 October 2017 - 03:43 AM

View Postp4steph, on 09 October 2017 - 10:35 PM, said:

Get Outlook for iOS

It's a trap!

(Steph's binary files thing is legit, though.)

#15 Miles O'Neal

Miles O'Neal

    Advanced Member

  • Members
  • PipPipPip
  • 43 posts

Posted 10 October 2017 - 04:19 PM

Thanks, moderator. It doesn't terribly surprise me. Between making it public-worthy and getting management and legal signoff, it can be not worth the effort. And that's assuming they would ever sign off.
Yeah,. I'm already waiting to hear from Steph. 8^)

#16 itechnology

itechnology

    Member

  • Members
  • PipPip
  • 20 posts
  • LocationToulouse, France

Posted 28 December 2017 - 02:18 PM

I did some preliminary optimizations for where i work.
As Garry pointed out, try to make sure perforce is not trying to compress already compressed files.

You can do this by typing "p4 typemap" and telling perforce how to treat different filetypes. For example here is the list i came up with after analyzing all the kinds of stuff i discovered that was being stored in perforce ..copy paste the list ..extend it, and/or adapt it to your usage style:

https://www.perforce....modifiers.html

binary+F tell perforce to store the file as a binary, and not to gzip it, but to store 1 file per revision

text //spec/...
binary+F //depot/....ai
binary+F //depot/....aiff
binary+F //depot/....apk
binary+F //depot/....aqt
binary+F //depot/....avi
binary+F //depot/....bdsp
binary+F //depot/....bcmap
binary+F //depot/....bin
binary+F //depot/....bmp
binary+F //depot/....cab
binary+F //depot/....car
binary+F //depot/....chm
binary+F //depot/....cpl
binary+F //depot/....ctx
binary+F //depot/....cur
binary+F //depot/....db
binary+F //depot/....dbmdl
binary+F //depot/....dll
binary+F //depot/....doc
binary+F //depot/....ds_store
binary+F //depot/....eot
binary+F //depot/....exe
binary+F //depot/....flv
binary+F //depot/....frx
binary+F //depot/....gif
binary+F //depot/....grm
binary+F //depot/....ico
binary+F //depot/....icns
binary+F //depot/....img
binary+F //depot/....jar
binary+F //depot/....jks
binary+F //depot/....jpg
binary+F //depot/....jpeg
binary+F //depot/....key
binary+F //depot/....keystore
binary+F //depot/....ldf
binary+F //depot/....lib
binary+F //depot/....nib
binary+F //depot/....m4v
binary+F //depot/....me
binary+F //depot/....mdb
binary+F //depot/....mdf
binary+F //depot/....mmt
binary+F //depot/....mov
binary+F //depot/....mp3
binary+F //depot/....mp4
binary+F //depot/....nupkg
binary+F //depot/....obj
binary+F //depot/....ogg
binary+F //depot/....ogv
binary+F //depot/....otf
binary+F //depot/....p7b
binary+F //depot/....p12
binary+F //depot/....pdb
binary+F //depot/....pdf
binary+F //depot/....pfx
binary+F //depot/....pdn
binary+F //depot/....png
binary+F //depot/....psb
binary+F //depot/....psd
binary+F //depot/....psp
binary+F //depot/....res
binary+F //depot/....rpt
binary+F //depot/....rtf
binary+F //depot/....scc
binary+F //depot/....sdb
binary+F //depot/....sdf
binary+F //depot/....snk
binary+F //depot/....so
binary+F //depot/....sqlite
binary+F //depot/....sqlite3
binary+F //depot/....swiftdoc
binary+F //depot/....swiftmodule
binary+F //depot/....swf
binary+F //depot/....swp
binary+F //depot/....tlb
binary+F //depot/....tif
binary+F //depot/....tiff
binary+F //depot/....ttf
binary+F //depot/....wav
binary+F //depot/....webm
binary+F //depot/....wmv
binary+F //depot/....woff
binary+F //depot/....woff2
binary+F //depot/....xcuserstate
binary+F //depot/....xps
binary+F //depot/....zip
binary+F //depot/....zsh


#17 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 490 posts

Posted 28 December 2017 - 06:27 PM

FWIW, Perforce does most of its decompression on the client side (e.g. during sync, which is the most common, the server transfers the zipped file to the client and the client unzips it), so if you turn off compression for files that would actually benefit from it, you're sacrificing storage on the server (which is potentially expensive) as well as network bandwidth (which is also potentially precious if you're geographically distributed) to save a very small amount of CPU time on the client (which is basically free).  Some of the filetypes in your typemap there will actually compress pretty nicely (for example BMP and WAV both *can* contain compressed data but in my experience are usually uncompressed and will shrink quite a bit when zipped).

Note that the built-in filetype detection specifically checks for common compression headers, including GIF, JPG, and GZ, so typemapping these should be unnecessary:  https://swarm.worksh...ilecheck.cc#302




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users