Jump to content


Best typemap compression practices for Spec depot

spec depot typemap best practices

  • Please log in to reply
2 replies to this topic

#1 Scott Gilliland

Scott Gilliland

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 21 February 2019 - 05:38 PM

Hello All,

Performed a quick search to see if this discussion had already been had and didn't see anything, so I'm creating a new topic. Apologies if this has already been covered. A quick question for admins regarding their spec depots and compression through the type map spec definition: do you use compression in your spec depot, and do you compress all or just some of it?

I'm interested in whether or not compression of the spec depot is done by most or considered a best practice, or if only some //spec types are compressed, or no compression is used. We have not been running compression (text+C in typemap definition for the //spec depot) but have recently turned it on for our client specs due to large revision counts now being encountered due to Jenkins builds. I know that we could exclude certain wildcard matches from being versioned at all, or limited with the +S flag, but our users have a wide variety of Jenkins client names making catching them with wildcard use problematic. To my knowledge regular expressions cannot be used in typemap definitions. So, we initiated compression of client specs as a first step our mitigation process. The question now becomes, if we're compressing clients should we simply compress every spec?

Interested in thoughts and my thanks to anyone willing to share their thoughts on this.

Regards,
Scott

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 800 posts

Posted 22 February 2019 - 12:49 AM

IMO the easiest thing is to use +C for all spec depots.  The main reason to use text+C vs the default "text" is that the default mode for text is RCS delta storage, which is generally pretty space-efficient (especially for things like source code which tend to have relatively small individual changes) but doesn't scale well on files with thousands of revisions.  With text+C each revision is stored independently, which eliminates the problem of having to compute an increasingly long series of deltas every time the spec is updated.

#3 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 163 posts
  • LocationSan Francisco, CA

Posted 17 April 2019 - 11:19 PM

I thought I was going nuts because we have no typemap entry for //spec/..., yet all our spec files are compressed. Then I remembered we have 'lbr.autocompress=1' set as a server configurable, which defaults to compressed storage for all text files.

You might also consider doing this globally because of what Sam mentions about RCS files. We have had human-observable performance issues with some large text files with thousands of revisions. To construct the revision, RCS files essentially have to take the base revision and apply patches up until your revision. If you're working with, say, a 10 MB XML file with 5,000 revisions it takes a noticeable amount of time to construct that, especially when compared to how fast it will deliver just one discrete gzipped revision.

This also has ramifications if you're in a commit/edge architecture. If you have an RCS file that's a few hundred megs in size then any new revisions have to send that entire file. If it's compressed, then it just sends one gzip.

Lastly, if you have a lot of large RCS files, it takes longer to 'p4 verify' them as it has to reconstruct every revision by doing a series of patches. Perforce can plow through those much faster when it just has to open a gzip file for each revision and checksum them.

Anyway, I'm very much pro +C on everything, not just //spec/...
-Matt Janulewicz
Staff SCM Engineer, Perforce Administrator
Dolby Laboratories, Inc.
1275 Market St.
San Francisco, CA 94103, USA
majanu@dolby.com





Also tagged with one or more of these keywords: spec depot, typemap, best practices

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users