Posted 15 August 2019 - 09:25 PM
Am I missing some built-in functionality? Is there another tool to help with this? I toyed with the p4 log analyzer but that is slow (we have large logs) and takes a lot of space if we want to track data over time. It's useful for some things, and I would consider it for this, but it doesn't excite me for this purpose.
Posted 16 August 2019 - 02:24 PM
Does your protection table have any lines with multiple wildcards? If so, I can shortcut a lot of investigation for you and tell you that's the thing to optimize out. Here's a script I wrote in large part to make it easier to eliminate expressions like "//depot/*/foo/..." which tend to be the biggest performance killers: https://swarm.worksh...main/protexp.pl
Posted 16 August 2019 - 05:13 PM
Far worse are the filters to keep us free of the myriad files third party tools generate that historically caused problems when checked in- either because of size or because of permission issues. I'm going to engage consulting on that because the list we got from the CAD team includes lines with two and three wildcards.
Posted 16 August 2019 - 05:43 PM
The ceiling on that number is pretty high (thousands if not millions). The catch is the protection table isn't interpreted in a vacuum; it gets joined with every other mapping involved in running a given command (which usually includes the client view). So in some situations your multiple wildcards behave fine, and then one person puts a couple multiple wildcards in their own client (or a branch view, or a command argument, or all of the above) and now suddenly the computed mapping is five million lines long because things went from O(n) to O(k^n) or whatever (I'm fuzzy on the math, but it's bad). Every time someone hits a map.joinmax error (or crashes their server because they overrode map.joinmax and overflowed memory) it's because they had multiple wildcards in their protection table and it was fine until it suddenly wasn't.
Here's what the different levels correspond to: https://swarm.worksh.../map/mapdebug.h
You can look at the debugging statements in the rest of the map/ folder that are gated on those flags to get an idea of what they're dumping out. This line seems like the most likely to be useful for answering the question of "how often does this mapping line get referenced?": https://swarm.worksh.../maphalf.cc#419
I'm not sure what you'd actually do with that data or if it would be in any way useful, but that's about the only window there is into what the mapping code is thinking.
Posted 19 August 2019 - 10:05 PM
Sam alludes to times when a user might do a goofy query that hammers that type of line in the protections table, so to speak. My memory is a bit shady but I feel like I remember that %% tags in a workspace view plus wildcards in the protections table could wreak havoc if wielded in just the right way.
I've got it down to five lines with an '*' in them, and all of them are at the end of the path/directory:
We may have been lucky in that the 'weird' lines we had weren't really necessary any more and were just old cruft. It was easy to get buy-in to remove them. The remaining wildcards contain no more than (checks notes) 47 directories each so it doesn't add up to much in our case.
Since we got through that I don't recall ever asking myself 'Maybe it's the protections table?'
We 'only' have around 2,800 lines in there. Anecdotally, 'p4 protects' is one of my more common daily commands that I run and I can't remember the last time I thought 'this is taking too long'. It returns in less than a second every time. That used to not be the case.
I don't think it does anything, but we also try to keep our protections table in alphabetical order by depot path. I think it generally has to take in the table as a whole every time anyway, but it's prettier this way and helps break it down by depot (now that we have real comments available!) and is easier for humans to read.
I also don't know if it would help any, but if you have a 'p4 protects' command you can run and it consistently takes a long time (or any other command for that matter), it helps to use 'p4 -Ztrack', which will dump out db table statistics about the transaction. I suspect complicated queries will have a higher row/scan value than expected. That's 100% conjecture, but whenever I get any random, odd performance issues, '-Ztrack' is the first line of attack (since I learned about it a few months ago.)
Staff SCM Engineer, Perforce Administrator
Dolby Laboratories, Inc.
1275 Market St.
San Francisco, CA 94103, USA
Posted 19 August 2019 - 11:14 PM
Right now all our projects are added in alphabetical order, which is about to change. We'll have a big section of older projects, since they get little activity, followed by a small section of active projects. But we still have a few dozen multi-wildcard lines near the bottom of the table that I've yet to find a way to eliminate. That's where I'm expecting to burn some prepaid consulting time.
Also tagged with one or more of these keywords: protect, protects, statistics, visibilily
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users