Jump to content


P4VS on large solutions

P4VS fstat

  • Please log in to reply
11 replies to this topic

#1 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 22 August 2014 - 09:06 AM

Hi there


We're using P4VS in Visual Studio 2013. The issue is that upon loading our (large) solution, VS hangs for ~5 minutes and issues about 20k fstat commands.

This is similar to: http://forums.perfor...-p4vs-slowness/


Here's my theory to why this is happening:
  • Looking at the logs, P4VS gets its list of all files already submitted by using this command: fstat -Olhp -m 100000 d:\xxx\yyy\...
  • However, running that command without the -m flag returns around 120k results, which means P4VS thinks around ~20k files aren't submitted.
  • And that's the problem: It seems that P4VS will call fstat for each of those 20k files, and this causes the hang.

Notes:
In the P4VS options, I've tried:
  • "Automatically add new files to Perforce": When unticked, those fstats are still performed.
  • "Treat Solution/Projects as directories when selected": When ticked, the hang doesn't occur but this breaks things like automatically checking out a file when you start typing, which defeats the point of using the plugin in the first place.
Versions:
  • Microsoft Windows 7 Enterprise, Version 6.1 (Build 7601: Service Pack 1)
  • Microsoft Visual Studio Professional 2013, Version 12.0.30324.00 Update 2 RC
  • Microsoft .NET Framework, Version 4.5.50938
  • P4VS 2013.3.78.1524 and 2014.1.85.4506


Any help appreciated.

Thanks!


Justin

#2 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 25 August 2014 - 03:03 PM

Hi Justin,

I'd expect the "Treat Solution/Projects as directories when selected" preference you mentioned to improved things. If P4VS is not automatically checking out files for edit (or at least prompting you to do so), that sounds like a bug.

Can you provide some details with regard to the workflow being broken when that setting is checked? Are you able to edit the file, but it is not checked out?

#3 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 26 August 2014 - 10:49 AM

With the 'Treat...' option checked:
  • There is no longer a long hang after loading the solution. However I do get a question mark '?' next to all items in the Solution Explorer.
  • When I try to make an edit, I get a popup: Managed Source Control Profider - The file xxx you are trying to edit is read only on disk. Do you want do make the file writable so you can continue editing it?

The server logs show fstat calls still being issued, for well over an hour now(!!) The calls are definitely being issued to folders that contain only by-products of compiling, which can only be making things worse.

2014/08/26 10:12:16 pid 327 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-clients -u <user> -m 1'
2014/08/26 10:12:16 pid 327 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-login -s'
2014/08/26 10:12:16 pid 327 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-client -o <workspace>'
2014/08/26 10:12:16 pid 327 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-opened -m 1'
2014/08/26 10:12:16 pid 327 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-changes -l -c <workspace> -m 10 -s pending -u <user>'
2014/08/26 10:13:22 pid 304 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\...'
2014/08/26 10:13:22 pid 304 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\...'
2014/08/26 10:13:32 pid 308 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat <workspace_root>\<solution_file>'
2014/08/26 10:13:43 pid 315 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<intermediate_files>'
2014/08/26 10:13:56 pid 318 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-depots'
2014/08/26 10:13:56 pid 319 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-client -o <workspace>'
2014/08/26 10:17:31 pid 851 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<subfolder01>\*
2014/08/26 10:17:46 pid 855 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<subfolder02>\*
2014/08/26 10:17:56 pid 860 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<subfolder03>\*
2014/08/26 10:18:11 pid 869 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<subfolder04>\*
...
2014/08/26 11:25:58 pid 3936 <user>@<workspace> <IP> [P4VS/2014.1.85.4506] 'user-fstat -Olhp -m 100000 <workspace_root>\<subfolderXX>\*


What it looks like to me is:
  • The fstat calls are now asynchronous, so although the IDE is quicker to respond to user input, it's still churning through fstats.
  • P4VS is uninitialized until all those fstat calls are done. This is why the option seems to 'break' the plugin for me, since it takes over an hour to initialize.
  • P4VS churns through every local folder under the workspace root regardless of whether it's relevant to the solution or not.
  • My workspace root contains 107k files and 13k dirs, many of which are irrelevant to the VS solution.
  • 'Treat...' option widens the scope of the fstat calls to the entire workspace, instead of just the solution itself.


#4 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 26 August 2014 - 11:11 AM

Thanks Bill, hope my previous post sheds more light on what I'm experiencing.

Also, my gut feeling is that this issue is caused by having >100k files checked in. This makes sense in that we didn't have issues with the plugin until a few months ago, and since then the issue has progressively become worse.

It seems compelling to update the plugin so that the -max flag value of 100k is exposed, so I can increase it just to try out.

Thanks,

Justin

#5 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 29 August 2014 - 02:15 PM

Thanks, Justin. With regard to this comment:

View PostJustin, on 26 August 2014 - 11:11 AM, said:


Also, my gut feeling is that this issue is caused by having >100k files checked in. This makes sense in that we didn't have issues with the plugin until a few months ago, and since then the issue has progressively become worse.


Are you saying that it previsouly worked better a few months back when there were less than 100K files checked in?

I can probably get you a test build sometime next week to expose that 100k limit.

Is your solution root the same as your workspace root? The way you describe the 'Treat...' option working (widening the scope), seems like a bug to me.

#6 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 12 September 2014 - 01:17 PM

Sorry for the delay in responding.

>> Are you saying that it previsouly worked better a few months back when there were less than 100K files checked in?
Yes definitely.

>> I can probably get you a test build sometime next week to expose that 100k limit.
That would be great to test out -- especially if it could help out others too.

>> Is your solution root the same as your workspace root? The way you describe the 'Treat...' option working (widening the scope), seems like a bug to me.
Yes they are the same; our workspace root is D:\foo\bar\ and the .sln file is D:\foo\bar\baz.sln

#7 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 12 September 2014 - 03:16 PM

That preference will not do you much good in your case. In fact, depending on the number of files not in the sln, it could be worse. The preference was put in to help with large solutions as Visual Studio notifys the plugin about files under source control one-at-a-time. So, rather than running fstat on each file individually, it will run it on /<sln dir/... Since that is the same as your workspace root, that is why all of those unrelated files are being returned in the data.

We've got a request for a Visual Studio specific INGNORE file. That may be something that could help in this case.

I'll message you directly about a test build.

#8 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 12 September 2014 - 03:25 PM

Ah right... yeah in my case we store everything in the workspace; our .sln just happens to be at the workspace root for convenience, so you're right in it making things worse.

Look forward to the test build.

Thanks!

#9 rmorse-EA

rmorse-EA

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 31 October 2014 - 04:04 PM

Hi,  I just discovered that this is the exact problem I've been having for months.  I was mistakenly attributing it to another tool in use, but I'm glad I know now.  I thought to turn on the logging for P4VS and that's how I discovered it, then I searched the forums and found this post.

I can confirm the symptoms are the same.  On solution open, after the automatic p4 connection is made, it runs the fstat command with the 100000 value.  After a few seconds, it will begin running fstat commands first, one at a time, on "build" directories (folders where items were built to, not source-controlled folders) then on every file in that directory individually, stalling VS for 40 minutes or more in my case.  We have well over 100000 files in our solution/work space.

Was it tested to increase that value and did it have any effect?  I'd love to get that change if so.

If not, here's another idea from observed behavior:

While it's fstat'ing every file (just after solution open) I'm able to "Close Connection to the Perforce Depot", and after a couple seconds, it stops the commands and VS gets back to normal response time.  I noticed that if I then Open a (same) connection via the File menu, it runs the fstat commands again, on the "build" folders, but doesn't run the individual commands.  I can't tell why as there isn't logging on that, but the behavior is desirable.

So a first thought is maybe it's an accident that there's a difference and I've gotten into trouble, but in testing this it appears to work fine.  If I then open a source file, it runs an fstat on the folder containing it, but no additional commands and no delay noticed.

Appreciate any assistance on this.  Thanks,
Ryan.

#10 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 04 November 2014 - 06:17 PM

Hello, yes it was tested with an increased value and there was an improvement (in Justin's case from ~20k fstat calls to ~12k). Though, we are not certain that upping the value, or allowing it to be set in preferences is the best fix. We are investigating the best way to handle these large solutions. It was probably mentioned earlier in the thread, but the 2 options we have with the current behavior are:

1. fstat on each file in a controlled solution that VS notifies P4VS of. This happens one-at-a-time.

2. attempt to be more efficient by determining the solution root and if the "Treat solutions as directories" preference is set, do the fstat on //<path to sln dir>/...

The potential issue with 1 is that a solution can have thousands of files so that would be thousands of fstats run.

The potential issue with 2 is that there could be thousands of files under the solution root that are not part of the solution but are part of returned fstat data.

We are still looking into this here. (Thanks to Justin for the help with the investigation thus far).

#11 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 05 November 2014 - 12:18 PM

Nice to know that I'm not alone in this matter, and that it's still being looked at. :)

Bill, have you been able to reproduce the issue at all? I think the biggest hurdle towards getting this sorted is providing solid reproduction steps (which I've found unwieldy to produce).

#12 Justin

Justin

    Advanced Member

  • Members
  • PipPipPip
  • 34 posts

Posted 21 November 2014 - 10:43 AM

I haven't heard many complaints about the issue since so I figured people were getting by happily enough, but I just had a user who needed an alternative and started using P4EXP; it must still be a critical issue (for many) if someone felt forced to abandon P4VS entirely.

I've advised him to instead use custom commandline tools in VS which are basic but covers 95% of his needs - he's happy with it now (he sticks to the same workspace most of the time so it should keep working).

Tools > External Tools > Add

Title:
Command:
Arguments:

[P4] Open for Add
C:\Program Files\Perforce\p4.exe
add $(ItemPath)

[P4] Open for Edit
C:\Program Files\Perforce\p4.exe
edit $(ItemPath)

[P4] Open Directory for Edit
C:\Program Files\Perforce\p4.exe
edit $(ItemDir)*

[P4] Recursive Open Directory for Edit
C:\Windows\system32\cmd.exe
/c dir /s/b $(ItemDir) | p4 -x- edit






Also tagged with one or more of these keywords: P4VS, fstat

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users