Jump to content


GetDepotDirs() - client map too twisted for directory list


  • Please log in to reply
20 replies to this topic

#1 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 29 November 2019 - 09:33 PM

I'm using the .NET API to make an app, and I'm basically trying to list the the directories/files which are mapped to a given workspace, i.e. the directories and files which would be synced if I get latest. I've created a new workspace which uses a certain stream, but have not synced it yet. I'm calling GetDepotDirs():

repository.GetDepotDirs(new GetDepotDirsCmdOptions(GetDepotDirsCmdFlags.CurrentClientOnly, null), "//my_workspace/*");

And I get the error "client map too twisted for directory list". Is there some way I can fix this, or is there a different command I should be using to get this information? I'd rather not have to parse the workspace mappings manually and then query all the directories individually. This works fine with GetDepotFiles.

#2 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 03 December 2019 - 05:09 PM

Is //my_workspace a depot directory? When I run that command using //p4bill_workspace from the command line or with p4api.net I get the same error (client map too twisted for directory list). p4bill_workspace is my workspace that uses a stream, but there is no //p4bill_workspace that exists as a depot directory. When I run the command with //* I get the expected results:

//first_immediate_mapped_subdirectory
//second_immediate_mapped_subdirectory
//third_immediate_mapped_subdirectory

GetDepotDirs is running the command p4 dirs:
https://www.perforce...rs.html#p4_dirs
and since it only returns the immediate subdirectories, you'd need to run it again on each directory returned from the previous command run to get subdirectories at lower levels.

#3 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 03 December 2019 - 09:13 PM

View Postp4bill, on 03 December 2019 - 05:09 PM, said:

Is //my_workspace a depot directory?
No, it's my workspace. It works fine if I try to get dirs on a depot folder, but what I'm trying to achieve is take a given workspace, and figure out what folders should be in there. So if I were to get latest on my clean workspace, what folders would I see in there afterwards? I can do this for files with GetDepotFiles (or p4 files from command line), but it doesn't work for dirs. Is the only way to achieve this for me to write some code to parse the workspace mappings and understand the rules of them etc, or is there some other command I should use for this?

#4 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 03 December 2019 - 10:54 PM

I think the best way to achieve that is by running GetDepotDirs on the depot root (//*) and then running GetDepotDirs on the directories returned from the initial command, repeating and building a list until running GetDepotDirs does not return any directories.
The GetDepotDirsCmdFlags.CurrentClientOnly command flag will filter //* to only include directories in the current client view.

The reason //my_workspace works for p4 files is that command will take client(workspace) syntax when specifying the file argument. p4 dirs only takes a depot directory for the dir argument.

Something like this should work:
List<String> allDepotDirs = new List<string>();
List<String> dirsToCheck = new List<string>();
List<String> lowerDirs = new List<string>();
GetDepotDirsCmdOptions opts =
new GetDepotDirsCmdOptions(GetDepotDirsCmdFlags.CurrentClientOnly, null);
List<String> topDepotDirs = (List<string>)rep.GetDepotDirs(opts, "//*"); ;
while (topDepotDirs != null && topDepotDirs.Count>0)
{
lowerDirs = new List<string>();
foreach (string dir in topDepotDirs)
{
allDepotDirs.Add(dir);
dirsToCheck.Add(dir);
}
foreach (string dir in dirsToCheck)
{
lowerDirs.AddRange(rep.GetDepotDirs(opts, dir + "/*"));
}
dirsToCheck = new List<string>();
topDepotDirs = lowerDirs;
}

where allDepotDirs will end up containing all of the directories (in depot syntax) that are in the current client (workspace) view. If you need those paths in local or workspace syntax, you'd need to run GetClientFileMappings on each of them (p4 where).

#5 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 09:25 AM

View Postp4bill, on 03 December 2019 - 10:54 PM, said:

where allDepotDirs will end up containing all of the directories (in depot syntax) that are in the current client (workspace) view. If you need those paths in local or workspace syntax, you'd need to run GetClientFileMappings on each of them (p4 where).
Ah ha, p4 where, awesome. Thanks!

#6 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 09:37 AM

Does p4 where work for directories? If I do:

p4 where //path/to/dir
I get "file(s) not in client view"

p4 where //path/to/dir/
Null directory // not allowed in //path/to/dir/

p4 where //path/to/dir/*
kind of works but not sure if this is the correct way to be doing it, what I get is
//path/to/dir/%%1 //workspace/path/%%1 C:\workspace\path\%%1

#7 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 04 December 2019 - 02:31 PM

Your first example is the correct way:
D:\test_ws>p4 where //depot/changenotify
//depot/changenotify //test_ws/changenotify D:\test_ws\changenotify
My guess on the "file(s) not in client view" message that you are seeing is that either //path/to/dir is not in your workspace mapping or you might not have the workspace set in your environment. You can see what P4PORT, P4USER, and P4CLIENT are set to with p4 set. Or you can add global variables to the command, specifying them:
p4 -p<port> -u<user> -c<client> where //depot/changenotify
If you use p4api.net, (GetClientFileMappings) you should just be getting the expected port, user, client from the connection you've established.

#8 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 02:47 PM

View Postp4bill, on 04 December 2019 - 02:31 PM, said:

My guess on the "file(s) not in client view" message that you are seeing is that either //path/to/dir is not in your workspace mapping or you might not have the workspace set in your environment.
It's definitely in my workspace, because I can sync it through p4v. I still get "file(s) not in client view" even when explicitly setting the client with -c my_workspace

#9 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 02:53 PM

AH - sorry, I'm being daft, the directory is empty (just contains subdirs). For directories with files in it works fine.

#10 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 04 December 2019 - 02:53 PM

And an example of GetClientfileMappings:
List<String> localDirs = new List<string>();
List<String> workspaceDirs = new List<string>();
foreach (string dir in allDepotDirs)
{
if (!dir.TrimStart('/').Contains("/"))
{
continue;
}
FileSpec fileSpec = new FileSpec(new DepotPath(dir), null, null, null);
IList<FileSpec> fileSpecs = rep.Connection.Client.GetClientFileMappings(fileSpec);
foreach(FileSpec mappings in fileSpecs)
{
localDirs.Add(mappings.LocalPath.Path);
workspaceDirs.Add(mappings.ClientPath.Path);
}
}
GetClientFileMappings does not work on depots, so that is what the .Contains("/) check is for - ensuring that the dir that p4 where is running against is at least one level below //depot.

#11 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1120 posts

Posted 04 December 2019 - 03:57 PM

View Postcodersblock, on 04 December 2019 - 09:37 AM, said:

Does p4 where work for directories? If I do:

p4 where //path/to/dir
I get "file(s) not in client view"

p4 where //path/to/dir/
Null directory // not allowed in //path/to/dir/

p4 where //path/to/dir/*
kind of works but not sure if this is the correct way to be doing it, what I get is
//path/to/dir/%%1 //workspace/path/%%1 C:\workspace\path\%%1

To refer to everything under a directory path use the "..." wildcard, exactly the same way it's specified in your View or the way you'd pass it as an argument to any command that operates on files:

p4 where //path/to/dir/...


The "*" wildcard refers only to top level files (i.e. all subpaths that do not include the "/" character).  The mapping logic expands "*" to positional wildcards like "%%1" so that if you specify more than one, each wildcard on one side of the mapping is easily associated with a specific instance on the other side (if you use the positional wildcards in a mapping that you specify yourself, you can reorder them to alter the way that different directories are nested).

#12 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1120 posts

Posted 04 December 2019 - 04:06 PM

Going back to your original post:

I'm basically trying to list the the directories/files which are mapped to a given workspace


I'd just use GetDepotFiles (or whatever the API equivalent of "p4 files //my_workspace/..." is) to get all the files.  If you want to collapse them into directories, do that as a post-processing step (i.e. by looking for common prefixes).

Bear in mind that you might have a situation where these two files both exist:

//path/to/dir/foo
//path/to/dir/bar

but only one of those files is mapped in your client view -- if you're post-processing the files list to turn it into a list of directories, think about whether it's okay to include directories like //path/to/dir that are only partially mapped!  Philosophical conundrums like this are why p4 dirs will sometimes give you a "too twisted" error -- there isn't always an unambiguously correct way to express the concept of "what depot directories are mapped to this arbitrary client path" due to that the fact that client mappings can operate at finer levels of granularity (i.e. the file level, which is why it's always trivial to get this information as a list of individual files) and can arbitrarily rearrange the entire tree structure as part of the mapping.  You might be okay with fudging it one way or another for your particular use case, just make sure you understand that's what you're doing.  :)

#13 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 05:01 PM

View PostSambwise, on 04 December 2019 - 04:06 PM, said:

I'd just use GetDepotFiles (or whatever the API equivalent of "p4 files //my_workspace/..." is) to get all the files.  If you want to collapse them into directories, do that as a post-processing step (i.e. by looking for common prefixes).
GetDepotFiles with //my_workspace/... is pretty fast, just 5 seconds to get everything for my workspace. However, they come back as depot paths which I then need to convert to workspace paths, ~40k, which is taking a loooong time.

#14 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1120 posts

Posted 04 December 2019 - 11:14 PM

View Postcodersblock, on 04 December 2019 - 05:01 PM, said:

they come back as depot paths which I then need to convert to workspace paths

In that case use "fstat", e.g.:

p4 -F %clientFile% fstat //my_workspace/...

Not sure offhand what the .NET API version of this is, but I'm sure there's a direct equivalent that you can pull all the fstat fields from.

You might also be interested in "p4 have" (which gives you the depot and client paths of everything you've previously synced) and "p4 sync -n" (which gives you the depot and client paths of everything that you need to sync).

#15 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 December 2019 - 11:30 PM

Is running fstat on 40k files going to be any faster than running where?

#16 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1120 posts

Posted 04 December 2019 - 11:36 PM

Running fstat once on 40k files is going to be much faster than running where 40k times on one file at a time, yes, because network I/O is the main bottleneck for simple db queries like these.  :)

If you ran "p4 where" once to fetch the entire mapping and then did the translation of the "p4 files" output on the client side (which you can do via the MapApi class in the C++ API; I'm not sure if there's a .NET equivalent to that one) then that might end up being a little faster.  But definitely running "p4 where" thousands of times is going to be a lot slower than running one or two commands that do the whole thing in one shot.

Either "sync -n" or "have" will be a little faster than fstat because those both return less data (fstat is kind of a one stop shop for all kinds of random file metadata).  Note that in your original post:

Quote

the directories/files which are mapped to a given workspace, i.e. the directories and files which would be synced if I get latest

you're asking for two different things -- the files which are mapped to a given workspace is best represented by "files"/"fstat", whereas the files which will be synced if you get latest is best represented by "sync -n" (running a command like sync with the "-n" flag is precisely "show me what would happen if I did this").  More information about what your app is going to do with the information or what it's supposed to represent would make it easier to steer you in the right direction.  :)

#17 p4bill

p4bill

    Advanced Member

  • Members
  • PipPipPip
  • 217 posts

Posted 05 December 2019 - 04:10 AM

Here is what you should use for p4 sync -n -f
FileSpec fileSpec = new FileSpec(new DepotPath("//..."), null, null, null);
SyncFilesCmdOptions opts = new SyncFilesCmdOptions(SyncFilesCmdFlags.Preview|
SyncFilesCmdFlags.Force);
IList<FileSpec> syncedFilesPreview =rep.Connection.Client.SyncFiles(opts, fileSpec);

I add the -f because if you have any files that are already synced, they will not appear in syncedFilesPreview. So, if you want to see all the files that would be synced to an empty workspace (whether it is or not), you need to use -f (force).

syncedFilesPreview will be a list showing depot path and local/client path for each file, including revision.

#18 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 09 December 2019 - 09:21 AM

View PostSambwise, on 04 December 2019 - 11:36 PM, said:

you're asking for two different things -- the files which are mapped to a given workspace is best represented by "files"/"fstat", whereas the files which will be synced if you get latest is best represented by "sync -n" (running a command like sync with the "-n" flag is precisely "show me what would happen if I did this").  More information about what your app is going to do with the information or what it's supposed to represent would make it easier to steer you in the right direction.  :)
Sync -n might be the droid I'm looking for in that case.. To give you more context, I'm attempting to write a VFS so I can mount a particular workspace/stream/CL/etc as a virtual drive/folder and sync files on demand. Switching streams at work is pretty painful, can take upwards of half an hour, and a lot of the files synced I don't even use.

#19 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1120 posts

Posted 09 December 2019 - 03:06 PM

View Postcodersblock, on 09 December 2019 - 09:21 AM, said:

a lot of the files synced I don't even use.

I assume you already know about "p4 switch --no-sync".  Have you looked at creating virtual streams that only sync the files you use?  (Create a child stream with type "virtual" and edit the Paths to only include the files you need.)

#20 codersblock

codersblock

    Member

  • Members
  • PipPip
  • 10 posts

Posted 09 December 2019 - 03:20 PM

View PostSambwise, on 09 December 2019 - 03:06 PM, said:

I assume you already know about "p4 switch --no-sync".  Have you looked at creating virtual streams that only sync the files you use?  (Create a child stream with type "virtual" and edit the Paths to only include the files you need.)
We do use virtual streams to help with this a bit, e.g. programmers streams won't include all the raw art assets they don't care about etc, but specifically what files I end up using will vary from task to task, and is pretty hard to pre-empt.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users