Jump to content


p4 reconcile -w -f and -//workspace/excludes/...


  • Please log in to reply
12 replies to this topic

#1 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 06 July 2018 - 10:29 AM

I'm trying to use "p4 reconcile -w -f" to clean a workspace in between builds, and it's leaving behind some files that are not in the repo.

Assume a workspace like this, which excludes 'tests' directories to save time and space:

//top/... //workspace/top/...
-//top/.../tests/... //workspace/top/.../tests/...

A build step causes the local creation of the following (note the "/tests/" in the 2nd path):

//top/module1/code/3rdpartything/thing.c
//top/module1/code/3rdpartything/stuff/tests/testthing.c

Running "p4 reconcile -w -f" correctly removes

//top/module1/code/3rdpartything/thing.c

but leaves behind

//top/module1/code/3rdpartything/stuff/tests/testthing.c

Can anyone come up with a way to rewrite the -//workspace/.../exclude/... or alter the clearup command such that "p4 reconcile" actually deletes these temporary, uncontrolled artefacts?

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 616 posts

Posted 07 July 2018 - 01:09 AM

For the workflow you describe I don't think it makes sense to exclude those files from the client view.

If you simply remove the exclusion, then Perforce commands (including "clean" aka "reconcile -w") will be able to operate on those files.

#3 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 08 July 2018 - 09:55 AM

Thanks for taking time to help out ... Given your comment I think I misused "//" in the original post, and made it look at a glance like the directories I need "cleaned" are in the repo - they are not. I should have shown directory paths.

Quote

If you simply remove the exclusion, then Perforce commands (including "clean" aka "reconcile -w") will be able to operate on those files.

That's true. However I want to exclude the .../tests/... repo data as our checkout is transatlantic and that saves not inconsiderable time as well as disk space, so removing that line works around the clean up problem (which feels like a bug to me) but increases build time.

To make it hopefully clearer, take this as an on disk workspace layout:

/my/p4/workspace/my_real_code.c # from //repo/my_real_code.c
/my/p4/workspace/tests/my_massive_test_stuff.c   # from //repo/tests/my_massive_test_stuff.c
/my/p4/workspace/3rdpartything/thing.c # not in the repo at all (generated at build time)
/my/p4/workspace/3rdpartything/stuff/tests/testthing.c   # not in the repo at all (generated at build time)

The goals are that:
1 On checkout I can avoid all version controlled directories with "/tests/" in the path
2 On cleanup I can remove everything which is not version controlled from the workspace
3 On cleanup I can make sure everything which is version controlled is as per repo

I can meet goals 1 and 3 with this in the workspace spec:
-//repo/.../tests/... //workspace/.../tests/...

But that has the side effect of making "clean" skip over 3rdpartything/stuff/tests/, even though the directory tree 3rdpartything/ is not even in the repo.

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 616 posts

Posted 09 July 2018 - 05:30 PM

Can you add "3rdpartything" to your view, then?

//repo/... //workspace/...
-//repo/.../tests/... //workspace/.../tests/...
//repo/do-not-submit/... //workspace/3rdpartything/...

The reason that path is currently skipped by "clean" is that if something isn't mapped in your view, it's not part of your workspace.  "p4 clean" will no more delete those files than it would delete your /usr/local/bin folder.  :)

#5 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 31 July 2018 - 09:41 AM

(sorry for the delay, been on holiday)

> Can you add "3rdpartything" to your view, then?

The objective is to stop doing what I currently do:

rm -rf workspace/
p4 sync -f workspace/...

and use the reconcile tool instead, crucially without any assumed prior knowledge about "3rdpartything".

This obviously isn't a developer machine use case, this is the "repeatable formal build" use case, where I want to know beyond doubt that the checkout is clean. I've been trying to set up something in Jenkins using the perforce plugin, which uses commands like those above to reconcile - and is definitely a whole lot faster than "rm -rf ; p4 sync -f" - but unfortunately doesn't seem like it can be relied on.

I think I'm going to have to get the plugin to brute force the cleanup :-(

#6 dave.foglesong

dave.foglesong

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 31 July 2018 - 08:19 PM

This may be more scripting then you'd want to do, but you could probably do something like: Run reconcile. Get list of all files in workspace. Run "p4 fstat -Rc" to get list of files mapped to client (there's a fstat option to get local workspace paths). Any file that's not in the fstat output gets deleted.

#7 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 616 posts

Posted 31 July 2018 - 11:09 PM

View Postcallmewilko, on 31 July 2018 - 09:41 AM, said:

This obviously isn't a developer machine use case, this is the "repeatable formal build" use case, where I want to know beyond doubt that the checkout is clean.

Map everything in the client then.

//repo/... //workspace/...

The exclusion for "tests" seems like the sort of thing that would be intended to keep a developer from accidentally checking in the tests folder.  Your automation has no need of that sort of filtering, and wants to have a high level of confidence that everything is controlled.  Just have the client map everything.  :)

#8 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 01 August 2018 - 09:29 AM

View PostSambwise, on 31 July 2018 - 11:09 PM, said:

Map everything in the client then.
//repo/... //workspace/...

Indeed - in fact we've been doing that for the last decade. And therein lies the rub. Old school does a big, monolithic, all-the-pieces build once a day, and the extra checkout time and disk space is insignificant.

I'm trying to drag "builds" into the modern world and build usable pieces (but still, for now, out of a monolithic code base) in timescales that reduce commit-to-feedback times drastically, and so the overhead of checking out stuff the build doesn't need (multiple times in multiple builds, as opposed to once in a monolithic build) has become "significant".

View PostSambwise, on 09 July 2018 - 05:30 PM, said:

Can you add "3rdpartything" to your view, then?
//repo/... //workspace/...
-//repo/.../tests/... //workspace/.../tests/...
//repo/do-not-submit/... //workspace/3rdpartything/...
Nice bit of lateral thinking, it works! I changed it slightly to match the exclusion:

//repo/.../do-not-submit/... //workspace/.../tests/...

It does however rely on someone who isn't me, in 6 months, needing to remember to match any exclusion with a ghost client side inclusion, which is the kind of thing I like to avoid - especially as it's for corner cases. But kudos to you: I tried to work out something like this, but by messing with the right side of the exclude, and couldn't make it work. Didn't think of this solution :-)

View PostSambwise, on 09 July 2018 - 05:30 PM, said:

The reason that path is currently skipped by "clean" is that if something isn't mapped in your view, it's not part of your workspace.  "p4 clean" will no more delete those files than it would delete your /usr/local/bin folder.  :)
Though of course /usr/local/bin isn't under the managed workspace root, whereas things "p4 reconcile" deletes are.

#9 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 01 August 2018 - 09:34 AM

View Postdave.foglesong, on 31 July 2018 - 08:19 PM, said:

This may be more scripting then you'd want to do, but you could probably do something like: Run reconcile. Get list of all files in workspace. Run "p4 fstat -Rc" to get list of files mapped to client (there's a fstat option to get local workspace paths). Any file that's not in the fstat output gets deleted.

This might be useful actually, as that could be written once in the generic checkout handling code that is shared between builds, and means nobody creating jobs needs to remember anything special about creating workspaces. Our file count is quite high so I'm wondering how long it might take, but it seems like it would still be an overall saving.

#10 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 616 posts

Posted 01 August 2018 - 03:31 PM

Another idea, if you want reconcile/clean to behave as if you had a wide open client view (i.e. clean everything under the Root rather than everything in your View) but you want your syncs to operate with a more limited View: have your script open up the View (such that it maps everything under the Root) just for the reconcile.  

p4 client -o > original_client.txt
p4 --field "View=//repo/... workspace/..." client -o | p4 client -i
p4 clean
p4 client -i < original_client.txt


#11 dave.foglesong

dave.foglesong

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 01 August 2018 - 07:19 PM

View Postcallmewilko, on 01 August 2018 - 09:34 AM, said:

This might be useful actually, as that could be written once in the generic checkout handling code that is shared between builds, and means nobody creating jobs needs to remember anything special about creating workspaces. Our file count is quite high so I'm wondering how long it might take, but it seems like it would still be an overall saving.

If you go this route, I realized you don't need to use fstat -- just run "p4 have" on the client after the reconcile and it will give you the list of local files. "have" hits less db files than "fstat" so it should run faster.

> our checkout is transatlantic and that saves not inconsiderable time as well as disk space,
> Our file count is quite high

You may want to look into setting up a proxy or edge server that is co-located with your build farm. The proxy is easier to setup and maintain but an edge server would provide more benefit. But either would help if you do large syncs frequently across a WAN.

#12 callmewilko

callmewilko

    Member

  • Members
  • PipPip
  • 10 posts

Posted 02 August 2018 - 02:49 PM

We have a proxy, which helps (considerably), but I hadn't heard of the edge server - I shall dig into that.

I also thought "why not use p4 have" - obviously the fstat route also tells you about things that you "p4 haven't", and I had yet to get around to working out which was right for my needs - but I think in fact p4 have is enough. Regardless, I liked learning about p4 fstat - another tool for the box.

Quote

have your script open up the View (such that it maps everything under the Root) just for the reconcile.

that's doable generically too, thanks.

I've shifted my understanding of workspace mappings from "definition of what content should be there" to "mask to use when actioning requests" :-)

#13 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 616 posts

Posted 02 August 2018 - 03:06 PM

View Postcallmewilko, on 02 August 2018 - 02:49 PM, said:

I've shifted my understanding of workspace mappings from "definition of what content should be there" to "mask to use when actioning requests" :-)

The next level of enlightenment is when you automatically conceptualize it as a map rather than a filter.  :)




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users