Jump to content


Reduce HD space


  • Please log in to reply
16 replies to this topic

#1 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 05 March 2017 - 04:02 PM

Hello
I am using Perforce mostly as an easy way to share work between collaborators and to keep a backup of the latest work.
But as you know every time you change and existing file and submit it makes a new revision, which is good and dandy for the first extra copy, an extra backup in case someone submits wrong. But after that it starts to eat disk space and I don't need it. Is there a way to clean up, limit the amount of revisions, or any other solutions? It feels like there should be a better solution than removing the whole server and re-make it every time it gets to big.

best regards
nux

#2 P4Jen

P4Jen

    Advanced Member

  • Staff Moderators
  • 143 posts

Posted 06 March 2017 - 10:50 AM

You may find the +s filetype useful, as it allows you to purge older revisions of files.

More info about it is here:
  https://www.perforce....filetypes.html

Cheers,
Jen.


#3 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 06 March 2017 - 07:08 PM

Thanks! That looks like exactly what I need.
Though, I can't say that the manual is really helpful in describing how to use it. I saw that I can add it to current files in P4V, but I have to mark each file and change it. Is there a way to apply it to all files in deposit directly and make it standard for any newly added files too?

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 764 posts

Posted 06 March 2017 - 07:47 PM

Use the typemap:

https://www.perforce...p4_typemap.html

I would caution against doing this too broadly -- maybe just apply it to files with certain extensions that you anticipate consuming a lot of disk space relative to the value of their version history (e.g. built binaries that you can reconstruct from source if needed).  For example (just using +S4 on EXE and ZIP files as a suggestion):

Typemap:
	+S4 //....exe
	+S4 //....zip

I would specifically recommend against ever using +S for source code files -- they're too small and too important for the tradeoff to ever be worth it IMO.  :)

#5 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 07 March 2017 - 09:20 AM

Alright. Thank you. Really helpful community so far!

So, I wrote p4 typemap in cmd, and added this to the temp file that opened:
TypeMap:
+S3 //depot/....uasset
+S3 //depot/....umap

But I don't see any changes yet. There is still files with 4 revisions. Do I need to do something else before it applies? Or make it more specific? (I only have one server running)

And +S3, will that keep three extra revisions or current with two backup?
And is it possible to check for prefixes on files when adding this typemaps? Like texture files will look T_....uasset and really only need the latest version on server as they will have their original on my comp, so can I filter them out?

I am working in Unreal Engine and in my project these files can easily eat a lot of space, there being a lot of them and some being more than 25 mb each (will probably get bigger, and there will be more soon), depot is already more than 7GB and that is with an average of two revisions...

#6 Robert Cowham

Robert Cowham

    Advanced Member

  • PCP
  • 269 posts
  • LocationLondon, UK

Posted 07 March 2017 - 12:34 PM

The new Typemap you have created will only apply to any new files that are added to the repository.

For files that already existed before creating the Typemap, nothing has changed. You need to change them explicitly.

This is quite easy from command line:

- create a workspace which contains all the files you want to work on, e.g. View: //depot/... //my_workspace/...

- Sync the files to change:

    p4 sync //depot/....uasset

- Change their type:

  p4 edit -t +S3 //depot/....uasset

- Run "p4 opened" and look at the output to see that it is as expected

- Submit your changes:

  p4 submit -d "Changed filetype" //depot/....uasset

Rinse and repeat for other wildcards.
Co-Author of "Learning Perforce SCM", PACKT Publishing, 25 September 2013, ISBN 9781849687645

"It's wonderful to see a new book about Perforce, especially one written by Robert Cowham and Neal Firth. No one can teach Perforce better than these seasoned subject matter experts"
  • Laura Wingerd, author of Practical Perforce, former VP of Product Technology at Perforce

#7 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 07 March 2017 - 03:18 PM

(Tries not to scream in frustration.)
So, how do I change which is the active Client?

Quote

D:\UE4 Projects\TheCity4_15>p4 client TheCityTsuzumi
Client TheCityTsuzumi not changed.

D:\UE4 Projects\TheCity4_15>p4 edit -t +S3 //depot/....uasset
Client 'ADAM3' unknown - use 'client' command to create it.

I have my workspace which includes the files. But I have no idea on how to make p4 use this client...

#8 Robert Cowham

Robert Cowham

    Advanced Member

  • PCP
  • 269 posts
  • LocationLondon, UK

Posted 07 March 2017 - 03:28 PM

set P4CLIENT=My_workspace

If you do:

p4 set P4CLIENT=TheCityTsuzumi

then it will save the value in the registry for your user name.


Co-Author of "Learning Perforce SCM", PACKT Publishing, 25 September 2013, ISBN 9781849687645

"It's wonderful to see a new book about Perforce, especially one written by Robert Cowham and Neal Firth. No one can teach Perforce better than these seasoned subject matter experts"
  • Laura Wingerd, author of Practical Perforce, former VP of Product Technology at Perforce

#9 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 07 March 2017 - 03:43 PM

Alright. Thanks for the quick respons.
I actually fixed it and was about to remove the "cry post" when I saw the new post.
I went and set the p4config to the one for the workspace which automatically set it as active client.

I am able to change all the files and submit, but there is still more than 3 revisions in the depot.
Posted Image
Anything else I need to do to update it?

#10 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 07 March 2017 - 03:45 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


Or
set P4CONFIG=.p4config
echo P4CLIENT=My_workspace > .p4config

Ref: https://www.perforce...f/P4CONFIG.html

--
Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

Quote

-----Original Message-----
From: perforce-user [mailto:perforce-user-bounces@perforce.com] On
Behalf Of Robert Cowham
Sent: Tuesday, March 07, 2017 10:30 AM
To: perforce-user@perforce.com
Subject: Re: [p4] Reduce HD space

Posted on behalf of forum user 'Robert Cowham'.

set P4CLIENT=My_workspace



--
Please click here to see the post in its original format:
  http://forums.perfor...reduce-hd-space
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#11 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 07 March 2017 - 03:50 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


The count (S3) starts only after the file type changes. It does not work retroactively (it's against the SCM philosophy).
If you want to remove (physically) old revisions, your only option is obliterate.
https://www.perforce...obliterate.html

Some SCM admins philosophically object to this approach. I used to object to it, too, but finally outgrew this feeling. :-)

--
Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

Quote

-----Original Message-----
From: perforce-user [mailto:perforce-user-bounces@perforce.com] On
Behalf Of nux
Sent: Tuesday, March 07, 2017 10:45 AM
To: perforce-user@perforce.com
Subject: Re: [p4] Reduce HD space

Posted on behalf of forum user 'nux'.

Alright. Thanks for the quick respons.
I actually fixed it and was about to remove the "cry post" when I saw
the new post.
I went and set the p4config to the one for the workspace which automatically
set
it as active client.

I am able to change all the files, but there is still more than 3 revisions.
Posted Image
Anything else I need to do to update it?



--
Please click here to see the post in its original format:
  http://forums.perfor...reduce-hd-space
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#12 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 07 March 2017 - 04:00 PM

It feels like the quickest way here is to remove the whole server and reinstall and set the new typemap to -S3 before adding anything new.
From what I can see in Obliterate, you cannot remove every rev lower then three from(and including latest) without checking each file before. I wouldn't want to remove 3/3, but want to remove 3/6, keeping top three. Or am I wrong?

#13 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 07 March 2017 - 04:50 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


I'm not sure what 3/3 or 3/6 are in this message.
If I have #1-#6 of a file, I can obliterate any of those revisions, say, #1-#3, leaving #4-#6 intact.
That's from Perforce POV, of course.
Our users don’t like "holes" in the history, so we have a convention not to obliterate "in the middle" (or for that matter, sources of integrations). But that's only because we don’t like it - Perforce would happily obliterate anything. :)

--
Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

Quote

-----Original Message-----
From: perforce-user [mailto:perforce-user-bounces@perforce.com] On
Behalf Of nux
Sent: Tuesday, March 07, 2017 11:05 AM
To: perforce-user@perforce.com
Subject: Re: [p4] Reduce HD space

Posted on behalf of forum user 'nux'.

I think the quickest way here is to remove the whole server and reinstall and
set the new typemap to -S3 before adding anything new.
From what I can see in Obliterate, you cannot remove every rev lower then
three
from(and including latest) without checking each file before. I wouldn't
want to remove 3/3, but want to remove 3/6, keeping top three. Or am I
wrong?



--
Please click here to see the post in its original format:
  http://forums.perfor...reduce-hd-space
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#14 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 144 posts
  • LocationSan Francisco, CA

Posted 07 March 2017 - 05:50 PM

"Perforce would happily obliterate anything. :)"

Please, I can't reiterate enough how important it is to keep this in mind. Never, ever run 'p4 obliterate -y' without running it without the '-y' first, then check your work line by line before you pull that trigger.

And have backups. Lots of backups.

Obliterate is extremely destructive.

And I wouldn't bother rebuilding the whole server. It's easy enough to, say, 'p4 obliterate //some/file.txt#1,#3' do just ditch the first three revisions. No problem there.
-Matt Janulewicz
Staff SCM Engineer, Perforce Administrator
Dolby Laboratories, Inc.
1275 Market St.
San Francisco, CA 94103, USA
majanu@dolby.com

#15 Domenic

Domenic

    Advanced Member

  • Members
  • PipPipPip
  • 93 posts

Posted 07 March 2017 - 06:46 PM

I haven't tested this but in theory I think it should work :) Caveat: we don't use +S.

It is possible to retype the existing files already checked in via the 'p4 retype' command. The command is unsupported "because it cannot safely run concurrently with other 'p4 retype', 'p4 snap', or 'p4 obliterate' commands." If you're on a recent version of p4d there is some locking that tries to maintain consistency (more info at http://answers.perfo...rticles/KB/3163) so I'm not sure if that warning is still applicable or not.

Anyhow, the steps you'd take would be to run "p4 retype +S3 //depot/....uasset" and "p4 retype +S3 //depot/....umap" then re-run the commands to add the "-l" (that's an L) flag (e.g. 'p4 retype -l +S3 <path>". More info on the command is at 'p4 help retype'.

What that should do is retype all the *.uasset and *.umap files so that previously checked in versions are also +S3. Then, when the next version gets checked in the server should purge all the versions so that only 3 are kept. That would save you from needing to figure out yourself "what are the last 3 revisions of this file?" then obliterating everything else. It does assume the files will be updated and checked in again at some point, so if there are old *.uasset or *.umap files with 4+ revisions that aren't being modified then you'd need to consider obliterating those files.

Note: because 'retype' is an under-the-covers action, you'll also want to run a 'p4 sync -f //depot/....uasset' and 'p4 sync -f //depot/....umap' to make sure each client has the right file type. If you don't, the next time a client checks the file out it will be without the +S3 modifier, so when they check it in the file will revert to whatever its current type is (binary? binary+l?) and you'll wonder what the heck happened :)

#16 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 764 posts

Posted 07 March 2017 - 07:07 PM

IIRC retyping to +S doesn't always cause retroactive purging in exactly the way you might like -- I don't think submit makes the assumption that the number of kept revisions may have changed from underneath it (but to the extent it errs it'll always err on the side of keeping more revs).  But I might just be remembering weirder cases where you change from +S to default to +S4 to +S2 to default... retyping everything to have the same storage type contiguously might work better.  :)

#17 nux

nux

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 09 March 2017 - 02:27 AM

Thanks for all the input.
Quickest and easiest way was to rebuild the whole thing, because it just took about 15 min. Compared to go through all files (~3k) and find the once that has more than 3 revs and remove that. But this is a test stage and nothing would be lost in the previous revs. But I will keep all this in mind for the future.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users