Jump to content


How to set up perforce for an Unreal Engine movie project

film perforce unreal shortfilm colaborate

  • Please log in to reply
7 replies to this topic

#1 insertmesh

insertmesh

    Advanced Member

  • Members
  • PipPipPip
  • 35 posts

Posted 24 June 2020 - 11:28 AM

I´ve recently contacted support with these questions, so I wanna make sure others like me, with little to now coding or admin experience are able to find these answers, wether they come from support directly or from other community members. Once I have all open questions answered, I will try to make this into a proper guide.

Disclaimer:
I want to use the P4V client for all tasks if possible, as I´m more familiar with a GUI than a commandline environment.
If something is not possible with the P4V client and needs to be done with the command line, please mention that
For the remote server I realise I will have to use the commandline client anyways.

Here is an overview of my plans:

I have a movie project I want to make in Unreal, so there is NO coding involved, I just want to version the unreal files associated with it.
I currently collaborate with a small team online and I´m still planning on how to set it all up, before creating any content.
Since a lot of the files involved will be large binaries and server space storage is expensive AND I´m still going to contribute at least 50% of the content all by myself with no need for collaboration, I´m trying to figure out how to split the project up into smaller parts that can be either put on a remote server for collaboration or purely stored locally on my workstation, but still version controlled by perforce.

I can split my project up logically into shots, where each shot would be a self contained unreal project with a single level in it.
Each Level would contain sublevels for particular aspects of the shot as described here:
https://docs.unreale...vels/index.html
So, for each shot I know that will need collaboration across my team, they could still work on separate aspect of it at the same time.

As I see it, I have two options:

Plan A involves setting up both a local and an online server and split up the project in many smaller projects, so I can put them each in their own place.
This is a lot more complicated to set up and maintain, but gives me most control of the project size on a remote server.

Plan B involves setting up just one remote server and using smart management via perforce to reduce server storage space needed.
This is a lot easier to setup, but more prone to explode server space.


Plan A:

1. Create "Master" Unreal Project (local).
Only I (director) am going to work on that.
Once a shot/environment is finished either online (by another user) or locally (by me) it is going to be migrated through unreal into the master project.
The "Master" project serves as a way to make postprocessing and settings consistent through all shots, and to adjust details as needed.

2. Create a core project (server). This is an unreal project that contains core assets needed in many or all shots/levels (main character, master materials etc). These have to be migrated manually into each shot they are needed in and will NOT be referenced otherwise. Whenever they change, they will need to be replaced within the shots that they were migrated to.

3. One project per shot (local OR server).
For each shot there will be a separate project containing all assets needed for that shot. Those can either reside on the local server, be put to a remote server or be created on a remote server, depending on the level of collaboration needed.


Questions:

1. Should I create a server using localhost:1666 for local only content (my master projects and the locally stored shot projects), as described here:

https://www.perforce...ient-windows-10

While setting up P4V for the first time, it seemed to me that when I create a workspace and a depot, they are stored in the same place on my workstation. Is this how its supposed to be?

OR

Should I create a personal server for that? Whats the difference between these two?


2. Do I need to create one separate depot for each separate unreal project?

Lets say:

MasterDepot
CoreDepot
shot010Depot
shot020Depot

3. Should I use ONE workspace for all lcoal stored projects, and ONE workspace for all online stored projects (with subfolders containing each separate project), or ONE workspace per project/depot?

Workspace A (local):

Projects_LOCAL\shot_010
                           \shot_020

Workspace B (online):

Projects_ONLINE\shot_030
                             \shot_040

OR

Workspace A (local):
shot_010\

Workspace B (local):
shot_020\

Workspace C (server):
shot_030\

Workspace D (server):
shot_040

4. In which order do I need to create these? Set up the remote server first or the local server first? First create all depots on my local server and my remote server, then create the workspaces for them?

5. Lets say I created a project on my local server, but now I need to use it on the remote server to collaborate.
Can I move it via the P4V client, or can this only be done through commandline and how would I do it?

6. In this workflow I might regularily switch between local and online projects.
Can I just open the unreal project and connect it to either the local or the remote server ONCE and then any subsequent time I open it, it stays that way?
Or do I need to do anything else  (like change some p4V client settings), anytime I switch between a locally stored project and a remote server stored project?

Plan B:

1. Create a single Master project, that contains a folder with core assets and one level per shot. Whenever the core assets get updated, they get updated for every level they are used in as well.
They are not to be changed by anyone but me (director).
2. Put this master project on an online server.
3. Set up p4ignore to ignore stuff like _builtData.uasset, that can be locally created by each team member.
4. Set up p4typemap with filetypemodifier +S3 for all .uassets, so only 3 revisions are stored on the server.
Its rarely necessary to go back on any unreal assets more often than that.
Levels (saved as "*.umap", where most of the work is being done, can have more revisions, as they don´t take up that much space.
5. Once a shot is finished, migrate that asset, save it on a local drive and obliterate from the server to save storage space.
6. Create a separate master project on a separate local drive, where the finished shots are being migrated to.

Questions:

1. Is it possible for other users to just pull the level they are working on, instead of the whole master project?
2. Can this be done from within Unreal, or does it have to be set up with P4V?
3. Are there other things I need to consider with this setup?

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1192 posts

Posted 24 June 2020 - 03:07 PM

Quick note:

Quote


This is an unreal project that contains core assets needed in many or all shots/levels (main character, master materials etc). These have to be migrated manually into each shot they are needed in and will NOT be referenced otherwise.


If "migrating" involves making a copy, make sure to use p4 commands (p4 copy, p4 integrate, whichever) to do this rather than copying them locally!  If you use Perforce to make a copy, it takes up no extra server space (because the server knows that one thing is a copy of the other), AND you can easily figure out via the Perforce history whether each shot is "up to date".  If you make copies manually, you're paying for extra space for each copy.

If you're using a remote/personal server setup, you'll definitely want to set up your remote spec so you can just "p4 fetch" updated copies of the core assets into your personal server.  (You can do this with a "local" server also; a personal server will automatically set up a remote spec if you create it via "p4 clone" but this is just a configuration step that you can easily replicate with any server.)

My best answers to your questions (I'll skip the ones that pertain to P4V and Unreal since I'm not sure how those work):

A1) Whichever is easier.  The only difference between a "personal" server that you create via "p4 init" and a regular server that you run on your local machine is that the personal server is automatically configured for you (when you run "p4 init" it creates a stream depot, a mainline stream, a workspace, and a P4CONFIG file that spawns a server instance for each command via the P4PORT=rsh... syntax) whereas if you just run the p4d installer and set up a service, you get a normal p4d daemon listening on a port and the configuration is up to you.  You can easily set up a p4d service to serve the content from your "p4 init"'d personal server after the fact (just set the service's P4ROOT to point at the .p4root created by the "p4 init"), and you can also very easily set up an existing "normal" server to act like a personal server.  There's not a fundamental difference between the two.

When you create a personal server via "p4 init", it puts it all in one root directory for ease of cleanup (the idea is to make it behave more like git where everything is in one place and all the versioning stuff is "hidden" in a subfolder rather than in a central location), with the server root (.p4root) carefully excluded from the workspace via the stream definition.  If you're setting things up from scratch, it's better to just have these be two distinct directories so you don't need to worry about getting that right.  Under no circumstances do you want the workspace to actually include the server files in its view because then every time you add a file, you just get an other copy of it in the server folder within your workspace, which means the next time you add files you'll add that copy, which means you get another copy... :)

A2) That seems like a useful way to organize it, but it's not required.  You could also just have each one be a different directory under a single top level depot, or have some mix (have a "core" depot for the core stuff and a "shots" depot with subfolders for the individual shots).  I'm assuming here that your versioning strategy doesn't include branching; if you're branching these projects, then the depot organization should be guided by whether they branch/release on the same schedule or independently.

A3) Again, this is "whatever is easiest".  If you're working on all these things at the same time, and especially if there are dependencies between them (i.e. you need to integrate things from the core project to the shots projects), IMO it's easier to have it all in one workspace rather than having to switch around a lot.

A4) If they're independent sets of files, it doesn't matter what you set up first.  If your local server depends on core assets in your remote server, set up the remote server first, and then you can potentially use "p4 clone" to more easily set up your local server with its own copy of the core assets (again, I'm still fuzzy on how you're managing the association between the core assets and the dependent shots, so maybe this is inapplicable).

B1) Yes; this is why client views are great!  Each user can be set up with a client view that only maps the specific stuff they're working on.  You can also use permissions ("p4 protect") to strongly enforce that certain users can't even see the projects they're supposed to be working on.

#3 insertmesh

insertmesh

    Advanced Member

  • Members
  • PipPipPip
  • 35 posts

Posted 24 June 2020 - 04:48 PM

Thanks again for answering these.

The whole workplace/depot concept is still a bit foggy to me, but probably because I didn´t pay much attention when I initially installed P4V, so I don´t actually know to which local directory its root is pointing to.
With a local workspace/depot connection I get that its just a mapping relation between a remote server folder structure (depot) and a local folder structure (workplace).

As I´ve gotten some more replies from tech support, its looking more like I´m going to be able to go with Plan B.
I found some providers, that offer much more storage space than the ones mentioned in the tutorials I watched.
With several hundred GBs at my disposal, I won´t have to worry as much about storage space and can just follow a remote server setup containing a complete Unreal project.

But I´d like to know a bit more about the client views, now that you´ve mentioned them...

1. Are client views set up per user?
2. If I don´t create a client view for specific users...I assume they can just see the whole depot and pull/push anything they like?
3. If I create a client view following this process:
https://www.perforce.../p4_client.html

...does that mean the user only gets to see what is specified in that client view and nothing else?
4. If so...what if the user adds folders to his local workspace...do I need to manually add these to his clientview, so he can submit them to the online depot?
5. Do I actually need to know where the users local workspace folder resides, so I can set up the client view? Or can I set this up with wildcards, so he just sets up the client (or connects to source control in Unreal) and can pick which drives or folder to put his workspace in and then the client view I create is just in reference to that?
I mean, technically I could just ask them, but those are more or less random people I know over the internet, not my employees, so asking that kind of information from them seems a bit invasive...

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1192 posts

Posted 24 June 2020 - 05:28 PM

View Postinsertmesh, on 24 June 2020 - 04:48 PM, said:

1. Are client views set up per user?

Yes, potentially with multiple per user.  Multiple users generally don't share one client spec, but one user can easily have multiple client specs.

Quote

2. If I don´t create a client view for specific users...I assume they can just see the whole depot and pull/push anything they like?

Even if you set up a client view for each user, nothing stops them from modifying the view, or creating more client specs, etc.  To actually protect specific parts of the depot you need to use "p4 protect", which acts as another layer of filtering on everything each user sees/does (and it can only be configured by the superuser, i.e. you, unlike the client views which are largely up to each user to configure for themselves).

Quote

3. If I create a client view following this process:
https://www.perforce.../p4_client.html

...does that mean the user only gets to see what is specified in that client view and nothing else?

For the purposes of commands which relate to the client workspace, yes (so anything involving syncing, editing, or submitting files).  Some commands can operate on arbitrary depot paths, and these are not limited by the client view.

Quote

4. If so...what if the user adds folders to his local workspace...do I need to manually add these to his clientview, so he can submit them to the online depot?

If the folders are already mapped by the view, then the view does not need to be modified to include them.  Typically you set up the view ahead of time to map everything that you're going to be working on, and it's not necessary to make a lot of adjustments to it after that.

Quote

5. Do I actually need to know where the users local workspace folder resides, so I can set up the client view? Or can I set this up with wildcards, so he just sets up the client (or connects to source control in Unreal) and can pick which drives or folder to put his workspace in and then the client view I create is just in reference to that?
I mean, technically I could just ask them, but those are more or less random people I know over the internet, not my employees, so asking that kind of information from them seems a bit invasive...

The easier way to do this IMO is to use streams and let users create their own client specs based on those streams.  Trying to set up the client specs remotely is possible, but it's a pain for all the reasons you just outlined.  One of the main purposes of streams is to automate all of the parts that you'd want to control centrally (i.e. the view creation) while still leaving the actual client specs under the control of the user (e.g. the Root is set based on the working directory at the time the user creates it, but the View is set up based on the stream definition that you've set up ahead of time).

#5 insertmesh

insertmesh

    Advanced Member

  • Members
  • PipPipPip
  • 35 posts

Posted 24 June 2020 - 06:04 PM

Hm. The more you explain it to me, the less sense it makes for me to set up client views.

I haven´t looked into streams yet, as it seemed to have to do with coding, branching and merging code, which doesn´t apply to our purpose.

Just glancing over the docs, it seems this might just overcomplicate things for me.
I don´t necessarily need to have a neat simplified view for other users, that only show what they are supposed to be working on.

It will always be a small team, so I can just...tell them what they need to sync...:)
And use "p4 protect" to lock things only I should be working on.

I´m still a bit foggy about how to separate my personal server location from my local workspace location but I hope that clears up, when I start again from scratch, because I didn´t pay much attention when setting things up as localhost:1666 for the first time.

#6 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1192 posts

Posted 24 June 2020 - 06:16 PM

View Postinsertmesh, on 24 June 2020 - 06:04 PM, said:

I don´t necessarily need to have a neat simplified view for other users, that only show what they are supposed to be working on.

It will always be a small team, so I can just...tell them what they need to sync... :)

That's pretty much the default mode if you don't set up streams -- you have a single default depot called "depot", and by default every client view maps that depot to the client root.  Easy!  Whatever folders you create in your workspace are exactly what will be added to the depot, and whatever is in the depot is exactly what gets synced to your workspace.

Quote

I´m still a bit foggy about how to separate my personal server location from my local workspace location but I hope that clears up, when I start again from scratch, because I didn´t pay much attention when setting things up as localhost:1666 for the first time.

The server's files live in the "server root", aka P4ROOT.  Exactly how you configure this depends on how you set up the server; it's a little different between Windows and Linux.  When you go through the installation steps just pay attention to anything that mentions either a "server root" or "P4ROOT" and be aware that that's where all the server stuff will live.  The "client root" (which you define when you run "p4 client" for the first time) should be some whole separate directory, and if the server isn't on localhost it'll in fact be on a whole separate machine.

Last time I'll pitch this, I promise: every question you've asked so far, as well as the next twenty or so questions you're likely to ask over the next week or month or so, is something you'd already know the answer to if you went through a good one-hour tutorial.  I don't mind answering questions, I just harp on this because I value your time.  ;)

#7 insertmesh

insertmesh

    Advanced Member

  • Members
  • PipPipPip
  • 35 posts

Posted 24 June 2020 - 07:04 PM

Well, I value your time as well...
its probably worth more than mine though, as IT specialists or admins usually get paid way better than artists like me...:)

That being said, I think you cleared all the biggest foggy areas for now.
Can you recommend a good one hour tutorial?
I watched several that were more focussed on P4V (which doesn´t explain why are things set up a certain way or are more geared towards coding projects) or Unreal integration (which is actually pretty straight forward).

#8 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 1192 posts

Posted 24 June 2020 - 07:40 PM

Step through the tutorial in the official docs here:  https://www.perforce...r.tutorial.html

I'm guessing it'll only take about an hour to go through, and that's allowing some wiggle room for false starts.  There used to be a version of this on the front page of the Perforce website called "The Ten Minute Demo" and it really did only take ten minutes to go through the whole process of installing p4, setting up a server, setting up the client, doing all the basic stuff (adding some files, editing them, etc), and then cleaning up at the end.  It's the sort of thing where you can spend an hour listening to someone talk about it or five seconds to just see it in action and say "oh, that's how that works".

I went through basically the same process when I first learned Perforce about twenty years ago -- I started off by spending the better part of a week reading the entire manual cover to cover and mostly just ended up confused (because most of it wasn't on the critical path of basic usage and most of it also assumed that you were already familiar with all the basics), and then I found the demo, spent the ten minutes on it, and said "oh" and felt silly for having largely wasted a week.  So I always urge people to do the easy thing first.  :)

The existing visual tools like P4V are great when you already know what's going on, but they can make it hard to learn the basic concepts because a lot of the presentation is very visually compressed or abstracted away.  E.g. it's very hard IMO to set up a client view in P4V because the interface tries to avoid ever showing you a "mapping", even though your client view mapping is something that's absolutely fundamental to about 90% of what you use Perforce for.  The #1 confusion point for new P4V users is always figuring out why their workspace files didn't get added to the spot in the depot they wanted them to go to (why is there an extra "depot" in this path?  why am I getting a "not in client view" error when I try to add this file?  etc), and I think that has a lot to do with the fact that P4V tends to hide that very important configuration step from a first time user.





Also tagged with one or more of these keywords: film, perforce, unreal, shortfilm, colaborate

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users