Jump to content


How to handle huge binaries that are needed multiple times?

symbolic links binaries Unreal Engine

  • Please log in to reply
3 replies to this topic

#1 SirHubert

SirHubert

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 02 March 2019 - 09:12 PM

Hey all,

I'm new to version control in general but I have to start using it since my project is getting too big to fail. Here is the thing: I am working in Unreal Engine and I have two projects that are similar. One is a server project and one is the graphical client. The way Unreal works is that you have to have a "content" directory in your project under which all the binary assets are. No way around that.

I have a set of static assets that I bought from their marketplace and that is about 100GB. It's problematic to have a copy of these assets in each project, possibly even in different codelines that I may check out. It's 100GB each in the workstation not to mention the space wasted on the Perforce server. So I tried using Windows symbolic links to link each project to one physical directory. That works so far and it worked in  Unreal Editor too until I enabled Perforce in which case Unreal looks at these not as links but as real directories. Now it nags me about checking in those binaries into perforce and when I give in it checks them into each perforce depot.

Actually I have one depot that holds both client and server Unreal projects because they are interconnected, no way I could make change to one of them and not need to make changes to the other.

The problem really is that neither Unreal nor Perforce can realize that these links go to the same directory and treat this as one only. I would check in the 100GB once if necessary but would rather avoid checking them in for each Unreal project.

I hope I described my problem clear enough.

Now what would be the best way to handle such a situation in Perforce? Is there a way around checking in the binaries multiple times at all?

Thanks

#2 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 917 posts

Posted 07 March 2019 - 12:41 AM

On the server side, if you use Perforce commands to copy them ("p4 copy", "p4 integrate", "p4 populate", etc) they're deduplicated ("lazy copied") on the back end.  (If you copy them locally and add them independently, you get redundant copies on the back end.  As a general rule, always try to use the native Perforce command to do a thing on the server rather than doing it locally and submitting it as a change that's unconnected to anything else; it'll let the server be smarter about reflecting your intent and optimizing its own recordkeeping.)

On the client side, only the files mapped by your current client view are ever in the workspace.  If you set up your different projects as different streams and "switch" between them, then the project-specific files will get shuffled around but the binaries will stay where they are (if they're in the same stream-relative paths).  Bonus: if you use streams you don't even have to branch the files because you can use "import" paths to import a single set of files into an arbitrary number of streams!

#3 SirHubert

SirHubert

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 08 March 2019 - 05:12 AM

Thanks, it's not quite what I was hoping for but I didn't expect there to be any links in perforce that allow the setup I have in mind. I guess in the end it will be multiple copies on the server and the client, I expected that much

#4 Sambwise

Sambwise

    Advanced Member

  • Members
  • PipPipPip
  • 917 posts

Posted 08 March 2019 - 09:01 AM

View PostSirHubert, on 08 March 2019 - 05:12 AM, said:

Thanks, it's not quite what I was hoping for but I didn't expect there to be any links in perforce that allow the setup I have in mind. I guess in the end it will be multiple copies on the server and the client, I expected that much

Per my previous post, that doesn't need to be the case, but ultimately you should set it up the way that makes sense for you.  :)





Also tagged with one or more of these keywords: symbolic links, binaries, Unreal Engine

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users