Smarter ideas worth writing about.

Git Large File Storage

Being a Unity3D developer, I am used to some of the hindrances to Unity including compression qualities, native embedding issues and the one related to this post, large files. Our GIT clients were also not impressed with these files.

For our purposes, we usually use a Gitlab setup in house, but for the purpose of this system, we tested it with Github and SourceTree.

The Issue

As my scenes and assets grew, so did the file sizes. 3D assets, movies files, large 360 photos. All of these files are multiple megabytes, or gigabytes. As I would push to our GitLab, I would come across broken pipes, timeout issues and recently, failure errors due to file size.

After doing a bit of research, I came across documentation that GitHub and GitLab would not accept general files over 100MB. Most of our files reach well over that.

The Solution

Back in April of 2015, GitHub introduced a new, open sourced, tool called Git LFS (Large File Storage). It was developed specifically for large files like audio, video, datasets and graphics that exceed the file maximum size.

Instead of forcing the file up onto the server or simply bypassing the max setting, Git LFS actually pushes the large file to another server (large file storage) and creates a text file that points to it on that server. The text file is then pushed to your GIT as a placeholder. Simple!

How To

The process of getting these files to the Large File Storage is fairly simple to setup. I am going to use SourceTree to show you how to do it because it has Git LFS built right in.

For your Git repository to push to the LFS, it needs to know which files are going to get over 100MB or ones that are already over 100MB. Luckily, you don’t need to provide the exact file name, but just the file extension it is associated with (IE, a *.PSD file).

So here we go, first let’s open up SourceTree and pick a repo that has large files in it. Go to the top menu bar and select Repository > Git LFS > Initialize Repository.

Then this popup shows up, click Start using Git LFS.

Next, add the file extensions you think, or you know, are going to be large files. When it comes to Unity, I know there are four files that are usually pretty big, *.a, *.asset, *.resS, and *.assetbundle. The first three are generated based on your scenes, scene links and the assets associated with those scenes. AssetBundles are like ZIP files with assets compressed inside them making them huge! Once in game, Unity is able to decompress and use those assets on the fly, making them a great resource for assets that need to be downloaded (IE. Large skyboxes, high poly models).

Once you hit ‘Ok’, SourceTree will create, or add to, a file called “.gitattributes”. Be sure to commit and push that file up to GitHub before pushing the rest of your files.

That's It!

Now you can commit your large files and GitHub will automatically track which ones get sent over to LFS.

Quick Note

Git LFS will also warn you when it notices a file that may be too large for the GIT. It gives you a list of files and you can add their extensions by going to Repository > Git LFS > Track / Untrack Files.


About The Author

App Dev Consultant

Matt is a mobile developer in Cardinal's Tampa office with experience in iOS and Android native development. He also has extensive experience in game development with Unity3D and is leading Cardinal's VR/AR application development. Matt will continue to develop the VR/AR space for Cardinal as the market grows.