Blog Archive

Backing up Lots of Data Remotely - Dec 12, 2009 6:41AM - By: Chad Halvorson

Recently we started backing up a few of our workstations using BackBlaze. So far it works great; it consistently backs up all our project files to a nice, cozy off-site location.

Our company file server is another story. We produce a lot of video programs, website designs, code, graphics, animations, etc. So our storage needs are huge. With our file server being Linux, finding an off-site backup service that didn't break the bank was virtually impossible.

At this point we have roughly 3TB of data on our file server. We needed a way to get this data backed up to a safe location in an automated, daily fashion.

Weighing all our options we chose to build a second Linux file server and run it at a different location. After that it was fairly straight forward. We used "rsync", a free Linux based remote file copying tool.

Configuration took some thought. We chose to sync all files 3GB or less to the remote server. That would cover all project files and assets, however, most rendered and finalized videos would be left out. We were OK with that since all final renders and videos are backed up to disc and/or tape any way.

The initial backup took the longest. We closely monitored progress over the course of about 5 days. Yup, it only took 5 days to backup just under 3TB over a 10Mbit upload connection.

So far the results have been great. The servers sync every night, updating modified files, backing up new files and removing deleted files. The whole process now only takes 30 -60 minutes each day depending on how much data we transferred that day.

The upfront cost comes with the purchase of the second server. Ongoing costs are negligible; —additional storage when needed, broadband internet connections at each location, etc.