Table of Contents
Expanding Data and Evolving Backups
Thanks to our travels and my obsessive photo habits, we have about 4TB of photos and related data (catalogs, backups, GPS tracks, rendered galleries, etc...) accumulated over the years, and the rate of accumulation may vary, but is generally increasing. My first digital camera was a 3 megapixel Nikon Coolpix in 2002 producing JPEG files, and my current camera is an Olympus OM-D EM1 Mark II, which can produce 20 megapixel RAW files at a rate of up to 60 frames per second.
Data Growth
Photo Gigabytes per Year
2015 was an outlier because that year we traveled to both New Zealand and Africa, but Iceland set the single trip record so far at nearly 1/2 a terabyte, with the Galapagos Islands not far behind.
Over the years, I've rotated through a few different means of backing up those photos redundantly. Once upon a time it was a daisy-chain of FireWire 400 drives. Then, for a few years I had a box running Windows Home Server, which at the time used a software-based file replication system that let you use drives with heterogenous capacities, which was very useful for upgrading capacity piecemeal. The HP MediaSmart Server was an adorable micro-tower with four drive bays that sat on a shelf and worked really well, until it didn't.
Most recently it's been back to the daisy-chain of drives. Thanks to ever-increasing hard disk densities and the transfer speeds of Thunderbolt 2, that's been a reasonable approach for quite a while. My backup strategy was:
- Primary storage on iMac
- rsync to clone storage onto primary backup drive
- rsync to periodically clone primary backup drive onto secondary drive (kept offsite)
- rclone to sync data to Backblaze B2 for a cloud archive
When the size of our photo archive exceeded the storage capacity of my photo editing iMac, that got a bit more complicated, as now the primary storage location for older photo sets had to be an external drive.
When I finally overflowed the capacity of that drive, I knew it was time for something else.
Enter Synology
My previous generation of 6TB Thunderbolt drives sitting on top of the new Synology NAS.
Long ago, I was hesistant to go for a NAS (Network Attached Storage) device when network speeds were slower, especially for photo editing. Several things changed my mind, chief amongst them being gigabit ethernet and 802.11ac WiFi, which I've reliably clocked reaching gigabit speeds in our apartment.
I'm still not editing photos directly from a NAS, but it's now realistic to swap entire multi-hundred-GB photo sets on and off the internal drive of my iMac.
So, about a month ago I purchased a Synology DiskStation DS1618+. One of the main attractions was that, not only is it a RAID-enabled storage device, it's also a quite capable home server. It can serve as a Time Machine backup for our Macs, serve media via Plex, provide basic, yet useful, web access to our terabytes of photos, and even run virtual machines or docker containers.
So now my backup workflow is:
- Download photos from memory card to workstation
- Sync workstation to Synology
- Synology automatically syncs to Backblaze B2 every night for off-prem backup
I'm currently still running a manually triggered sync to keep an additional copy of all photos on a standalone 6TB external hard drive (RAID protects against drive failure - not accidental deletion!), but will be setting that up as an automated periodic process soon as well.
It certainly wasn't cheap, but I've been able to consolidate and centralize all my data (not just photos, but computer backups, personal documents, source code, etc...) with redundant RAID storage, automated offsite backups, and serving media and remote access.
More to come soon on how I've set up Graphite and Tessera for monitoring it.