Sometimes I feel lucky that I’m a geek. Most days, actually, being that I have to deal with computers on a daily basis. Today is no exception. Digital photographers either have to know their computers, or pay big bucks to someone to know computers for them. I think that’s why film photographers often resist digital so much. The learning curve and amount of reliance upon others can be overwhelming and intimidating.
Anyway, today I solved a major problem, and I hope it will stay solved for a few years at least. If you’re not interested in geeky stuff, then stop reading now. This entry is partly for my own notes, and partly to help my geeky digital photography compatriots when they run into the same problems I’m having.
Up until now I’ve been using portable hard drives for backing up my photos. If I don’t anticipate needing a set of photos in the near future, I remove them my desktop computer so I keep enough space free. That means that the only copy of these photos is now on the backup drive. But, I was starting to feel a bit nervous about this, because I know that hard drives sometimes fail, often without notice. Losing 300GB of photos at once was a scary thought. Also, I had to keep upgrading to bigger backup hard drives, because I was always running out of room. So, I had a couple of options:
- I could start backing up photos onto DVDs.
- I could get yet another big backup hard drive.
- I could just delete photos I don’t think I’ll ever need again.
- I could get a NAS (network attached storage) which is a big disk that you plug into the network.
- Pre-built NAS
- Homemade NAS
Since I’m too lazy for option 1, tired of option 2, and option 3 is just out of the question, I was stuck with option 4. I had a feeling that I’d be going with option 4 eventually, which is why I upgraded my network to Gigabit Ethernet when I got back from China. It was necessary, since my existing 100Mbps network switches had inexplicably died while I was in China. It also wasn’t as expensive as I thought it would be, since I found some reconditioned Gigabit Ethernet switches for $25 each. So, for fifty bucks I’d made my home network ten times faster. Sweet!
For option 4, I wanted to have at least 1TB (1000 gigabytes) of hard drive space available for my photos. I figured that would hold me for a couple of years at least. The cheapest pre-made 1TB NAS options are about $500. That’s pretty reasonable, but the performance of these devices in benchmarks seems kind of questionable, topping out at about 12-15MB/second. That’s not fast enough for photos, and barely touches the bandwidth of Gigabit Ethernet. The other problem is that these devices are only single disks, which means they’d be subject to failure and loss just like my regular portable hard drives.
To solve the problem of having single disks, I could get an array, which means there are multiple disks inside one box backing each other up. The simplest arrays are called RAID 1, which means there are two disks that mirror each other. If one disk dies, you have an identical copy to back it up. The problem with that is that if you want 1TB, you have to get a device that has 2 x 1TB disks in it. That pushes the price up to the $800+ dollar range. Also, you can’t expand that array. It’s stuck at 1TB forever.
To solve that problem, I could get a device that uses RAID 5. In RAID 5, you use a number of smaller disks. The contents will be duplicated across a minimum of three disks, with a portion of space allocated to information necessary to rebuild the array in case one of the three disks fails. This means that if you put three disks into a RAID 5 array, you’ll get the equivalent of two disks worth of space. This seems like a bit of a drawback, but it pays off if you ever want to expand the array. To expand, you just need to add another disk, and the array will grow to use that new disk’s space. Also, since 500GB drives are about a third of the price of 1TB drives, it actually costs the same to buy three 500GB drives to build a 1TB RAID 5 array than to get a single 1TB drive. Make sense?
So, it looked like RAID 5 was the way to go. Commercial pre-built RAID 5 NAS devices are expensive though. You’re looking at a minimum of $1000 for 1TB of space, and the price spikes significantly if you want it to be fast too. Curiously, on some of the benchmark sites that show the performance of various NAS devices, I saw “Home-made DIY NAS” consistently showing up near the top of the benchmarks. I’m a do-it-yourself kind of guy, so this was intriguing. I started pricing out the parts necessary to cobble together a Linux-based NAS. I already had an old computer case handy, a Gigabit Ethernet PCI card, and some memory. I am also already running a Linux server at home to store my movies and music. I could steal parts from that and replace it with my new NAS server. The things I ended up buying are:
- 512MB of memory
- AMD Sempron 3800+ CPU
- El-cheapo Asrock motherboard with 4 x SATA ports on it
- A good-quality Thermaltake 460W power supply
- 3 x 500GB Western Digital SATA hard drives
These parts came to about $470 before taxes. I installed all of this stuff into the old case. I added a cooling fan to keep the drives from overheating. It’s ugly as sin. Probably uglier, since I actually quite like sin:
The pink foam is just there to block the air from going out the front again after the fan blows it in. The proper path for air is for it to go in the front, over the hard drives, and then out the back. It works too. After I installed the fan in the front, the temperature of the hard drives went from an alarming 50C to a comfortable 30C.
Unfortunately, I couldn’t just upgrade my old Linux server running Redhat Fedora 4 to Fedora 8. That would have been my preferred route, since my old server was running perfectly and it would have saved me a lot of configuration. However, it gave me an opportunity to see how Linux installs these days. It turned out to be painless. The new Redhat installer (Anaconda) asks a few simple questions about what packages and services you want to run, and then completely ignores your choices and installs a standard setup of Fedora 8. Oh well, at least it was painless.
After installing the base Fedora 8 packages on the 160GB hard drive scavenged from my old server, I installed Webmin, which allows you to configure almost anything on a Linux server through your web browser. It makes even complicated jobs (like configuring a RAID 5 array) simple. I hooked up my three new 500GB drives. I used Webmin to partition them and set them up as a RAID 5 array. Voila!
Copying all my photos from my backup hard drive to the new RAID array took a couple of hours, but went without a hitch.
I used IOMeter and some advice from the Internet to tweak and test the speed. My new NAS is fast. Transferring 4MB files, it tops out around 39MB/second, which is in the top range of Gigabit Ethernet NAS devices, even those costing thousands of dollars. Not bad, considering the total cost of my new parts to build this server was less than $500.
brilliant! maybe i can use one of our old boxes. now if i could figure out how to hide the computer entirely, i’d be perfectly happy. thank you for the write up; its something to definitely do in 2008. too many close calls…
how do you sort your photos? i always have a headache on how to sort them so that i can find them later. i guess if its all in a raid i wouldn’t worry about that. hehe.
less than $500 is awesome.
thanks! 🙂
well, i don’t have a special method for sorting. i just create new folders for each shoot with the name of the person/thing/place i’m shooting. just last night i put them in those folders you see in the screenshot, for headshots, commercial, travel, etc. that’s easy enough for me to find.
wow that sounds like a lot of work! 🙂
can you explain to me why you feel safer with putting your phones on those huge harddrives, do they fail less frequently than the ones you used before?
Wow .. this is a very sweet setup. I just bought myself a terrabite of raid 1 storage as an external hard drive. I think in the future this raid 5 array will be better, for sure!
Ps. Currently investigating purchasing a barebones media system to store all my movies, music and whatnots 😉
Happy Christmas and Geek Days!
hesi the RAID array is made up of three hard drives, and they make backups of each other constantly, so if one hard drive fails, you can just replace it and the data rebuilds itself. it’s much safer than having just one big hard drive, because if that single drive fails, you lose everything 🙂