The 2014 Data Storage Thread

Joined
Sep 2, 2006
Messages
15,602
So I've taken stock of the state of my personal computers and their hard drive storage over the weekend and decided it was quite bad. I have three different USB keys with files backed up, a broken-down computer that can only be booted with a Linux CD to access the files on the hard drive, and three external hard drives acquired over the years (1x750 GB, 2x1 TB). To make sure I'm not the next guy running into the Questions thread with a busted computer and no way of recovering his files, I want to invest a bit of coin over the next year here to get an effective and modern data storage/backup system running. However, I have no idea where to start looking.

The idea behind this thread is to discuss some of the options available (all hard drives, online cloud storage, personal cloud storage), drive capacities, and other recommendations for someone who wanted to start putting together a system like this. Do people prefer to use software to auto-back up files, or save directly to their folders on cloud storage or external hard drives? What kind of computer knowledge is required to set up something like a personal cloud, and are those a good investment? Or are they too new and buggy to be relied upon?
 
I only do a non-regular backup on an external harddrive, so I'm not really better in these regards :mischief:...but I guess you're looking like something like a NAS http://en.wikipedia.org/wiki/Network-attached_storage .
It's basically a harddrive with a network connection.
While it's often used to stream data, I think the second most used option is as a backup for a network.
If your mainboard allows it (should, if it's not ancient), then you should probably also invest money in a second harddrive and set it up as Raid 1 http://en.wikipedia.org/wiki/RAID#RAID_1 .
 
So WD has started offering a 6/8 TB external hard drive block, and it has a RAID 1 option built in so it converts into 2x 3/4 TB drives. The idea being (obviously) that whatever you save on this massive 3/4 TB drive is auto-backed up on the other drive, and so long as you don't throw magnets at the thing you are basically secure and can recover files easily.

I have no idea how well these kinds of products work, though. They weren't on the market when I bought my last external hard drive. Are they reliable? The review on Amazon are pretty mixed.
 
What I do:

All my important data is synced with https://skydrive.live.com/. The full file contents are synced to a couple computers (ones with more space) while the others use smart files, so they've got access to my files but don't store them locally.

I do a full non-incremental system image backup of the OS (and everything else other than music/movies/tv) drive of my primary computer weekly onto an external 2.5" USB3 drive that I leave plugged in. I get about a month's worth of backups this way, entire process takes about 20 minutes.

I've got http://snapraid.sourceforge.net/ SnapRAID on my home server with all my music/movies/tv media. Currently have 11 data drives of various type, so using two parity drives. I run a sync on this weekly. Entire server contents are uploaded to Backblaze.

All my non-media data is stored only on encrypted drives.

Answers to your questions:

What kind of computer knowledge is required to set up something like a personal cloud, and are those a good investment? Or are they too new and buggy to be relied upon?

The main problem I see is lack of upside - you're going to get better reliability, redundancy, speeds, management, etc. from Skydrive, Dropbox, etc. and up to 100GB or so is pretty reasonable in price. The main upside from home cloud storage setups is that you can get large amounts of space more cheaply, but on a typical home internet connection it's going to be brutally slow to actually upload anything from your cloud storage setup.

If you're concerned about privacy, use SpiderOak. (And PM me or come onto #fiftychat for a referral code for 1GB bonus space on signup.)

So WD has started offering a 6/8 TB external hard drive block, and it has a RAID 1 option built in so it converts into 2x 3/4 TB drives. The idea being (obviously) that whatever you save on this massive 3/4 TB drive is auto-backed up on the other drive, and so long as you don't throw magnets at the thing you are basically secure and can recover files easily.

I have no idea how well these kinds of products work, though. They weren't on the market when I bought my last external hard drive. Are they reliable? The review on Amazon are pretty mixed.

RAID is not a backup.

Hard drive reliability tends to not matter enough vs drive price (ie. you come out ahead by getting the cheapest drives available and replacing as necessary) - assume any drive will fail catastrophically at any time and have backups of everything.
 
M-Discs and a fire safe. Get yourself a new M-Disc burner, discs, and a quality fire safe and you're good to go. 1,000 year data life.
 
Zelig, thanks for that link to SpiderOak. I had heard of them before but never really looked into it. I feel welded to Google [Drive, Gmail, etc], but this service seems like a way out. In our situation we have a couple of macs at home, an iPhone, and an Android. We need to sync some text files across all devices, and also contacts. Photos and videos we need to backup are currently handled by TimeCapsule onsite, and I try to do manual backups every couple of months onto a portable external that I keep off-site. Last time I managed to do a backup was in June :sad:

Can we talk about encrypting drives? I don't quite understand the actual details of it. I think I recall hearing that you can't encrypt a TimeCapsule after the fact - it has to be encrypted when you first set it up. So that raises the question "should I wipe my TimeCapsule and start over with encryption?"

I found out about LastPass from a thread in this subforum, and I've been using it for nearly a year. I'm very comfortable with it now, though there are still some shortcomings and glitches. But that's likely due to my own idiocy. I would recommend it for anyone who wants to save and integrate passwords across multiple devices.

Speaking of hardware and cloud encryption, how do the NSA backdoors impact decisions on which services or devices to trust?
 
I'll start with what I do, which isn't quite up to what I'd recommend, but is better than what I had in my first years of CFC membership, which sadly led to accidentally losing some Civ save files.

I have two hard drives, and use one as backup. They're both in the same computer (which is bad, because a bad power surge might fry them both, although the computer is on a surge protector which minimizes that risk). They aren't in RAID - I'd considered it, but have never done it. I back up every 2 months or so on average, which isn't ideal, but is probably still above average. I use my desktop as a main computer, so everything important on my laptop is copied over to my desktop periodically (which means not very often these days as I don't use it very often), and both the desktop's main hard drive and its SSD are backed up to its secondary hard drive.

Software-wise, I use a simple file copier to copy all my files over. I used to use a piece of software called Personal Backup, which was nice when my backup drive was smaller than my main one and I couldn't just back up everything. But the upside of just using a simple copier is that no special software is required to read the backup. It isn't ideal, but it's not complicated.

The good thing about my setup is that I have a drive that's a backup for everything, so I know where to find the backup. And everything that's backed up there is on at least one other disc. So at least I usually know where my files are.

Now, ideally, I'd (a) do backups more regularly, (b) have a backup that isn't connected to a computer except when doing backups, and (c) occasionally back up off-site. Although if I needed the off-site backup, I'd probably have things I was more concerned about than computer files. What's the best way to accomplish those goals, is a good question. I've never used one of those external mass-storage devices like the WD one mentioned in post #3. One question I would have is, how easy is it to swap drives out if one or more of them fail? I bought two Western Digital external drives in late 2011, with mediocre reviews on Amazon (but half the price of any other drive at the time, since prices were very high), used them as internal drives since that's what I really needed but external ones were cheaper, and since then one of them has failed. It wasn't a problem for me since I just bought a Toshiba drive to replace the failed WD one, but judging by how intentionally difficult the WD external enclosures were to open, I'd definitely check whether the 4x3-or-4 TB enclosures are easy to open if need be.

The other nice thing about when the drive failed was that since I had two, it was no big deal. Granted, I would have lost some data had it been the primary drive rather than the backup that failed, but I wouldn't have been hosed.

I'm skeptical of cloud storage for a few reasons. The possibility of the provider closing shop, the higher price over the long term, and the fact that it would take a really long time to back up large quantities of data on typical North American Internet connections. And of course, the privacy questions. SpiderOak is one of the safest choices in that regard from what I've read. The NSA is obviously doing whatever it can to spy on Google (intercepting data between data centers is pretty hardcore determination), so whether Google is really doing the best they can to keep the NSA out or not, I wouldn't want to back up my data to them. SkyDrive is great if you want to, for example, share pictures, but I don't think Microsoft can really be trusted for critical files if you don't want the NSA to see them, either. SpiderOak is American, but unless they have flaws in their encryption (intentional or otherwise), they're a better choice, since they encrypt the data on your computer (and thus before they can see it).

I think the prerequisites for a cloud service being trustworthy depend on how you want to use it. If you're encrypting the data locally before uploading, using something such as TrueCrypt with strong 256-bit or higher encryption, then theoretically any service would work. You have a link to download it from the front page of the New York Times, and it would require enough computing power to crack it that it would be extremely impractical. There are still questions such as whether your credit card information could be trusted with them, but the data itself should be safe, assuming there were no flaws in the cryptography used to protect it.

If you aren't encrypting the data yourself first, it gets more complicated. You don't really want to use a service that doesn't encrypt the data, because then anyone who gets access to their server can read the data. You also don't really want a service that encrypts the data on their servers, because then any front doors in their server will be able to read the data before it's encrypted (as well as anyone who hacks in to their front-facing server that receives the unencrypted data). SpiderOak, and I'm sure a few competitors, encrypt on your computer before uploading, which is preferable. Although even with these services, check and see if there's a password reset feature. If there is, and it allows you to recover your data when used, that's a red flag. That means that your password isn't actually needed to decrypt the data, and that there must be another password that they have that can decrypt it. There's also the question of whether their encryption is trustworthy - SpiderOak's technology is proprietary, so although it's regarded as one of the better options out there, independent cryptographers can't verify that their encryption is as good as they say it is and without backdoors.

As for encrypting drives, there's two categories, software-based and hardware-based. I don't know enough about software-based on Macs to comment on it. Hardware-based drives encrypt everything on them automatically. So even if you have an unencrypted TimeCapsule, if the drive itself is hardware-encrypted, you couldn't access it without unlocking the drive. All the drives I've seen with hardware encryption use AES (Advanced Enryption Standard). AES does not currently have any known security flaws, although the NSA was involved with its creation in the mid-2000's. Some self-encrypting hard drives an SSDs (such as the Intel 320 series) use 128-bit encryption, which is the minimal considered adequate today. Others use much stronger 256-bit encryption, such as Samsung's self-encrypting SSDs. I'd look for the latter - even if 128-bit is impossible to break today, it will be much longer before 256-bit can be broken, assuming there are no flaws in the AES algorithm itself.

Thanks for the reminder to do a backup, it is now running...
 
Zelig, thanks for that link to SpiderOak. I had heard of them before but never really looked into it. I feel welded to Google [Drive, Gmail, etc], but this service seems like a way out. In our situation we have a couple of macs at home, an iPhone, and an Android.

Well Spideroak, Dropbox and Skydrive are all roughly equivalent and cross-platform, the main advantage of SpiderOak is full encryption on their end.

We need to sync some text files across all devices, and also contacts. Photos and videos we need to backup are currently handled by TimeCapsule onsite, and I try to do manual backups every couple of months onto a portable external that I keep off-site. Last time I managed to do a backup was in June :sad:

Can we talk about encrypting drives? I don't quite understand the actual details of it. I think I recall hearing that you can't encrypt a TimeCapsule after the fact - it has to be encrypted when you first set it up. So that raises the question "should I wipe my TimeCapsule and start over with encryption?"

Probably.

I'm not really familiar with timecapsule, I never really saw the point. I encrypt all my Windows drives with bitlocker and my Mac OS ones with filevault.

Speaking of hardware and cloud encryption, how do the NSA backdoors impact decisions on which services or devices to trust?

What do you mean by "NSA backdoors"?

The possibility of the provider closing shop

What's the problem? It's trivial to switch providers.

strong 256-bit or higher encryption

128-bit AES is functionally equivalent in strength to 256-bit.

As for encrypting drives, there's two categories, software-based and hardware-based. I don't know enough about software-based on Macs to comment on it. Hardware-based drives encrypt everything on them automatically. So even if you have an unencrypted TimeCapsule, if the drive itself is hardware-encrypted, you couldn't access it without unlocking the drive. All the drives I've seen with hardware encryption use AES (Advanced Enryption Standard). AES does not currently have any known security flaws, although the NSA was involved with its creation in the mid-2000's. Some self-encrypting hard drives an SSDs (such as the Intel 320 series) use 128-bit encryption, which is the minimal considered adequate today. Others use much stronger 256-bit encryption, such as Samsung's self-encrypting SSDs. I'd look for the latter - even if 128-bit is impossible to break today, it will be much longer before 256-bit can be broken, assuming there are no flaws in the AES algorithm itself.

Thanks for the reminder to do a backup, it is now running...

Hardware-based encryption is garbage in either security or ease-of-use on non-SSDs. Just use filevault or bitlocker, they tie into your OS login and on modern SSDs bitlocker will automatically use the hardware encryption built into the drive anyway, and I assume filevault will be updated to support this as well in the next release of Mac OS.

AES has several known security flaws, they're just not particularly serious.
 
SkyDrive is great if you want to, for example, share pictures, but I don't think Microsoft can really be trusted for critical files if you don't want the NSA to see them, either.

Obviously SpiderOak if you're paranoid, but MS is probably the best of the big providers on that front.

http://www.washingtonpost.com/world...fc36de-6c1c-11e3-a523-fe73f0ff6b8d_story.html

Microsoft general counsel Brad Smith took to his company’s blog and called the NSA an “advanced persistent threat” — the worst of all fighting words in U.S. cybersecurity circles, generally reserved for Chinese state-sponsored hackers and sophisticated criminal enterprises.

http://arstechnica.com/information-...en-networks-code-against-government-snooping/

the company will open "transparency centers" worldwide to expand its existing program to allow government customers to review its source code. "We’ll open these centers in Europe, the Americas, and Asia, and we’ll further expand the range of products included in these programs," Smith wrote.

http://www.theverge.com/2014/1/22/5...seas-data-storage-in-response-to-nsa-concerns

Today, Microsoft announced an unpredecented response to concerns of NSA data access, offering customers in foreign countries the option of having their data stored outside US borders.
 
My solution:

I use a 1TB hard drive to back up all the data from my computer on as well as my a backup for my smartphone's SD card. To back up the hard drive, I burn all of the data on it to DVD's periodically. Some essentially files are stored on Skydrive and a few others on Dropbox. I'll be using Skydrive more and more as a backup for my data in the future, I only realized a few weeks ago that I could use it that way.

For your needs I would suggest thinking about how much effort you feel like putting into your storage solution. For someone like me, I don't feel the need to set up fancy networks and servers, etc. It's overkill for me but of course for others it's absolutely necessary. If you just want to back up your data, a hard drive and DVD's/Blurays/M-Discs are fine. If you want to be able to also access your data remotely, then services like Skydrive* and Dropbox are pretty awesome. Beyond that, I guess it's a matter of judging how much utility you would get out of doing something similar to Zelig compared to the effort you put into it.



*I'm guessing you already have some version of MS Office - and if you don't have Office 365 then you should think about it the next time you buy a computer and have to buy MS Office, it's totally worth it because in addition to Skydrive storage, you get to use MS Office programs from any computer with an internet connection.
 
I back up everything critical and irreplaceable to Skydrive (mostly University work and personal stuff), and put everything else on a 3 TB external hard drive. I'm safe from anything aside from physical disaster so long as both my backup and live data don't fail at the same time.
 
Top Bottom