Friday, December 24, 2010

Seriously people, back up your data!

I write today not - as I would want - with a long-overdue update on our road trip (now long over), but rather with a cautionary tale about data loss. Read and learn from my example! A road trip retrospective will follow later and I hope that will be much more entertaining. This post is going to be a bit long, but please read it all, my experiences are an important lesson.

So why am I talking about data loss today? Well, surprisingly it has nothing to do with my laptop's unexpected death in the middle of our road trip. In fact, this road trip seems to have ended up something of a lighting rod for technology issues (fortunately none involving the car!). After my laptop died, I of course continued taking photos, loading them on the external hard drive I had brought, using Katy's still very much working laptop (thankfully we both brought one). So even though I was mostly out of communication, things continued along just fine. It was when I got back that real trouble began.

Naturally one of the first things I did upon my return was to load all my photos from the trip (about 10,000) onto my home data storage system. I had been using a Lacie 4big Quadra for primary mass data storage (10's of 1000's of photos, documents, projects, work files, and much more) for the past year or so, which has a total of 2.7TB of formatted space when using RAID5. I had about 1.5TB of data on it after loading my photos.

Now, I felt pretty secure with the data on the 4big given it was RAID5. For those not familiar with the technology, it basically uses multiple disks with a sophisticated data distribution system that allows for redundancy. This means that theoretically an entire disk can fail and your data is still ok because it can be rebuilt from the other disks. If 2 disks fail simultaneously (or the controller fails), then you have a problem. Theoretically however the chance of a double disk failure is lower than that of a single failure, so one would imagine the data is safer than with a single drive.

Unfortunately double disk failure or single disk failure combined with other corruption can and does happen, as I found out much to my dismay. I loaded all my photos onto the unit shortly after my return and began sorting through them and posting new sets every day or two. After a week or so of working on photos off and on, I started to see some issues reading certain images. I checked my Windows event log and found a whole bunch of disk-related errors essentially saying my 4big drive was corrupted and it needed to be scanned for errors. I rebooted shortly after and a disk scan ran automatically. Though I've never had much faith in Windows' chkdsk utility, I soon found out that it's even worse to run it on a RAID.

Chkdsk ran, finding a lot of errors, and "correcting" most of them. When I got back in to Windows, I found some missing images (by examining the disk scanning log I was able to see what files it had found "orphaned" or otherwise identified as corrupted and attempted to fix). Fortunately the files lost were fairly minimal, and things seemed to be working ok. That didn't last long though. Within hours my 4big unit became unreadable. Windows still recognized it but it no longer showed the drive as being formatted and no data was accessible. I shut down the computer and the 4big unit and rebooted. The 4big began showing a series of varying configurations of warning and error lights. Eventually it just showed a total failure, meaning at least 2 disks has failed. I was shocked, but not yet panicked. I've seen odd - but correctable - issues like this before (not with this particular unit).

I contacted Lacie technical support, beginning an odyssey of 2 weeks of back and forth during which I received very little useful advice beyond resorting to extremely expensive data recovery options. The first tech I got essentially told me that the indicator lights on the unit didn't necessarily indicate anything useful and that the only way to determine if it was an actual drive failure vs. say a controller failure would be to reinitialize the unit. This meant losing all the data for certain, so it really was not an option. One option I would have liked them to offer would be to ship an identical unit without the drives so I could switch my unit's drives into it and test there. If it still failed, it would indicate it was a drive issue rather than controller or other hardware problem. But they weren't willing to do that or provide any other remedy besides shipping the drive to them in Oregon for data recovery at a cost of thousands of dollars.

At this point I was fairly resigned to having to do data recovery, but still hoped for some cheaper alternatives. I didn't relish the idea of shipping the unit hundreds of miles up to Oregon so I started looking into local data recovery options. I looked around on Yelp and did other general searches online. The world of professional data recovery is rather mysterious and almost universally very expensive. Your average computer shop can sometimes handle very basic recovery, but I can do the same things they can at home, so that wasn't an option.

After a bit of research I came up with a list of local options and started calling to get an idea of rates, turnaround time, etc. The first company I called, Hard Drive 911 Data Recovery, was actually extremely helpful and although I didn't end up going with them, I would still recommend them if only on the basis of their extremely informative and frank reps. The fellow I spoke to told me in plain terms the cost of recovery (minimum $1000 for a RAID 5 with 4 drives, up to $9000!) and the up-front inspection fee (others offer free inspection, 911 credits the inspection fee toward a recovery if you decide to go ahead). Their costs did not end up being particularly more expensive than many other options (this is a shockingly expensive service overall, no matter where you go), but their rep was far more helpful. We had a very candid discussion about the costs and value of professional data recovery and he made clear that they want their customers to feel like the service is worth the cost. He gave me a lot of information and links to resources online to attempt my own recovery, and advised me to get back to them should I want to pursue professional recovery options.

I spent the next 2 weeks attempting recovery myself. My very first task was to image all 4 1TB disks. This takes a very long time to do in pure RAW mode, 1:1 copying (all 1TB of data), so that took up several days in itself. Once I had the RAW images I could attempt recovery on them, rather than further risking the originals, which I wanted to preserve for others to work on should that become necessary. Note that the simple process of attempting to create images on my basic home setup through regular SATA could have caused further damage to the disks, so there was some risk inherent in what I was doing. However I didn't hear any unusual noises from the drives and didn't suspect physical damage, so I felt it was safe to do.

Over the next weeks I tried a number of software tools to recover the data. The biggest challenge turned out to be figuring out the RAID parameters so that the data could be actually be accessed properly. Because RAID distributes data across all the disks, including "parity" (redundancy) information, it can't be read like a normal disk; you need to use special RAID emulation approaches in software, and you need the correct RAID parameters. I didn't have these, and Lacie was extremely reluctant to provide this information to me, which was another frustration I feel was unnecessary. I eventually escalated my support request to a senior staff member who did provide this information, but it ultimately did not help.

In the end I was unable to recover any meaningful data myself. I feel like I was pretty close, I got to the point of being able to list a lot of files and actually see meaningful file names, etc. But the actual data contents wasn't correct, and I suspect I either didn't have enough information on the RAID parameters, or (perhaps more likely) there was corruption in the RAID data model itself. It was time to think seriously about professional data recovery again.

By this time it was almost a month since the failure had occurred. The vast majority of my important data, including 5+ years of photos, 10+ years of documents and project files, and much more was unavailable during this time. I didn't realize it until several weeks in, but it was really affecting my happiness and overall mood, but at a fairly subconscious level. This is not surprising, but as I said I didn't even realize it until after several weeks. I knew I had to get the data back, and that professional data recovery was likely the only option.

Lacie operates their own in-house recovery service called D2; in fact Lacie support initially indicated I was required to use them in order to maintain my warranty (I later found out there were several other authorized providers). I actually had a great back-and-forth with Patrick, the manager there, in which we discussed the recovery options and costs. Since it's an in-house service, they can also perform warranty service, and their costs were lower than most local options. They also offer free 2-way shipping and free evaluation. The service sounded pretty good. But something in me balked at the idea of giving the same company potentially thousands more of my dollars just to fix a piece of hardware that I felt shouldn't have failed in the way it did anyway (not to mention my relatively negative experience with their tech support earlier). I initially asked to proceed but, after not receiving a response for a few days (they did eventually send me a shipping label), I decided to try an evaluation with a local company first.

I had done price comparison and reputation research on a number of local companies. One of the more well-known companies is DriveSavers in Novato. I had initially steered clear of them, partly on the basis of some negative Yelp reviews. Eventually I did call them and ended up having a very nice conversation with one of their reps. We even discussed the Yelp reviews, and having been on the other end of some negative Yelp reviews for businesses I work with, I sympathized with where she was coming from. They were local, in Novato, I could drop off the drive on my way to or from work in Penngrove, and they could do a free analysis within a couple days. So it seemed like a worthwhile option. Their pricing was also comparable with other local options, and their high-end ($4410) was lower than several other local options, so in the worst case scenario I'd pay less than with the same problem at another vendor. The high end of the range usually corresponds with significant *physical* damage and at this point I didn't think that was the case; after all I felt pretty close to recovery with my own simple tools. With the slow response from D2, I decided to go ahead with it.

I dropped the drive off and waited... A few days later I got a call back and they said they estimated they could recover 90% of the data and the cost would be between $3900 and $4400, the maximum they said it could cost before they even looked at the unit. At this point I was tempted just to get the drive back as I was fairly sure the level of corruption didn't justify the cost. I called DriveSavers to talk to the tech working on my recovery, asked him a few questions, and felt his answers were rather condescending and uninformative.

He said there was physical damage on one of the drives, which I had also encountered when I imaged the drives originally. 9 sectors to be precise, which is not much on a 1TB drive. I told him I had some images that were made prior and offered to bring them by if it would help, but he brushed that off as if it was ridiculous that my images taken earlier might be of use. Maybe he's right, but his reaction was not particularly nice. Then I asked him why, if it was just 1 drive with physical corruption, it was going to cost so much (and theoretically be so difficult) to recover the data; after all, isn't RAID5 *meant* to recover from a single drive failure? He barely answered the question, saying only that there was other unspecified corruption, possibly due to chkdsk, and that it was necessary to do advanced recovery.

Thinking back to the Yelp reviews, I recalled some indications there of dubious information from DriveSavers tech support as well. Tales of "motor failure" and "intermittent hardware failure" sounded similar to the lines I was getting. In several of the Yelp reviews people even got their drives back and found other, cheaper ways to recover, discovering that the problem indeed was not as dire or complex as they indicate (for example a simple power supply replacement). That being said I felt by this time that I'd exhausted most of my own simpler diagnoses and recovery options, so even though I didn't trust what I had been told, I wasn't really clear there were many other options.

Then I called Patrick at D2 to see if he felt he could do better on the recovery if I sent it up to him; after all at this point I was still within the free eval at DriveSavers. He felt - and I tended to agree, though reluctantly - that if DriveSavers already have the drives and had them dismantled in their labs, that having them reassemble and box up so I could ship to D2 would incur further risk and I might not get the data back (or not recover as much).

This is where the mystery of the data recovery industry plays into their hands, theoretically justifying the high costs. After all, they're the experts, they can tell you whatever they want, and who are you to argue? The uncertainty customers feel, and the fear of losing data, can justify almost anything in that moment. The lack of information and transparency, the difficulty in trusting these companies, is the biggest problem I have with the whole industry, to this day. I still felt like they weren't telling me the whole truth, but I didn't have any real facts to back that up, just my own experiences with drive failure in my time in IT. So ultimately I said yes, and the final bill was indeed $4410. I paid $4400 to get my data back. And you know what? It was worth it. Fully worth it.

But I don't trust DriveSavers, I don't trust D2 (who never gave me an actual maximum figure, they just said "I've never seen a recovery cost more than $2500", which is no kind of commitment), and I don't trust anyone else in this industry either, except maybe that guy I talked to at Hard Drive 911, who told me quite clearly that it was reasonable for me to try my own recovery first (once he confirmed I was aware of the risks).

The fact is I just don't know. It's possible that my drives were so damaged that it justified the cost. It sure didn't seem like it from any actual evidence I ever saw. And the reaction of the DriveSavers tech was not particularly confidence inspiring. It seemed like he was more interested in stifling questions than clearly informing me about the complexity - and thus justifiably high cost of - the recovery.

So I don't trust any of them, but I used their services because the alternative wasn't worth it to me. Even if they're full of crap and the recovery difficulty did not justify the cost (which I'm fairly certain of), it was still worth the price to me, and that's what matters. What matters more, though, is that I am going to do my best to never have to pay for something like this again.

As to whether I'd recommend DriveSavers or anyone else? Sure, if you have a really serious data loss scenario and the data is worth $1000s of dollars to you, by all means, do it. Not because it may be strictly necessary, but because the risks outweigh the potential cost savings. Just resign yourself to paying the maximum possible quoted amount. If you're ok with that, then it's the right way to go, no questions. DriveSavers is essentially going to treat each recovery like it's the worst possible case, from the get-go, and in some sense they may not be wrong to do that. Sometimes you don't get a 2nd chance, and if they power up a drive and it fries itself before they even get a chance to do anything, they'd have been better off disassembling it from the beginning and running it in a clean room environment. Just be aware that this is going to be their approach and don't be fooled by the low-end of their cost scale; you will almost never pay that.

So on the other hand if your data is only worth a few hundred dollars, or nothing at all, then try to do it yourself, or take it to a local computer shop or trusted IT consultant and see what they can do. It's all about how much the data is really worth to you. If you have a medical problem that could be life threatening and it will only be determined in surgery whether that is in fact the case, you don't go to the bargain clinic to get opened up, you go straight to the best, because the risks outweigh the potential savings, plain and simple. I don't like this reality, but it's a simple truth, and it's what props up an industry of chronically bloated fees. I'm now a registered DriveSavers reseller so theoretically I have some investment in getting them more clients, but I will not be recommending them lightly. I may ultimately refer some people their way if I feel it's justified. But to tell you the truth I'd much rather spend my time and energy teaching people how to back up their data properly!

As this post is now very long, I'm going to continue in Part II: Backing Up Your Data - Software Recommendations and Strategy.

Monday, September 27, 2010

Oh noes!

Ladies and gentlemen, the unthinkable has happened (well, not so unthinkable really): my laptop has died. Sadly this may mean no chance of further blog or photo posts until I return. Given my stellar record so far it may not be a huge loss but its rather a bummer as I was about to put up a bunch of Bryce and Zion pictures. But alas fate has intervened.

I will of course be tinkering with my laptop trying to fix it but so far it does not look good. I'm posting this from my new phone and while the typing experience is surprisingly good for a 4" keyboard, there's still no way this could work.

Fortunately our trip is otherwise going quite well, despite some annoyingly persistent rain (it really does seem to be following us around). We're in Georgia now, heading down to New Orleans next.

So friends, wish me luck and email me if you want to keep in touch!

Friday, September 24, 2010

Delayed reaction

So obviously I'm a "little" behind in my blog updates on the road trip. Fortunately I'm starting to catch up now (yes, I know I've said this before - shhh!), and I'm also uploading photos from further on in the trip before the blog posts go up (partly because the photos go into the blog posts).

For photos I haven't been using my Photoblog as much as I expected, though I'll be uploading my favorite picture from each day there once I select them. In the meantime it's been much easier and faster just to upload to my Picasa and Facebook galleries immediately after culling and editing. So keep an eye on either of those links for photos to show up prior to a related blog post (could be as much as a few days). Right now I'm working on a batch of Zion National Park and Bryce Canyon pictures, then comes the Grand Tetons and Yellowstone. Those should all show up within the next few days, along with more perhaps. Related blog posts hopefully within the week.

Also, don't forget the occasional YouTube upload. I'm taking a fair amount of video on this trip and trying to upload those along with the photos for a given location. For example, have you seen The Glorious Moose? Or The Gauntlet of Mono Lake Flies? Not to be missed! (especially the moose - Katy took that one)

Sorry about the delays, and thanks to anyone who is still following along! ;-)

Tuesday, August 31, 2010

Great American Road Trip Days 5 and 6: Yosemite - Tripping in the valley

We started out the day in Fresno with ambitious plans to leave by 9:30AM. We only had a bit of shopping and some goodbyes to take care of, but in the end we barely got out before 11AM. Still we were well equipped and had a chance to see my grandparents again before heading out, so no regrets.

The 2 hour drive to Yosemite passed quickly, though the foothills of the Sierras greeted us long before we actually made it to the valley. It's a pleasant drive and I hope those Fresno residents realize how lucky they are to be so close! Our Annual Access Pass for US National Parks ($80) began to recoup its value here, good for the $20 Yosemite entrance fee.

Sunday, August 29, 2010

A brief stop in civilization

A quick update from Midvale, Utah, just south of Salt Lake City. The trip is going fantastically so far. From Yosemite we stopped briefly in Benton Hot Springs (lovely!), then crossed Nevada in a day to arrive in Zion, and on to Bryce from there. Both of these southern Utah parks are outstanding, but Bryce is definitely my favorite so far (I have yet to visit Capitol Reef or Arches, not to mention the many smaller parks in this whole amazing region).

So we've been camping and without much in the way of Internet access (or power) since Monday of last week. I had hoped to be able to put up some photos and do a bit of posting here more frequently, but so far that has proved difficult. I've been keeping mental notes though and I still intend to go back and write a post about each day. So far there has been more than enough of interest in almost every day that passes.

Look for those blog posts to begin in a week or less, depending on whether we find Internet access in any of our upcoming stops - The Grand Tetons, Yellowstone, The Badlands, Devil's Tower. If nothing else, I'll get it all up here once we reach Chicago around the 8th or 9th of September. I'll begin on "Day 5", our trip in to Yosemite, since that's when things really kicked-off.

Friday, August 20, 2010

Great American Road Trip Days 1-5: Fresno - All packed and ready to shop

After leaving San Francisco on Tuesday, we have spent the last several days in Fresno visiting family and making sure we have all the essentials for our long-distance trek. We packed very well, but there are always things that get missed, or that you don't have time to buy before you leave. Everything from new hiking shoes to water containers to jumper cables and more.

In-between trips to sprawling mega-stores, we had a moment to visit the Tower District of Fresno, which was apparently recommended to Katy by my first cousin (once removed) as the "cool part of Fresno". I mainly remembered it for the classic old theater and its tower that unmistakably identify the area.

We found a rather nice little tea shop and had some enormous iced teas while we relaxed and did some people-watching. One or two other little gems presented themselves...

But my favorite was the pie shop.

We unfortunately didn't have time to sample, but I'm curious for next time.

Tomorrow we leave for Yosemite which is the real start of our trip, though we'll be in a cabin so the first test of the tent will have to wait for Benton Hot Springs on Monday. From there we drive straight across Nevada to Zion National Park, the first major destination that neither Katy nor I have seen before. I expect to put up the first pictures from the trip on my photoblog as soon as we get Internet access again after Yosemite.

It's been great to see everyone down here, lots of relatives we don't get to spend much time with, and some of whom were in from out of town as well. There's no hospitality like that of close family and we're fortunate to have this be our first stop. It's a nice way to ease into the coming weeks of travel. Soon enough we'll be on our own, fending for ourselves with nothing but our new (and largely untested) camping equipment to rely on!

Next check-in will probably be from Benton or beyond...

Tuesday, August 10, 2010

Time Flies and the Epic Road Trip

Just a quick update this time. I started writing a longer, more involved post, and then I realized it was kind of a waste of time. A lot has happened since my last update and I can't possibly do justice to it all in any single blog post. So rather than try to do that, I'll just mention a few things and leave the rest for future updates.

Most importantly, although this blog has seen very infrequent use, it's about to get a lot more action. I've quit my "day job" at Bauman College, where I have worked for the past 4 1/2 years, and I'm going on an 8 week road trip with my girlfriend who just graduated from college. We'll be doing a big loop around the US, first North then South, and we've got a ton of things in the itinerary already.

We're also hoping for input from our friends and family on recommended places to visit and things to do on our route, so take a look a the PDF of our planned itinerary and let us know if you have something we just have to do or see on the way (in comments or by email). This is by no means a complete list of what we'll be doing, it's just an overview of the route and major "way points" (and we're leaving it somewhat open to allow for a good amount of flexibility in the details).

Along the way we'll both be blogging, taking photos, and even trying to get some decent video. I've got a new compact digital camera with high definition video recording capability (Panasonic ZS7), an intervalometer for my dSLR for shooting time-lapse (Pclix), and a new wide aperture prime lens as well (Sigma 24mm f/1.8). Photos will be at my photoblog, and video likely on my as-yet unused Youtube (or Vimeo) account. Reviews of all the new equipment will probably show up somewhere on our journey as I get the chance to use it all.

So if you'd like to follow us on our trek, we'd love to have you along for the ride. We'll try to make it interesting. ;-) When I get back I expect to have some more time to keep the blog updated on a regular basis and I've got some potentially interesting posts waiting in the wings that I'll be finishing up and publishing. Not least of which is the 2nd part of my 5 day week concept.

You can expect to see a new blog post at least once a week from here until the end of the trip!

Thursday, February 18, 2010

A new neighborhood

So it's been a few months again since my last post. But this time I have a good excuse - I moved to a new neighborhood!

I've been in San Francisco now for almost 2 years, most of that spent at one place near Japan Town and Hayes Valley. That place definitely had some nice aspects, and the neighborhoods were not bad, in fact I thought living near downtown was really the way to go. So when we started looking for new digs due to mold problems and a lack of natural light in the old place and ultimately found a great place in the Inner Richmond, I figured we'd be settling as far as the neighborhood was concerned, at least in relation to our old one. But it turns out I'm way more into Inner Richmond than Hayes Valley, Japantown, Downtown, or just about anywhere else I've spent much time in SF!

Read on to find out why the Inner Richmond rules (and where to go if you find yourself here)...