I've moved on...

It probably comes as a surprise to no one that I've moved any of my writing efforts to new places long ago. More than 10 years ago by now, as it seems. Yet Blogger is still here. So here's a pointer to where you can find any of my future ramblings and other projects:
https://www.oshyan.com/

Most recently I have been experimenting with the "digital garden" format:
https://garden.oshyan.com/

CNet screws the pooch, wraps all downloads in crapware installer

Update 2: According to Renegade on the DonationCoder forums, my initial understanding of what CNet is doing is not entirely accurate. Rather than actually wrapping software installers in an additional install routine that makes unrelated offers to users for CNet's profit, CNet is instead giving users a download manager with ads in it that then downloads the original software's installer. This is not as bad as initially thought, in particular it probably has far less significant legal consequences (if any), but it's still a bad practice on CNet's part, especially without notifying developers or giving them a clear, free opt-out method.


Update: Some good-ish news from Seth Rosenblatt of CNet, posted in a comment on GHacks.

In case you haven't heard, CNet has decided to take the morally reprehensible and legally questionably step of wrapping all downloads on its Download.com web property in a proprietary installer that prompts users to install a toolbar or other "offer". The profits from this go straight to CNet, authors don't see a dime. Not only that but they have the gall to charge authors for the ability to remove the crapware installer. They're claiming all this is "for the benefit of the user", but there's no real benefit to the user anyone seems to be able to point to. Needless to say many software authors are horrified, angry, and looking for options. DonationCoder has some discussion and thoughts from a few such software authors affected by this new policy.

As an added sad little wrinkle, many authors don't even choose to upload their software to download.com, it gets up there through other means, user submission or editor selection for example. Authors then have to request their software be removed, and sometimes it takes a while to get a response. Meanwhile CNet is now profiting off their app as long as it's up there.

From a moral standpoint this is clearly very bad. CNet is making money off of the work of others while doing little or nothing themselves. Providing hosting is great, but it's pretty cheap these days. Reviews are helpful but take little time in the overview style CNet uses. What other value is CNet providing? But obviously nobody expects big companies to act on moral grounds. What's more surprising is that they see this as a legally sound move given that many software products are distributed with explicit EULA and/or distribution agreements that prohibit modification, commercial use, and profiting by 3rd parties. While CNet may or may not have dealt with this for larger, commercial vendors, I'm certain they haven't done so for smaller authors whose legal right to determine the distribution terms of their software is just as valid and important as larger publishers.

Many are now wondering what to do in the wake of this new policy. I think the options are pretty clear:

If you are a software downloader who might have used download.com in the past, stop immediately, do not pass go, do not download another file from them, and don't return to their site. If you're really feeling fired up about it like I am, write them a polite by clear email. You can do that here: http://www.cnet.com/2723-13403_1-461-13.html?tag=rb_content;contentNav
It also wouldn't hurt to Twitter, post on Facebook, Google+, or write your own blog post. Spreading awareness is the strongest weapon we as normal users have.

To get your software fix now that you can't use download.com, consider Fileforum, Snapfiles, Softpedia, or Brothersoft. Note: I'm not personally vouching for any of these, they're just other sites I'm familiar with and which, to my knowledge, are not yet bundling crapware. Of course the best policy is always to download from the official website for the software in question, if possible.

If you are a software author whose software is currently hosted on download.com your choices might be a bit more complicated, but they are still fairly clear. You can of course simply remove your software outright (and email a strongly worded complaint to CNet while you're at it). I think this is the simplest and perhaps best approach, but you might be concerned you'll lose search engine placement, reputation, reviews, download stats, etc. Some of this is true, yes, but the question is whether it's worth compromising your relationship with your users for this. When a user downloads your software from download.com, their user experience is compromised with no real benefit to you, the customer is likely to be unaware of the source of the crapware that may get installed, and they're likely to blame you. By removing your software you may be losing some downloads, but you are ultimately protecting your long-term relationship with your users and that's worth a lot more than a few extra downloads. 

Some might question whether it has any affect on CNet for you to remove your software. The likely answer is no, unless your software is extremely popular, but if 100s or 1000s of authors do this, then it may in fact impact their bottom line. But more importantly it's the value of your relationship with your users that is at stake here, not whether you can in fact strike back at CNet. Being able to do that could be vindicating, perhaps even bring about change eventually, but it is much harder to achieve and the effect is questionable. What is not questionable is that your relationship with your actual users is damaged by CNet's download wrapper policy, so you owe it to yourself and your users, if nothing else.

One possibility that has been mentioned if you do want to keep your software on download.com, perhaps even use it as a weapon against CNet, is to use your own installer detection routines to trigger informative or anti-CNet messages whenever your installation is launched from a CNet wrapper installer. This may in fact be an effective weapon, though I doubt CNet would tolerate it for long. But I'm very curious to see someone try it.

One last thought for developers, if your EULA or distribution terms are clearly stated and indicate that what CNet is doing are illegal, you might consider letting them know about this fact (politely but firmly). They should be aware that they are on legally dubious ground. I doubt any real lawsuits will come of it, but you never know. Perhaps an enterprising software author with a bee in their bonnet could even setup a petition or Facebook group to collect interest in a class action suit...

The sad thing is download.com has been a pretty decent site for a long time and has been around since the earlier days of the Internet. I myself have used it as a source of free software downloads for years. Yes, I've enjoyed the benefits of these free services and, yes, taken them for granted. But they survived this long without screwing over their users, I can't help but wonder why now. Maybe CBS Interactive is looking to balance the books and squeezing some of its business units for revenue to make up the difference, who knows. 

All of this upset could have been avoided, too. CNet has a need to remain profitable to survive, that's certainly understandable. They provide some services that do deserve some kind of compensation, even if "only" ad revenue. They made the decision that they needed additional revenue sources, which is their right. Where things got off track is not consulting the authors of the very software they depend on for their high traffic numbers and existing income. What they may be doing now is gambling on a new revenue source at the possible expense of an older one, if the outrage I'm seeing is indeed any real indication of future usage and traffic numbers. It could in fact affect their bottom line and they may end up seeing a net negative in profit as a result.

What CNet should have done is roll out a new developer incentive program. Something like OpenCandy, but CNet-driven and opt-in. Share the revenue from installer-based offers with the software authors themselves. Then they become your biggest advocates and sales agents. This is what OpenCandy has done, and while I am not totally in favor of it, it has been relatively successful so far. CNet has the enviable position of also being able to offer software authors not only a revenue source from software bundling options, but also an effective, proven platform for promoting and distributing their software. Add a few premium services and they've got an additional revenue source without compromising existing value and reputation. This approach could have been very successful.

It's not too late. CNet has gotten a lot of publicity from this - negative publicity but publicity nonetheless. If they changed to an opt-in, revenue-sharing model, they could turn the negative into positive and while rolling out an opt-in service from the beginning might not have made such a big splash, you can bet those who already reported on the current policy would mention an update, thus ensuring greater publicity for the venture. Could CNet be that savvy that it was planned this way all along? I doubt it. But that would be pretty clever, not to mention devious...

Google+ Impressions

Well, it's official. I am PLUSSED. Wooo! So what do I think of Google+ so far? It rocks! Yes, there are some rough edges, yes some things don't work all of the time. But the things they are doing are either an improvement over Facebook (Circles), a totally new and potentially interesting feature (Sparks), or something Facebook doesn't have and may potentially be a "big deal" (Hangouts).

There are 4 main tabs/buttons to the primary UI. First is Home, this is basically like FB's home, including the stream of what your circles are sharing (like FB's "wall", of course), as well as controls and tools like stream selection, Spark links, Chat, Hangout, Contact Suggestions (suggestions from inside your own contacts as far as I can tell, but maybe later like FB's "do you know this person?"), and a people search bar.

Next is photos, which get top billing as one of these 4 buttons. Here you see photos from your circles (they also seem to show up in the Streams), photos from your phone, photos *of* you (face tagging is of course available, with one critical difference: you can control who is allowed to tag you without explicit confirmation), and "your albums" which are your Picasa albums. Google has a nice photo system in Picasa, so this is a great thing to get integrated. They've come up with a decent UI for browsing photos too, although sharing settings are not entirely clear yet (there's no explicit "share this photo" that I can see, although you can share whole albums; but if you make a comment on a photo it posts the photo and comment to your wall, so it's basically sharing it). You can also upload new photos. The photo viewing and commenting experience is definitely an improvement over FB (unsurprisingly as Picasa has been around for ages and is fairly mature).

Then there is a Profile link/tab. This kind of surprised me. My profile is actually somewhat barren at present. It shows my photo and my activity stream, like looking at your own profile and wall on FB, and there are additional tabs for About, Photos, Videos, +1's, and Buzz. This is in fact the only place I have seen Buzz, and it's interesting that it's still around and being semi-integrated into G+. But I would expect it to be a major tab/button up top if they actually intended to keep it. Otherwise one wonders how it differs from +1 and posts to the Stream(s). A nice touch is you can click on a button to easily and quickly view your profile as *anyone* in your contact list or as a general anonymous web surfer, so you can check to make sure your privacy settings are working very fast. Also nothing (or almost nothing) appears to be public by default, i.e. I click to view as the public web and nothing shows up in my Stream. When you click to Edit your profile, you just point and click on any part of your profile to edit it, including adding photos, links, places you've lived, personal description, etc, etc.

Last but not least is the Circles button/tab. Circles are exactly what you've heard, they're groups you can use to categorize your friends and enable sharing of specific content with specific groups extremely easily (or, seen the other way, they enable you to easily *avoid* sharing content with specific groups, e.g. the general public). They've made the Circle-making UI fun and this helps a lot in wanting to make your circles. It's also powerful, you can multi-select, drag-select, etc. so you can easily deal with hundreds of contacts, and it's all drag-and-drop with nifty animations. This functionality is definitely a big step over Facebook (and yes I've tried FB's "Groups" system). You can add people to multiple Circles, share with 1 or multiple circles, all your circles, or the public. It's a nicely flexible system that is also easy and fun to use. This to me is a big win. I know it's not as important to many other people as it is to me, but this is a "killer feature" that I have been waiting and wishing for. Once nice touch is you can easily control people's Circle membership from almost anywhere you see their profile pic (e.g. hover over a pic of "Joe" on a post he made to your Stream and you get a pop-up allowing you to add or remove him from your Circles). Little things like that set the UI apart.

Unsurprisingly the Home area seems to be where you'd hang out the most, just like Facebook. On Home are two things that also deserve some description. First there's Sparks, which I think is probably the least impressive and interesting feature, but still has some potential. This is basically like a topic-driven web link discovery system. You enter some key words and it finds you content based on them. This is not like a regular Google search, I'm not sure what they're doing, but it seems much more culled than that. I think how recent a result is plays heavily into whether it will be shown. Better than a simple search though is that you can save this to your "Spark List" and it will continually update you on new info for that area of interest. Naturally you can easily click to share anything in your Sparks. Like I said it's not too exciting, lots of other services already do similar stuff and probably do it better, but if they continue to flesh it out it could make the G+ home page even stickier by making it a real dashboard not only for all the info on your friend's activities, but also *all* your interests around the web.

Then there's Hangouts. This one seems like a possible game changer. It's not revolutionary in basic function, but its integration with a social network just may be. Basically it's "just" group video chat. It works through Google's existing Google Chat system that has included video for years. Only now it allows multiple people, and it has a pretty slick system for keeping focus on whomever is talking, or letting you choose focus, or selectively mute people. One cool feature is that it allows you to watch YouTube videos "together" (simultaneously), so you can get a fun shared experience. It's actually pretty novel, believe it or not, though it's not necessarily amazing or something you're likely to use a lot in the long run (though some people who like watching YT videos with friends more than I do might love it). It also includes text chat, and lets you invite more people in very easily. Incidentally inviting people to Hangout is the way I've found to get anyone a G+ invite. Shh, keep it under your hat! ;) When you start a Hangout it posts to your stream and others can easily join. It also documents who was part of the Hangout ("Oshyan was Hanging out with Katy and Corey").

I jumped into a Hangout with a friend who was in the room and another friend half way across the country who happened to just call me and I was able to send him an invite. I really just wanted to test the functionality but it ended up being really fun and cool. It's a well-done system and, despite the occasional glitch, works nicely. It has a simple, clean UI and just enough features to be exciting and useful, while not being overwhelming.

So you're probably wondering why I think Hangouts could be a "game changer". Well, so far group video chat hasn't really been easily or widely available. Skype, for example, just introduced it recently and they *charge* for the feature. I'm sure we all know someone who uses Skype or even Google Chat to keep in touch with friends and - especially - relatives in distant places. It's becoming almost cliche for grandma and grandpa to video chat with their little niece or nephew on the opposite coast. But until now it's just been one person at a time. Now imagine grandpa and grandpa being able to get on to chat with little Johnny and their parents, plus Sarah and her new husband living in the UK, and Bob on his trip in the Philippines. The reason I think this could be significant is because it's a potential "killer feature" that might drive adoption with "everyone else", i.e. people who aren't techies, aren't hanging out on this site, but love Facebook because it lets them (re)connect with people easily, and love Skype because it lets them see and hear their loved ones for free. Google is giving them more, and it's giving it to them in the context of a social network. People will want to be able to do this, and the only way - so far as I know - is through G+. So sign up to G+ and you can group chat with anyone you want, for free. This could be a big incentive.

Back to the Home page. The "stream" (like FB's wall) is nicely formatted and clean, and otherwise almost exactly like FB's version. All the controls and tools you'd expect are there including who is involved with a given post (for Hangouts it shows thumbnails of the people who were there for example), details of the sharing status (public, with particular groups, etc.), the ability to comment on, +1 and share any post, to mute a post/thread or the entire person or report abuse, etc. You can even disable comments and reshares on individual posts which is quite nice and another privacy-preserving feature (after all, just because you post something privately doesn't mean others will keep it private!). There are also individual streams for each Circle as well as an aggregate stream.

Another piece of the puzzle is that new black Google bar you've been seeing on the Google home page. When you're signed in to G+ it follows you around on any Google web property, letting you easily share, +1, and do other G+ type stuff. It's nice to have and I'd even like the option to let it follow me around on general browsing to make sharing easier from other sites. I imagine they might implement that sooner than later. Surprisingly +1 and other features don't appear to be rolled out across all of the major Google properties even, with YouTube one notable omission. But Shopping, meanwhile, does have +1, as does the general web. Interesting.

Last but not least, there are lots of controls in your G+ settings allowing you to adjust notification types and methods (you can even be notified via SMS). The "Data Liberation" system you may have heard about is built right in to the settings, allowing you to export data for Picasa, your Profile, Stream, Buzz, and Circles and Contacts. Facebook has similar functionality, although it didn't debut with it, and I haven't compared the two. But the easy accessibility and existence of the feature right from the start is very nice. You can also, interestingly enough, link other accounts with your G+/Google account, including Facebook, Twitter, Flickr, Yahoo, and more. I'm not entirely sure what the linking really does yet, but it at least (optionally) adds a link to your page on that service to your profile's About area.

The one thing some have said is odd is that it's not integrated with Gmail like Facebook's new "messaging center" email thingy. I don't think that just because FB is doing it Google should too, but I do note a distinct lack of a real "private message", "direct message", or similar type feature (as far as I can tell). Your G+ contacts are your Gmail contacts though I think (haven't tested whether adding someone through G+ adds them to my Gmail contacts though), so theoretically you can just jump over to gmail to email them if you want. There's also direct chat and Hangouts. Still I see this as something that needs addressing somehow. Many a mini embedded Gmail interface or something.

Last but not least, Google nicely includes a slick Feedback tool/link. Clicking Send Feedback in the lower-right of any G+ page pops up a nifty feedback tool. It has the comment form you'd expect, but also tools to highlight or black out multiple areas of the screen just by clicking and dragging. Highlight and blackout areas can also be easily removed. When you're done marking up the page and commenting, you preview your feedback and it shows how they'll receive it. This is all very nicely done and makes giving feedback easy and even kind of fun.

Now I should mention that my impressions about the UI might need to be taken with a grain of salt depending on your perspective. I hate and have always hated the Facebook UI and find it very unintuitive. I'm usually quite good with web UIs and software UIs in general, so it's always frustrated me that FB's is so opaque to me. Google's is much better, but I don't know how objective that is. I'm curious to hear the opinions of people here as I trust the viewpoints to critical, diverse, and interesting.

I've also just installed the Google+ Android app but haven't had a chance to play with it much yet. The UI is fairly nice though and some good ideas like auto-upload of new photos and videos for later sharing. So I'm looking forward to exploring it and expect to be using it much more than FB's app.

Overall I'm impressed and I was *not* impressed by Wave or Buzz. They appear to have gotten this right for private beta, and hopefully the rough edges will me smoothed out soon so the whole world can check it out. I'm super curious to see if this makes a dent in Facebook's domination. Like I said there are some potential killer features (Circles, Hangouts) that might just make the difference...

More photos and general update

Woah, hey, it's been a while. Surprise! OK, not really. As is so often the case, a lot has changed. And as is also often the case, I come with promises to finish all the previous multi-part blog posts on here, soon I swear. At least the backup one, because I'm finally getting my whole data scene together. More on that soon.

On the subject of life in general, many things have changed in the past 3, 4, 5 months. Katy and I broke up (a mutual thing, but still sad and difficult), consequently I moved to a new place in the Mission (loving it!), and I finally (re)left my Bauman College job (eek, I have to set my own schedule now). Life is a bit unclear for me right now, but also full of possibility, so that's a good thing. I'm working on setting a comfortable schedule for myself, balancing work and play, getting things done for Planetside while also enjoying my new neighborhood, doing more hiking than ever (took a great one to Mt. Tam recently), and trying to meet new people. Things are good, but not quite settled yet...

Before I go, I want to mention that I'm still slowly going through the road trip photos. It's funny to me now to think that I expected to be able to edit and post these as we were on the road. Even if I'd had a working computer and Internet connection, I doubt I would have had the time or patience. Granted I'm learning a new tool (Lightroom), but I've also realized I can be super indecisive when picking between several similar photos. So culling down several hundred (or a thousand) to less 50-70 (a comfortable amount for a single gallery) can be tough. At any rate I'm doing it now, and it's enjoyable to look back at these places and remember the experiences, already starting to fade only 6 months past.

I've posted a gallery for the next stop we made after Bryce Canyon which is the Grand Tetons in Wyoming. We had some rather poor weather while we were there (it rained most days), so I really want to go back and get more hiking and exploring in. But we still saw and did some great stuff. Here's the gallery link, and the description I wrote for it:

https://picasaweb.google.com/oshyan/GrandTetons#

We spent several cloudy, often rainy days at the Grand Tetons in Wyoming. Lots of challenges in exposing sky vs. ground, water, etc. Some HDR experimentation, and liberal use of Lightroom's sweet-ass graduated filter. Speaking of which, this is the first full album I developed and culled entirely using Lightroom. So far I'm liking it, but I'm really just getting used to its workflow. Once I'm "in the groove" with it I think it'll be a much better experience overall than Picasa. But I will miss the face recognition!

Writing that made me realize I ought to blog more about these experiences. My explorations may be of interest and/or use to others, especially those looking to transition from an amateur to pro tool (Picasa to Lightroom). I used to write about my techniques a bit on my photoblog (and I hope to one day again), but now that I'm not really using it, I realize I've stopped giving as much info. I'm not sure anyone was ever really super interested in the details, but hopefully they're worthwhile.

Thanks for looking and reading! More to come...

Seriously people, back up your data!

I write today not - as I would want - with a long-overdue update on our road trip (now long over), but rather with a cautionary tale about data loss. Read and learn from my example! A road trip retrospective will follow later and I hope that will be much more entertaining. This post is going to be a bit long, but please read it all, my experiences are an important lesson.

So why am I talking about data loss today? Well, surprisingly it has nothing to do with my laptop's unexpected death in the middle of our road trip. In fact, this road trip seems to have ended up something of a lighting rod for technology issues (fortunately none involving the car!). After my laptop died, I of course continued taking photos, loading them on the external hard drive I had brought, using Katy's still very much working laptop (thankfully we both brought one). So even though I was mostly out of communication, things continued along just fine. It was when I got back that real trouble began.

Naturally one of the first things I did upon my return was to load all my photos from the trip (about 10,000) onto my home data storage system. I had been using a Lacie 4big Quadra for primary mass data storage (10's of 1000's of photos, documents, projects, work files, and much more) for the past year or so, which has a total of 2.7TB of formatted space when using RAID5. I had about 1.5TB of data on it after loading my photos.

Now, I felt pretty secure with the data on the 4big given it was RAID5. For those not familiar with the technology, it basically uses multiple disks with a sophisticated data distribution system that allows for redundancy. This means that theoretically an entire disk can fail and your data is still ok because it can be rebuilt from the other disks. If 2 disks fail simultaneously (or the controller fails), then you have a problem. Theoretically however the chance of a double disk failure is lower than that of a single failure, so one would imagine the data is safer than with a single drive.

Unfortunately double disk failure or single disk failure combined with other corruption can and does happen, as I found out much to my dismay. I loaded all my photos onto the unit shortly after my return and began sorting through them and posting new sets every day or two. After a week or so of working on photos off and on, I started to see some issues reading certain images. I checked my Windows event log and found a whole bunch of disk-related errors essentially saying my 4big drive was corrupted and it needed to be scanned for errors. I rebooted shortly after and a disk scan ran automatically. Though I've never had much faith in Windows' chkdsk utility, I soon found out that it's even worse to run it on a RAID.

Chkdsk ran, finding a lot of errors, and "correcting" most of them. When I got back in to Windows, I found some missing images (by examining the disk scanning log I was able to see what files it had found "orphaned" or otherwise identified as corrupted and attempted to fix). Fortunately the files lost were fairly minimal, and things seemed to be working ok. That didn't last long though. Within hours my 4big unit became unreadable. Windows still recognized it but it no longer showed the drive as being formatted and no data was accessible. I shut down the computer and the 4big unit and rebooted. The 4big began showing a series of varying configurations of warning and error lights. Eventually it just showed a total failure, meaning at least 2 disks has failed. I was shocked, but not yet panicked. I've seen odd - but correctable - issues like this before (not with this particular unit).

I contacted Lacie technical support, beginning an odyssey of 2 weeks of back and forth during which I received very little useful advice beyond resorting to extremely expensive data recovery options. The first tech I got essentially told me that the indicator lights on the unit didn't necessarily indicate anything useful and that the only way to determine if it was an actual drive failure vs. say a controller failure would be to reinitialize the unit. This meant losing all the data for certain, so it really was not an option. One option I would have liked them to offer would be to ship an identical unit without the drives so I could switch my unit's drives into it and test there. If it still failed, it would indicate it was a drive issue rather than controller or other hardware problem. But they weren't willing to do that or provide any other remedy besides shipping the drive to them in Oregon for data recovery at a cost of thousands of dollars.

At this point I was fairly resigned to having to do data recovery, but still hoped for some cheaper alternatives. I didn't relish the idea of shipping the unit hundreds of miles up to Oregon so I started looking into local data recovery options. I looked around on Yelp and did other general searches online. The world of professional data recovery is rather mysterious and almost universally very expensive. Your average computer shop can sometimes handle very basic recovery, but I can do the same things they can at home, so that wasn't an option.

After a bit of research I came up with a list of local options and started calling to get an idea of rates, turnaround time, etc. The first company I called, Hard Drive 911 Data Recovery, was actually extremely helpful and although I didn't end up going with them, I would still recommend them if only on the basis of their extremely informative and frank reps. The fellow I spoke to told me in plain terms the cost of recovery (minimum $1000 for a RAID 5 with 4 drives, up to $9000!) and the up-front inspection fee (others offer free inspection, 911 credits the inspection fee toward a recovery if you decide to go ahead). Their costs did not end up being particularly more expensive than many other options (this is a shockingly expensive service overall, no matter where you go), but their rep was far more helpful. We had a very candid discussion about the costs and value of professional data recovery and he made clear that they want their customers to feel like the service is worth the cost. He gave me a lot of information and links to resources online to attempt my own recovery, and advised me to get back to them should I want to pursue professional recovery options.

I spent the next 2 weeks attempting recovery myself. My very first task was to image all 4 1TB disks. This takes a very long time to do in pure RAW mode, 1:1 copying (all 1TB of data), so that took up several days in itself. Once I had the RAW images I could attempt recovery on them, rather than further risking the originals, which I wanted to preserve for others to work on should that become necessary. Note that the simple process of attempting to create images on my basic home setup through regular SATA could have caused further damage to the disks, so there was some risk inherent in what I was doing. However I didn't hear any unusual noises from the drives and didn't suspect physical damage, so I felt it was safe to do.

Over the next weeks I tried a number of software tools to recover the data. The biggest challenge turned out to be figuring out the RAID parameters so that the data could be actually be accessed properly. Because RAID distributes data across all the disks, including "parity" (redundancy) information, it can't be read like a normal disk; you need to use special RAID emulation approaches in software, and you need the correct RAID parameters. I didn't have these, and Lacie was extremely reluctant to provide this information to me, which was another frustration I feel was unnecessary. I eventually escalated my support request to a senior staff member who did provide this information, but it ultimately did not help.

In the end I was unable to recover any meaningful data myself. I feel like I was pretty close, I got to the point of being able to list a lot of files and actually see meaningful file names, etc. But the actual data contents wasn't correct, and I suspect I either didn't have enough information on the RAID parameters, or (perhaps more likely) there was corruption in the RAID data model itself. It was time to think seriously about professional data recovery again.

By this time it was almost a month since the failure had occurred. The vast majority of my important data, including 5+ years of photos, 10+ years of documents and project files, and much more was unavailable during this time. I didn't realize it until several weeks in, but it was really affecting my happiness and overall mood, but at a fairly subconscious level. This is not surprising, but as I said I didn't even realize it until after several weeks. I knew I had to get the data back, and that professional data recovery was likely the only option.

Lacie operates their own in-house recovery service called D2; in fact Lacie support initially indicated I was required to use them in order to maintain my warranty (I later found out there were several other authorized providers). I actually had a great back-and-forth with Patrick, the manager there, in which we discussed the recovery options and costs. Since it's an in-house service, they can also perform warranty service, and their costs were lower than most local options. They also offer free 2-way shipping and free evaluation. The service sounded pretty good. But something in me balked at the idea of giving the same company potentially thousands more of my dollars just to fix a piece of hardware that I felt shouldn't have failed in the way it did anyway (not to mention my relatively negative experience with their tech support earlier). I initially asked to proceed but, after not receiving a response for a few days (they did eventually send me a shipping label), I decided to try an evaluation with a local company first.

I had done price comparison and reputation research on a number of local companies. One of the more well-known companies is DriveSavers in Novato. I had initially steered clear of them, partly on the basis of some negative Yelp reviews. Eventually I did call them and ended up having a very nice conversation with one of their reps. We even discussed the Yelp reviews, and having been on the other end of some negative Yelp reviews for businesses I work with, I sympathized with where she was coming from. They were local, in Novato, I could drop off the drive on my way to or from work in Penngrove, and they could do a free analysis within a couple days. So it seemed like a worthwhile option. Their pricing was also comparable with other local options, and their high-end ($4410) was lower than several other local options, so in the worst case scenario I'd pay less than with the same problem at another vendor. The high end of the range usually corresponds with significant *physical* damage and at this point I didn't think that was the case; after all I felt pretty close to recovery with my own simple tools. With the slow response from D2, I decided to go ahead with it.

I dropped the drive off and waited... A few days later I got a call back and they said they estimated they could recover 90% of the data and the cost would be between $3900 and $4400, the maximum they said it could cost before they even looked at the unit. At this point I was tempted just to get the drive back as I was fairly sure the level of corruption didn't justify the cost. I called DriveSavers to talk to the tech working on my recovery, asked him a few questions, and felt his answers were rather condescending and uninformative.

He said there was physical damage on one of the drives, which I had also encountered when I imaged the drives originally. 9 sectors to be precise, which is not much on a 1TB drive. I told him I had some images that were made prior and offered to bring them by if it would help, but he brushed that off as if it was ridiculous that my images taken earlier might be of use. Maybe he's right, but his reaction was not particularly nice. Then I asked him why, if it was just 1 drive with physical corruption, it was going to cost so much (and theoretically be so difficult) to recover the data; after all, isn't RAID5 *meant* to recover from a single drive failure? He barely answered the question, saying only that there was other unspecified corruption, possibly due to chkdsk, and that it was necessary to do advanced recovery.

Thinking back to the Yelp reviews, I recalled some indications there of dubious information from DriveSavers tech support as well. Tales of "motor failure" and "intermittent hardware failure" sounded similar to the lines I was getting. In several of the Yelp reviews people even got their drives back and found other, cheaper ways to recover, discovering that the problem indeed was not as dire or complex as they indicate (for example a simple power supply replacement). That being said I felt by this time that I'd exhausted most of my own simpler diagnoses and recovery options, so even though I didn't trust what I had been told, I wasn't really clear there were many other options.

Then I called Patrick at D2 to see if he felt he could do better on the recovery if I sent it up to him; after all at this point I was still within the free eval at DriveSavers. He felt - and I tended to agree, though reluctantly - that if DriveSavers already have the drives and had them dismantled in their labs, that having them reassemble and box up so I could ship to D2 would incur further risk and I might not get the data back (or not recover as much).

This is where the mystery of the data recovery industry plays into their hands, theoretically justifying the high costs. After all, they're the experts, they can tell you whatever they want, and who are you to argue? The uncertainty customers feel, and the fear of losing data, can justify almost anything in that moment. The lack of information and transparency, the difficulty in trusting these companies, is the biggest problem I have with the whole industry, to this day. I still felt like they weren't telling me the whole truth, but I didn't have any real facts to back that up, just my own experiences with drive failure in my time in IT. So ultimately I said yes, and the final bill was indeed $4410. I paid $4400 to get my data back. And you know what? It was worth it. Fully worth it.

But I don't trust DriveSavers, I don't trust D2 (who never gave me an actual maximum figure, they just said "I've never seen a recovery cost more than $2500", which is no kind of commitment), and I don't trust anyone else in this industry either, except maybe that guy I talked to at Hard Drive 911, who told me quite clearly that it was reasonable for me to try my own recovery first (once he confirmed I was aware of the risks).

The fact is I just don't know. It's possible that my drives were so damaged that it justified the cost. It sure didn't seem like it from any actual evidence I ever saw. And the reaction of the DriveSavers tech was not particularly confidence inspiring. It seemed like he was more interested in stifling questions than clearly informing me about the complexity - and thus justifiably high cost of - the recovery.

So I don't trust any of them, but I used their services because the alternative wasn't worth it to me. Even if they're full of crap and the recovery difficulty did not justify the cost (which I'm fairly certain of), it was still worth the price to me, and that's what matters. What matters more, though, is that I am going to do my best to never have to pay for something like this again.

As to whether I'd recommend DriveSavers or anyone else? Sure, if you have a really serious data loss scenario and the data is worth $1000s of dollars to you, by all means, do it. Not because it may be strictly necessary, but because the risks outweigh the potential cost savings. Just resign yourself to paying the maximum possible quoted amount. If you're ok with that, then it's the right way to go, no questions. DriveSavers is essentially going to treat each recovery like it's the worst possible case, from the get-go, and in some sense they may not be wrong to do that. Sometimes you don't get a 2nd chance, and if they power up a drive and it fries itself before they even get a chance to do anything, they'd have been better off disassembling it from the beginning and running it in a clean room environment. Just be aware that this is going to be their approach and don't be fooled by the low-end of their cost scale; you will almost never pay that.

So on the other hand if your data is only worth a few hundred dollars, or nothing at all, then try to do it yourself, or take it to a local computer shop or trusted IT consultant and see what they can do. It's all about how much the data is really worth to you. If you have a medical problem that could be life threatening and it will only be determined in surgery whether that is in fact the case, you don't go to the bargain clinic to get opened up, you go straight to the best, because the risks outweigh the potential savings, plain and simple. I don't like this reality, but it's a simple truth, and it's what props up an industry of chronically bloated fees. I'm now a registered DriveSavers reseller so theoretically I have some investment in getting them more clients, but I will not be recommending them lightly. I may ultimately refer some people their way if I feel it's justified. But to tell you the truth I'd much rather spend my time and energy teaching people how to back up their data properly!

As this post is now very long, I'm going to continue in Part II: Backing Up Your Data - Software Recommendations and Strategy.

Oh noes!

Ladies and gentlemen, the unthinkable has happened (well, not so unthinkable really): my laptop has died. Sadly this may mean no chance of further blog or photo posts until I return. Given my stellar record so far it may not be a huge loss but its rather a bummer as I was about to put up a bunch of Bryce and Zion pictures. But alas fate has intervened.

I will of course be tinkering with my laptop trying to fix it but so far it does not look good. I'm posting this from my new phone and while the typing experience is surprisingly good for a 4" keyboard, there's still no way this could work.

Fortunately our trip is otherwise going quite well, despite some annoyingly persistent rain (it really does seem to be following us around). We're in Georgia now, heading down to New Orleans next.

So friends, wish me luck and email me if you want to keep in touch!

Delayed reaction

So obviously I'm a "little" behind in my blog updates on the road trip. Fortunately I'm starting to catch up now (yes, I know I've said this before - shhh!), and I'm also uploading photos from further on in the trip before the blog posts go up (partly because the photos go into the blog posts).

For photos I haven't been using my Photoblog as much as I expected, though I'll be uploading my favorite picture from each day there once I select them. In the meantime it's been much easier and faster just to upload to my Picasa and Facebook galleries immediately after culling and editing. So keep an eye on either of those links for photos to show up prior to a related blog post (could be as much as a few days). Right now I'm working on a batch of Zion National Park and Bryce Canyon pictures, then comes the Grand Tetons and Yellowstone. Those should all show up within the next few days, along with more perhaps. Related blog posts hopefully within the week.

Also, don't forget the occasional YouTube upload. I'm taking a fair amount of video on this trip and trying to upload those along with the photos for a given location. For example, have you seen The Glorious Moose? Or The Gauntlet of Mono Lake Flies? Not to be missed! (especially the moose - Katy took that one)

Sorry about the delays, and thanks to anyone who is still following along! ;-)