Problem with my Iomega StorCenter Pro 150d

philipgecko
philipgecko used Ask the Experts™
on
Hi there, houston....

All of a sudden our mapped drives have stopped working, we have this iomega nas box and cannot access the data on the device, the ip is 192.168.0.50 and when i go to \\192.168.0.50 and have a look at the data thats in there and its all gone!!!! there are 4 x 500gb drives in there and through the software interface it was configured to a RAID 5 configuration. I mean how the dickens could this have happened..???? i am so concerned that this data on here is gone as it was not backed up anywhere else.  I have spoken to Iomega support and they are useless and pretty much said that I have to go to their sister disaster recovery company that would charge something like £4000 to recover the data, the device is still under warranty and it feels like blackmail.  Is there anyone who can help me with this? i really am stuck!


Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2007

Commented:
not sure what choices you have other than another disaster recovery service.

 Gillware (www.gillware.com).   Their fee is (by data recovery standards) very reasonable ($378.99) and they have a "no recovery, no fee" policy.
  http://www.ontrack.com


I hope this helps !

Author

Commented:
I dont really want to go down this road unless I absolutely have to... Im based in the UK.  there must be a way to get the data off these drives somehow.... surely??
DavidPresident
Top Expert 2010

Commented:
Well, you could have lost 2 drives, or lost one drive and had bad blocks on surviving disks.  Lots of scenarios.

But it is a moot point, as it is down.  First, crack it open and run some diagnostics on each drive to see where things stand.  If 2 disks are totally dead, then give up and get your checkbook out, as you are quoted a reasonable price for this sort of thing.

If only one disk is bad, then it comes down to the drive signatures with a binary editor and figuring out what embedded O/S they used, and the RAID layout.   It is probably linux software-based, but no idea myself.    

I will reluctantly defined Iomega.  Remember, all disks have 100% probability of drive failure.  A warranty covers drive failure, not data recovery.  You won't find anybody that recovers your data for free in event of a drive failure, RAID is not a substitute for backing up.

If you can hook up one of the disks to a JBOD controller and get a binary editor and post the first 8KB or so of raw data from one of the disks, then I could likely identify the RAID methodology, which is vital to get your data back.
DavidPresident
Top Expert 2010

Commented:
Gillware and the others will NOT reconstruct a RAID set.  They will try to get files from the HDD.  In a RAID configuration, with parity and various chunk sizes, then this will pretty much be useless.   The software has no idea what blocks are already known bad to the controller, and will potentially recover blocks that are already invalid, causing data corruption.  

Ontrack will charge in the neighborhood of $2000 per disk if it is part of a raid set, and you will pay for all the disks in the set, so ontrack will probably charge same amount of money.

Author

Commented:
thanks dlethe thats really helpful and I will try and do this albeit that I have never done this sort of thing before :s, i might be a little while before I post back the results for you.....

Author

Commented:
just looked in the iomega event viewer and it has posted this:

0300 : 01/15/2009 22:57:49 : A Volume has failed.
0300 : 01/15/2009 22:58:35 : A Volume has failed

date and time is wrong i know

Author

Commented:
0300 : 01/15/2009 22:57:49 : A Volume has failed.
0300 : 01/15/2009 22:58:35 : A Volume has failed.
0203 : 01/15/2009 23:02:08 : RAID rebuild complete.
0201 : 01/15/2009 23:27:50 : RAID rebuild in progress; please do not shut down.
0201 : 06/30/2010 21:17:17 : RAID rebuild in progress; please do not shut down.

this is the complete log...

Author

Commented:
screen shot of status page attached.....
iomega-1.PNG
Top Expert 2007

Commented:
Sounds like it is still rebuilding.

Author

Commented:
i think that happened when i reseated one of the drives and it came up with this message but what perhaps is interesting is the fact that it confirms that there is a RAID present and that a volume has failed... which i presume is a drive...?

Author

Commented:
my gut reaction here is that the top drive has failed as its the only one that is flashing blue whilst all the other drives are solid blue.....
DavidPresident
Top Expert 2010

Commented:
Tips
 - every moment the device is powered up could potentially destroy more data.  So only power up one disk at a time
 - DO NOT USE WINDOWS IN ANY WAY, SHAPE, OR FORM.  Windows does not have ability to do a read-only mount, and can write to the disks
 - Download an ubuntu live CD, and get yourself a true non-RAID controller to use.  Be careful here, Jmicro, Intel, Sis and other chipsets have a fake-raid mode that can screw things up, so make sure you look carefully.  You don't want controllers writing signatures or anything.  
 - You could buy a 2TB disk drive very inexpensively, and then use dd to put the raw data from the surviving disks on a single 2TB drive, so it can be reconstructed by somebody that knows what they are doing, but dd (the native unix raw copy) will throw errors if you have unrecoverable blocks, so this whole process has the potential to be difficult if you don't have some commercial software designed to get around such things depending on the health of the disks.

But for now, do the Live CD, run the smartctl program and run the SMART and self-test programs on all disks to see where things stand, and then you should still be able to get a hex dump by entering
dd if=/dev/hda of=/tmp/rawdump0.bin bs=64k count=1

above will create a 64KB file called /tmp/rawdump0.bin in a ramdisk, which you can then FTP.  It assumes path of disk is /dev/hda,   Sorry this is not trivial and a learning curve, but it has potential to save you thousands of dollars.

Assuming that you can get the dumps, and only one disk is completely dead, then the data can probably be reconstructed.  But let be warn you that there is a reasonable possibility that it is going to be beyond your skillset. Realistically, mentally prepare yourself to spend money on some software and/or farming it out to somebody who knows what he/she is doing and has the hardware & software.  Still paying $500 - $1000 beats the heck out of paying 5-10X as much.   Also this prep work will let you know exact nature of the problem so that you can potentially just give up before spending $7500 or so on what may be a partial recovery.

DavidPresident
Top Expert 2010

Commented:
Let it run, don't touch it, since it started a rebuild then you probably just had a drive disappear from the RAID array, and now it is resyncing.  You got lucky.  :)

Author

Commented:
thanks ever so much I really do appreciate your recommendations and yes data recovery is a skillset definately above my station!!  I can see why companies can charge what they like..!!  I have just had a look at the event log and the latest is pasted below.  to me it says that it cant complete the RAID rebuild because a drive has failed, will this continue in a loop until the faulty drive is replaced? when i look at the nasbox the top drive is flashing blue where the other 3 are solid blue.  

0300 : 01/15/2009 22:57:49 : A Volume has failed.
0300 : 01/15/2009 22:58:35 : A Volume has failed.
0203 : 01/15/2009 23:02:08 : RAID rebuild complete.
0201 : 01/15/2009 23:27:50 : RAID rebuild in progress; please do not shut down.
0201 : 06/30/2010 21:17:17 : RAID rebuild in progress; please do not shut down.
0300 : 06/30/2010 21:17:33 : A Volume has failed.
0201 : 06/30/2010 21:47:30 : RAID rebuild in progress; please do not shut down.
0201 : 06/30/2010 22:17:40 : RAID rebuild in progress; please do not shut down.
0300 : 06/30/2010 22:18:09 : A Volume has failed.
0201 : 06/30/2010 22:47:55 : RAID rebuild in progress; please do not shut down.
0201 : 06/30/2010 23:17:56 : RAID rebuild in progress; please do not shut down.
0300 : 06/30/2010 23:18:11 : A Volume has failed.
0201 : 06/30/2010 23:48:02 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 00:18:09 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 00:18:21 : A Volume has failed.
0201 : 07/01/2010 00:48:09 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 01:18:24 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 01:18:25 : A Volume has failed.
0201 : 07/01/2010 01:48:26 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 02:18:33 : A Volume has failed.
0201 : 07/01/2010 02:18:35 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 02:48:52 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 03:18:33 : A Volume has failed.
0201 : 07/01/2010 03:18:53 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 03:48:54 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 04:18:50 : A Volume has failed.
0201 : 07/01/2010 04:18:58 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 04:49:10 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 05:19:22 : A Volume has failed.
0201 : 07/01/2010 05:19:23 : RAID rebuild in progress; please do not shut down.
0201 : 07/01/2010 05:49:33 : RAID rebuild in progress; please do not shut down.
0300 : 07/01/2010 06:19:27 : A Volume has failed.
0201 : 07/01/2010 06:19:40 : RAID rebuild in progress; please do not shut down.
Top Expert 2007

Commented:
Replace the bad drive and hope it rebuilds OK

DavidPresident
Top Expert 2010

Commented:
Replacing the drive may work, but do not get rid of it.  Also do not mount or try to use it in any way as this can create further corruption.   Looks like you are going to have a lot of additional corruption because volumes keep failing and going back online.  A drive is intermittent, and if a disk fails during a rebuild (you may have a 2nd bad drive, who knows), then all disks will be required.

So bottom line, if replacing that drive does not fix the problem, then this is not the sort of thing anybody is going to be able to walk you through, you are going to have to find somebody who has the software and experience to figure it out.  You obviously have hardware problems but no way to assess data corruption w/o running XOR parity tests at low level, using this intermittent disk (or disks).

Other than running some diagnostics and reporting hardware health, then you are going to have to ship the disks to somebody to do what you need.  It will take several days worth of work, so $1500/day USD is not unusual, but realistically 90% of that is automated software and maybe only a few hours of human time.  It will take a good 2 days to binary copy and reconstruct, and you are clearly being billed for that time.

There are no guarantees either. Remember that too, you may end up spending $2500 or so, just for them to tell you that it is a total loss.   Only alternative I can offer, is that you can try to find somebody, nudge, nudge, wink wink via a blog site that can be contracted to look at it by looking at their profile and dropping them a private message.  I am not saying that I or anybody in particular can or would do this.  You are in the UK, so it is a pain to ship, and are better off working with somebody locally.

Since this drive is intermittent then perhaps everything will be fine, but you could very well have massive data corruption, so no matter what, you should mentally prepare for an expensive lesson in the necessity for having a backup.

Author

Commented:
thank you, I just cant believe this has happened just like "that", was absolutely fine no indication whatsoever and then "dink" off.... I have ordered through iomega warranty a replacement drive which will arrive within the 3-5 days and I was going to see if it will rebuild.  I am assuming that the reason why the blue light is flashing next to the drive is because it has failed...  the nasbox is attempting to rebuild which is good news so I may get lucky....  the expensive route may not be an option so will have to try a different route......
DavidPresident
Top Expert 2010

Commented:
Well, look at the log, it happened on Jan 15th.  Did you replace a drive at that time?  Looks like answer is no, so a drive went offline then came back online.  Of course hindsight is 20:20, as we yanks say. So you had 6 months warning that a drive failed and it didn't recover properly.

As a matter of fact, are you 100% sure that this is RAID5?   How many GB usable was reported?   Seems to me that the behavior is consistent with RAID0.  The array goes offline when a disk goes offline, and rebuild only takes a minute or so.   This behavior is consistent with an array that does not have parity protection.

Rebuilds with a RAID5 should take several hours minimum.

Author

Commented:
when the guy at iomega said reset the web interface it affected the date and time so thats not a true reflection on what actually happened unfortunately, no drive has been replaced only reseated,  it was most definately RAID 5 and I honestly cant remember the total capacity, think it was 1.5TB with the RAID 5 setup.  and yes with hindsight things would have been done very differently.  

should i continue and get myself this JBOD controller and post the first 8kb of raw data using the binery editor..??

You sound very much like you know what you are doing, is this what you do for a living..?? ive been really impressed with your knowledge of this subject and it sounds like you have had plenty of experience of such instances....  Really appreciate your help btw
DavidPresident
Top Expert 2010

Commented:
You probably don't need a JBOD controller.  Disable RAID on your motherboard if you have it (but if you have a matrix controller or one that supports RAID, then you may not have a choice). Plug one of the disks into one of the ata ports you use to boot from.  Disconnect all other HDs if you have any, so just the CD/DVD and one of the HDDs in the iomega is installed.

Boot a LINUX live cd (this does NOT install O/S, it boots the kernel, gives you a ramdisk, so you can configure networking and inspect the HDD w/o mounting it, and save contents.  You could also save it to a USB flash drive if you don't have a site you can FTP the data to.

Then run smartctl (do man smartctl for manual), and run the -A dump, and the self-tests, and dump out first few blocks as I explained before.     I will look at it and asses.

Yes, I am in the biz. But i do not do data recovery full time.  I own a  company that writes storage diagnostic software and RAID + SAS/NAS appliance management software, and do recoveries as time permits, mostly for RAID configs, and then usually when somebody else is in over their head and they seek me out.   But for the record, I have never been inside the iomega, so the techniques I give you are for me to assess what you have and degree of difficulty.  It could  be an absolute no-brainer for me to get you going if I have in-house code or iomega uses one of the controller chipsets I already have code for.  It comes down to figuring out the metadata and RAID architecture and health.  

Author

Commented:
Hi there,

Iomega have sent through a replacement drive but its made little difference, see logs below..

0201 : 07/05/2010 12:18:17 : RAID rebuild in progress; please do not shut down.
0300 : 07/05/2010 12:41:49 : A Volume has failed.
0201 : 07/05/2010 12:48:28 : RAID rebuild in progress; please do not shut down.
0100 : 07/05/2010 13:15:22 : A drive was removed or has failed, please check the unit.
0102 : 07/05/2010 13:15:22 : A drive has an invalid partition.
0200 : 07/05/2010 13:15:23 : RAID volume degraded; please check the unit.
0102 : 07/05/2010 13:15:42 : A drive has an invalid partition.
0104 : 07/05/2010 13:15:48 : HDD has been repartitioned and prepared for use.
0103 : 07/05/2010 13:15:58 : A new drive has been inserted.
0203 : 07/05/2010 13:17:22 : RAID rebuild complete.
0201 : 07/05/2010 13:18:43 : RAID rebuild in progress; please do not shut down.
0300 : 07/05/2010 13:42:12 : A Volume has failed.
0201 : 07/05/2010 13:48:50 : RAID rebuild in progress; please do not shut down.
0201 : 07/05/2010 14:18:56 : RAID rebuild in progress; please do not shut down.
0300 : 07/05/2010 14:42:13 : A Volume has failed.
DavidPresident
Top Expert 2010

Commented:
Contact them.  You probably have some other issue as root cause, like intermittent power.  At this point, looks like the entire unit should be replaced.

Author

Commented:
yes and its under warranty so i can get another although the exact model is obselete now, you think I can take the drives out and put them straight into the new unit..?? Wont it think its a new volume and start creating a new array..??
DavidPresident
Top Expert 2010

Commented:
Best to ask them if there is binary compatibility.  I know that they use a LINUX kernel, but that does not mean that there isn't some metadata at beginning and they didn't muck with that.   Be sure to write down serial numbers and full config info, including firmware revision.   Make sure when you ask them if there is full compatibility that they consider firmware revisions.

Personally, I would source a 1.5 - 2TB disk drive, and do a full backup.  It won't be RAID protected, but at least you will have an archive copy that you can take to a family member's house or office or something so you have a disaster recovery copy.

Author

Commented:
I think this is over my head... you can say no to this if you want to and I wouldnt blame you if you did but i was wondering if I got this unit over to you would you be able to have a look..? I would compensate you naturally.  I can get the unit over to you next day via dhl....  you interested..?? you can email me at philipgecko@gmail.com if you like to keep this private.

Thanks

phil

President
Top Expert 2010
Commented:
Sure,  I will drop you an email.  

It is fair game, in rules of EE, when you get to a point where you need hands-on consulting talent because the nature of the problem is too complicated to be placed in form of a question and answer, but you aren't supposed to just publish your email address.   The point of EE is that everything gets in the KB, but clearly this is an issue where the answer, to pay somebody $6000+ USD to recover & reconstruct a broken RAID is a correct answer, but not the "best" answer.  Sometimes a better question, is, "Are there any experts with the skillset to recover this data who will charge a more fair rate, based on human time rather than project time"?

Click on my profile to get an address where you can reach me through a blog site.    

Not sure of proper way to close this out, so I would contact moderator by flagging this thread and asking for guidance.  We don't want to break rules, and we experts like to stay anonymous as we get plenty of emails already :)

Author

Commented:
haha yes quite...!!!  well I think I will close this for you as you bloody well deserve the meagre 500 points... if it were up to me id probably give you a few thousand!!!!  Ill have a look on your profile and contact you through your blog then... many thanks...

Author

Commented:
dlethe you have been one of the most helpful experts I have come across, for that I thank you ... just goes to prove what an excellent community forum this is...

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial