RAID 1 Failed

Posted on 2010-09-02
Last Modified: 2016-12-08
First, a warning, I am a non-expert working for a smaller company with no real IT staff.  

We have two servers running running Red Hat Linux ES3 using RAID 1 on an Adaptec 2120S controller.  The 2nd server is actually set up to mirror parts of the first using RSync.  

One morning, the 2nd server became unresponsive, and was listing the following errors on the console:

SCSI disk error: host 2 channel 0 id 0 lun 0 return code = 8000002
Info fld=0x0, Current sd08:02: sense key hardware error
Additional sense indicates internal target failures
    I/O error: dev08:02, sector 39040

This repeated with many different sector numbers.  I could not copy the text, so this is a hand-written example.  After continuing these errors for a while, I saw the following messages, also repeated multiple times:


EXT3-fs error (device sd(8,2)) in ext3_orphan_add: IOfailure
EXT3-fs error (device sd(8,2)) in ext3_reserve_inode_write: IOfailure
EXT3-fs error (device sd(8,2)) in ext3_get_inode_bc: unable to read inode block

I eventually powered down the machine and re-started.  After the controller POST, I was presented with the message:
Following Arrays have Missing or Rebuilding or Degraded Members and are Degraded
Array#1-RAID-1    136.7GB    FAILED

I went into the Array Management Utility, and tried the "Verify Disk Media" option on both disks, which did result in remapping a few bad sectors, but I still got the same failed array message after reboot.  Since both disks were visible, and could be verified, I didn't assume that I needed a replacement.  Does that assumption seem correct?

Eventually I tried just pressing return to "accept the current configuration."  However, I then got an error "no boot filename received," and was asked for the system disk.  I tried using the Red Hat disk in rescue mode, but that only resulted in "no linux partitions found."

The data on this server is not critical, as this is the mirror, but it seems to me that I should be able to recover it.  I have zero training for this kind of work, unfortunately, and researching this topic has led to a variety of information that doesn't help me very much.  I may have a workable image of the server to restore from, but it is years old, and not my first option.  Does anyone have any suggestions?  

My first goal is to identify what went wrong, if possible, as the disks and controller appear functional to me.  If I can get the server back up without having to resort to re-installing/restoring, that would be great.
Question by:horvack
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4

Expert Comment

ID: 33590818
From your hand copy of error messages not clear, but:

Might be your  RAID  was runing in degraded mode from earlier HDD failure, and might be you mised it. And when second disk went off, RAID failed.  

You might try to remove one disk and try to boot, if it fails try other disk. If you succeed booting in one of them , then you may add new disk to array and try to rebuild array.

Author Comment

ID: 33591269

Sorry about the error messages, the screen was scrolling quickly, and non-responsive, so I tried to copy them accurately, but may have failed.  Given that I haven't been able to boot since, I don't suppose there's any way I could access the system log.

I did try removing one disk, rebooting, and then again with the other disk.  Neither booted successfully, and I got the same "no boot filename received" message.

I have since tried using cntrl+r to "enable/recover" the array in the RAID array management utiltity.  It lists the array as rebuilding 1% currently, and is going quite slowly.  This may be an all-night process, so I'll update the status when it is finished.

Accepted Solution

zzx999 earned 500 total points
ID: 33591508
From error log it would be useful to know how many disks failed. Important info is those lines:
>host 2 channel 0 id 0 lun 0.... just this disk or other too.
But now you can wait for rebuild result. If it succeds and lets you boot (then first do backup!!!) - add additional disk , and select it as spare. Then degrade raid (remove failed disk). It will rebuild again.
But if there was failure in both disks it may not rebuild.

In critical situation (witch is not your case as you can just install or copy other new server ) you'd had to  try reconstruct raid from both disks by reading full disks info and combining it. But thats expensive work. And it requires special knowledge,software,expirience and time (or just money :) ).
Manage your data center from practically anywhere

The KN8164V features HD resolution of 1920 x 1200, FIPS 140-2 with level 1 security standards and virtual media transmissions at twice the speed. Built for reliability, the KN series provides local console and remote over IP access, ensuring 24/7 availability to all servers.


Author Comment

ID: 33596640
OK, still in rebuild/verify, and only at 16%!  Wow, I guess this is going to be an all-weekend process...

Where would I look to find the error log that you are referring to?  If you mean the messages printed during POST, then both disk are listed, and they are both listed in the array management utility as well.  Or did you mean IF the rebuild works and I can boot from the array again?

Thanks for the suggestions, your point about doing an immediate back-up is well taken.   :)

Expert Comment

ID: 33596685
well, both variants. But if you saw that both disks were listed having errors, then I doubt reconstruction will be succesful...

Author Comment

ID: 33597311
No, I didn't see any errors, just that both disks were recognized during POST, and can be accessed in the array management utility.  That's why I thought both disks were still good.

Expert Comment

ID: 33597547
POST recognition does not mean that disks are good. They may have many  unreadable sectors. that can be detected only by disk check.

Author Comment

ID: 33618490
Well, after a long weekend of rebuilding/verifying, the server is back up.  Now to figure out what the problem was (and perform backup!)...

Thanks for your help.

Author Closing Comment

ID: 33618517
Didn't actually come up with the process that fixed the problem,  but was helpful in determining options with limited information.

Featured Post

Webinar: Aligning, Automating, Winning

Join Dan Russo, Senior Manager of Operations Intelligence, for an in-depth discussion on how Dealertrack, leading provider of integrated digital solutions for the automotive industry, transformed their DevOps processes to increase collaboration and move with greater velocity.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
UM7 Stop Sending packets with GPS Sensor 1 45
database connection error mysql stops 7 82
Clone drive for rack Server 4 94
Choosing the right Nas server 3 28
More or less everybody in the IT market understands the basics of Networking, however when we start talking about Storage Networks, things get a bit dizzier, and this is where I would like to help.
this article is a guided solution for most of the common server issues in server hardware tasks we are facing in our routine job works. the topics in the following article covered are, 1) dell hardware raidlevel (Perc) 2) adding HDD 3) how t…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

710 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question