Solved

VMWare ESXi v4.1.0 hardware sensors on RAID 10 rebuild on a Dell PowerEdge 2950 with a Perc 5i RAID controller.

Posted on 2013-06-06
16
2,318 Views
Last Modified: 2016-12-08
I'm running VMWare ESXi v4.1.0 on a Dell PowerEdge 2950 with a Perc 5i RAID controller.  Member HDD 5 failed on a RAID 10 array.  I put in a new SATA drive and it started to rebuild.  I'm monitoring via the hardware sensors on the VMWare vSphere client.  It still shows it rebuilding.  I refreshed the sensors but have not reset all sensors yet.  It seems that it's still rebuilding for disk I/O is slower than normal.  Each member (4 members) is 1 TB.  It's been a little over 24 hours now...  Is it possible or seem normal that the rebuild is still going on?  Do the VMWare ESXi hardware sensors refresh and update automatically?  Also, I have a DRAC board but it just shows the drive being placed into the server.  It doesn't show rebuild status.

Now,

I just checked the VMWare hardware sensors and I don't see rebuild anymore... I see a parity check status.  Does the parity check come after the rebuild status?
0
Comment
Question by:gopher_49
  • 9
  • 6
16 Comments
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 39228216
RAID rebuild can take some time to rebuild the RAID array. Yes, after rebuild, comes parity checking.

Are you using the Dell version of ESXi, and you would be better checking using Dell OpenManage, here's my EE article, of how to install on ESXi.

HOW TO: Use Dell OpenManage Server Administrator to Create and Manage a Virtual Disk (RAID Array) on a Dell PowerEdge Server with VMware vSphere 5.0, ESXi 5.0 installed.
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39228611
yes after rebuild.

Cheers,
SA
0
 

Author Comment

by:gopher_49
ID: 39228815
Great.  Sounds like the sensors are showing good results.  I'm using the Dell ESXi v4.1 image.  I recently found another article in regards to installing Dell OpenManage Server.  I plan to install it in my lab and if all goes well I'll do it in production.  

Thanks!
0
 

Author Comment

by:gopher_49
ID: 39228941
It's still on the parity check and we're taking a huge performance hit.  I'm surprised we're taking such a performance hit during the rebuild of the array...
0
 
LVL 118
ID: 39228965
Any RAID rebuild, affects performance, until the procedure is completed.
0
 

Author Comment

by:gopher_49
ID: 39229006
It's been going on for 2 days now... Horrible...  I'll open up a different ticket in regards to selecting a different RAID controller / drive selection for my next install to help reduce the impact during a RAID 10 rebuild...
0
 
LVL 118
ID: 39229058
RAID rebuild depends on size, speed of disk, type of disk, number of disks, controller, and during this time of rebuild, you are also at high risk of another disk failure, if all disks were purchased at a similar time.

think about RAID types, hot and cold spares, and more importantly have good valid backups.
0
 

Author Comment

by:gopher_49
ID: 39229168
All of my backups are good and saved to another mirror/appliance and also offsite.  I take application level backups and VMWare snapshots once a week.  So, we're good there, however, in the future I want a stronger RAID controller that doesn't take such a hit... I would also like drives to rebuild quicker... This is just too long of a rebuild and too much of a performance hit.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 118
ID: 39229205
I hope you do not use VMware Snapshots as Backups!?

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

Snapshots are not backups, and if you have VMs running on snapshot disks, there is a performance overhead, and VMs can struggle.
0
 

Author Comment

by:gopher_49
ID: 39229416
I use application level backups 6 days a week.  Once a week I shoot off a VMWare backup using the VMWare backup/storage API.  I should of not used the word 'snapshot'.  But.. Both the application level backup and the VMWare backup save to a different appliance that is a mirror.  They are also sent offsite to another RAID 10 array.  My backup design is solid.
0
 
LVL 118
ID: 39229792
Just make sure you test your "Backup Design is solid".
0
 

Author Comment

by:gopher_49
ID: 39230678
I normally do with VEEAM Sure Backup...  This environment doesn't not have that... I do randomly download SQL and various documents to make sure they restore correctly...  Sometimes I even mount their Exchange in a recovery group.
0
 

Author Comment

by:gopher_49
ID: 39233105
The rebuild and parity check is complete, however, I'm having serious disk I/O performance issues still.  My exchange server event log even mentioned the below...  Users say every application accessing the file is system extremely slow...  Of course this makes sense since all VMs are sharing the same RAID 10 array.. But...  Something is wrong for sure.  The array controller is not reporting anything via the hardware sensors in ESXi - vSphere client..  Any idea?  Should I install Dell OpenManager?

Event Type:      Warning
Event Source:      ESE
Event Category:      Performance
Event ID:      507
Date:            6/9/2013
Time:            9:01:25 AM
User:            N/A
Computer:      BACKEND1
Description:
Information Store (2880) A request to read from the file "\BACKEND1\$dd623858-6171-4a2e-bec0-79ef2e870fcb\$CopySrc000000005564EC3A$" at offset 1441792 (0x0000000000160000) for 65536 (0x00010000) bytes succeeded, but took an abnormally long time (70 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.
0
 
LVL 118
ID: 39233114
Check the VM is not on a snapshot disk?
0
 

Author Comment

by:gopher_49
ID: 39233207
It might be right this second for its being backed up..  But...  These symptoms existed when backup where not running.  I'll see if the snapshot is being removed when its done.  All VMs are having performance issues.  I'll check the snapshots.
0
 

Author Comment

by:gopher_49
ID: 39233598
I just checked and snapshots where not present.  I'll see what users say in regards to performance tomorrow when they log on..  I'm working remote so maybe the performance I'm noticing aren't as bad as they truly are.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
Teach the user how to install vSphere Update Manager  Console to Windows system:  Install vSphere Update Manager: Configure vSphere Update Manager plug-in in vSphere Client: Verify vSphere Update Manager settings in vSphere Client:
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now