Solved

VMWare ESXi v4.1.0 hardware sensors on RAID 10 rebuild on a Dell PowerEdge 2950 with a Perc 5i RAID controller.

Posted on 2013-06-06
16
2,347 Views
Last Modified: 2016-12-08
I'm running VMWare ESXi v4.1.0 on a Dell PowerEdge 2950 with a Perc 5i RAID controller.  Member HDD 5 failed on a RAID 10 array.  I put in a new SATA drive and it started to rebuild.  I'm monitoring via the hardware sensors on the VMWare vSphere client.  It still shows it rebuilding.  I refreshed the sensors but have not reset all sensors yet.  It seems that it's still rebuilding for disk I/O is slower than normal.  Each member (4 members) is 1 TB.  It's been a little over 24 hours now...  Is it possible or seem normal that the rebuild is still going on?  Do the VMWare ESXi hardware sensors refresh and update automatically?  Also, I have a DRAC board but it just shows the drive being placed into the server.  It doesn't show rebuild status.

Now,

I just checked the VMWare hardware sensors and I don't see rebuild anymore... I see a parity check status.  Does the parity check come after the rebuild status?
0
Comment
Question by:gopher_49
  • 9
  • 6
16 Comments
 
LVL 119

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 39228216
RAID rebuild can take some time to rebuild the RAID array. Yes, after rebuild, comes parity checking.

Are you using the Dell version of ESXi, and you would be better checking using Dell OpenManage, here's my EE article, of how to install on ESXi.

HOW TO: Use Dell OpenManage Server Administrator to Create and Manage a Virtual Disk (RAID Array) on a Dell PowerEdge Server with VMware vSphere 5.0, ESXi 5.0 installed.
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39228611
yes after rebuild.

Cheers,
SA
0
 

Author Comment

by:gopher_49
ID: 39228815
Great.  Sounds like the sensors are showing good results.  I'm using the Dell ESXi v4.1 image.  I recently found another article in regards to installing Dell OpenManage Server.  I plan to install it in my lab and if all goes well I'll do it in production.  

Thanks!
0
Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

 

Author Comment

by:gopher_49
ID: 39228941
It's still on the parity check and we're taking a huge performance hit.  I'm surprised we're taking such a performance hit during the rebuild of the array...
0
 
LVL 119
ID: 39228965
Any RAID rebuild, affects performance, until the procedure is completed.
0
 

Author Comment

by:gopher_49
ID: 39229006
It's been going on for 2 days now... Horrible...  I'll open up a different ticket in regards to selecting a different RAID controller / drive selection for my next install to help reduce the impact during a RAID 10 rebuild...
0
 
LVL 119
ID: 39229058
RAID rebuild depends on size, speed of disk, type of disk, number of disks, controller, and during this time of rebuild, you are also at high risk of another disk failure, if all disks were purchased at a similar time.

think about RAID types, hot and cold spares, and more importantly have good valid backups.
0
 

Author Comment

by:gopher_49
ID: 39229168
All of my backups are good and saved to another mirror/appliance and also offsite.  I take application level backups and VMWare snapshots once a week.  So, we're good there, however, in the future I want a stronger RAID controller that doesn't take such a hit... I would also like drives to rebuild quicker... This is just too long of a rebuild and too much of a performance hit.
0
 
LVL 119
ID: 39229205
I hope you do not use VMware Snapshots as Backups!?

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

Snapshots are not backups, and if you have VMs running on snapshot disks, there is a performance overhead, and VMs can struggle.
0
 

Author Comment

by:gopher_49
ID: 39229416
I use application level backups 6 days a week.  Once a week I shoot off a VMWare backup using the VMWare backup/storage API.  I should of not used the word 'snapshot'.  But.. Both the application level backup and the VMWare backup save to a different appliance that is a mirror.  They are also sent offsite to another RAID 10 array.  My backup design is solid.
0
 
LVL 119
ID: 39229792
Just make sure you test your "Backup Design is solid".
0
 

Author Comment

by:gopher_49
ID: 39230678
I normally do with VEEAM Sure Backup...  This environment doesn't not have that... I do randomly download SQL and various documents to make sure they restore correctly...  Sometimes I even mount their Exchange in a recovery group.
0
 

Author Comment

by:gopher_49
ID: 39233105
The rebuild and parity check is complete, however, I'm having serious disk I/O performance issues still.  My exchange server event log even mentioned the below...  Users say every application accessing the file is system extremely slow...  Of course this makes sense since all VMs are sharing the same RAID 10 array.. But...  Something is wrong for sure.  The array controller is not reporting anything via the hardware sensors in ESXi - vSphere client..  Any idea?  Should I install Dell OpenManager?

Event Type:      Warning
Event Source:      ESE
Event Category:      Performance
Event ID:      507
Date:            6/9/2013
Time:            9:01:25 AM
User:            N/A
Computer:      BACKEND1
Description:
Information Store (2880) A request to read from the file "\BACKEND1\$dd623858-6171-4a2e-bec0-79ef2e870fcb\$CopySrc000000005564EC3A$" at offset 1441792 (0x0000000000160000) for 65536 (0x00010000) bytes succeeded, but took an abnormally long time (70 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.
0
 
LVL 119
ID: 39233114
Check the VM is not on a snapshot disk?
0
 

Author Comment

by:gopher_49
ID: 39233207
It might be right this second for its being backed up..  But...  These symptoms existed when backup where not running.  I'll see if the snapshot is being removed when its done.  All VMs are having performance issues.  I'll check the snapshots.
0
 

Author Comment

by:gopher_49
ID: 39233598
I just checked and snapshots where not present.  I'll see what users say in regards to performance tomorrow when they log on..  I'm working remote so maybe the performance I'm noticing aren't as bad as they truly are.
0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
Teach the user how to convert virtaul disk file formats and how to rename virtual machine files on datastores. Open vSphere Web Client: Review VM disk settings: Migrate VM to new datastore with a thick provisioned (lazy zeroed) disk format: Rename a…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now