Solved

VMWare ESXi v4.1.0 hardware sensors on RAID 10 rebuild on a Dell PowerEdge 2950 with a Perc 5i RAID controller.

Posted on 2013-06-06
16
2,376 Views
Last Modified: 2016-12-08
I'm running VMWare ESXi v4.1.0 on a Dell PowerEdge 2950 with a Perc 5i RAID controller.  Member HDD 5 failed on a RAID 10 array.  I put in a new SATA drive and it started to rebuild.  I'm monitoring via the hardware sensors on the VMWare vSphere client.  It still shows it rebuilding.  I refreshed the sensors but have not reset all sensors yet.  It seems that it's still rebuilding for disk I/O is slower than normal.  Each member (4 members) is 1 TB.  It's been a little over 24 hours now...  Is it possible or seem normal that the rebuild is still going on?  Do the VMWare ESXi hardware sensors refresh and update automatically?  Also, I have a DRAC board but it just shows the drive being placed into the server.  It doesn't show rebuild status.

Now,

I just checked the VMWare hardware sensors and I don't see rebuild anymore... I see a parity check status.  Does the parity check come after the rebuild status?
0
Comment
Question by:gopher_49
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 6
16 Comments
 
LVL 120

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 39228216
RAID rebuild can take some time to rebuild the RAID array. Yes, after rebuild, comes parity checking.

Are you using the Dell version of ESXi, and you would be better checking using Dell OpenManage, here's my EE article, of how to install on ESXi.

HOW TO: Use Dell OpenManage Server Administrator to Create and Manage a Virtual Disk (RAID Array) on a Dell PowerEdge Server with VMware vSphere 5.0, ESXi 5.0 installed.
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39228611
yes after rebuild.

Cheers,
SA
0
 

Author Comment

by:gopher_49
ID: 39228815
Great.  Sounds like the sensors are showing good results.  I'm using the Dell ESXi v4.1 image.  I recently found another article in regards to installing Dell OpenManage Server.  I plan to install it in my lab and if all goes well I'll do it in production.  

Thanks!
0
Webinar: Aligning, Automating, Winning

Join Dan Russo, Senior Manager of Operations Intelligence, for an in-depth discussion on how Dealertrack, leading provider of integrated digital solutions for the automotive industry, transformed their DevOps processes to increase collaboration and move with greater velocity.

 

Author Comment

by:gopher_49
ID: 39228941
It's still on the parity check and we're taking a huge performance hit.  I'm surprised we're taking such a performance hit during the rebuild of the array...
0
 
LVL 120
ID: 39228965
Any RAID rebuild, affects performance, until the procedure is completed.
0
 

Author Comment

by:gopher_49
ID: 39229006
It's been going on for 2 days now... Horrible...  I'll open up a different ticket in regards to selecting a different RAID controller / drive selection for my next install to help reduce the impact during a RAID 10 rebuild...
0
 
LVL 120
ID: 39229058
RAID rebuild depends on size, speed of disk, type of disk, number of disks, controller, and during this time of rebuild, you are also at high risk of another disk failure, if all disks were purchased at a similar time.

think about RAID types, hot and cold spares, and more importantly have good valid backups.
0
 

Author Comment

by:gopher_49
ID: 39229168
All of my backups are good and saved to another mirror/appliance and also offsite.  I take application level backups and VMWare snapshots once a week.  So, we're good there, however, in the future I want a stronger RAID controller that doesn't take such a hit... I would also like drives to rebuild quicker... This is just too long of a rebuild and too much of a performance hit.
0
 
LVL 120
ID: 39229205
I hope you do not use VMware Snapshots as Backups!?

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

Snapshots are not backups, and if you have VMs running on snapshot disks, there is a performance overhead, and VMs can struggle.
0
 

Author Comment

by:gopher_49
ID: 39229416
I use application level backups 6 days a week.  Once a week I shoot off a VMWare backup using the VMWare backup/storage API.  I should of not used the word 'snapshot'.  But.. Both the application level backup and the VMWare backup save to a different appliance that is a mirror.  They are also sent offsite to another RAID 10 array.  My backup design is solid.
0
 
LVL 120
ID: 39229792
Just make sure you test your "Backup Design is solid".
0
 

Author Comment

by:gopher_49
ID: 39230678
I normally do with VEEAM Sure Backup...  This environment doesn't not have that... I do randomly download SQL and various documents to make sure they restore correctly...  Sometimes I even mount their Exchange in a recovery group.
0
 

Author Comment

by:gopher_49
ID: 39233105
The rebuild and parity check is complete, however, I'm having serious disk I/O performance issues still.  My exchange server event log even mentioned the below...  Users say every application accessing the file is system extremely slow...  Of course this makes sense since all VMs are sharing the same RAID 10 array.. But...  Something is wrong for sure.  The array controller is not reporting anything via the hardware sensors in ESXi - vSphere client..  Any idea?  Should I install Dell OpenManager?

Event Type:      Warning
Event Source:      ESE
Event Category:      Performance
Event ID:      507
Date:            6/9/2013
Time:            9:01:25 AM
User:            N/A
Computer:      BACKEND1
Description:
Information Store (2880) A request to read from the file "\BACKEND1\$dd623858-6171-4a2e-bec0-79ef2e870fcb\$CopySrc000000005564EC3A$" at offset 1441792 (0x0000000000160000) for 65536 (0x00010000) bytes succeeded, but took an abnormally long time (70 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.
0
 
LVL 120
ID: 39233114
Check the VM is not on a snapshot disk?
0
 

Author Comment

by:gopher_49
ID: 39233207
It might be right this second for its being backed up..  But...  These symptoms existed when backup where not running.  I'll see if the snapshot is being removed when its done.  All VMs are having performance issues.  I'll check the snapshots.
0
 

Author Comment

by:gopher_49
ID: 39233598
I just checked and snapshots where not present.  I'll see what users say in regards to performance tomorrow when they log on..  I'm working remote so maybe the performance I'm noticing aren't as bad as they truly are.
0

Featured Post

Simple, centralized multimedia control

Watch and learn to see how ATEN provided an easy and effective way for three jointly-owned pubs to control the 60 televisions located across their three venues utilizing the ATEN Control System, Modular Matrix Switch and HDBaseT extenders.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
ARP not working as expected 11 80
Creating 2 network Adapters via PowerCli 7 38
Redundant Power Supplies failing? 8 52
adding RAM to VMware VM 18 49
If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
This article outlines why you need to choose a backup solution that protects your entire environment – including your VMware ESXi and Microsoft Hyper-V virtualization hosts – not just your virtual machines.
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question