Esxi 5.5 "virtual machine disk consolidation is needed"

Hi all,

Here's our issue. We have two servers.

Server A (Named CLOUD1)
ESXi 5.5 U2 (HP Build)
DL385 G2
32GB RAM
P400 Array Controller, 256MB BBWC
We have Array controller caching set to 25% / 75% ratio, we also have disk caching enabled.
8 SATA III disks in a RAID1+0 array.


Each VM on this server runs in it's own store which is made up of 1 pair of disk (as per RAID arrangement above).

All was well with this server until we lost a disk in one of the pairs (STORE1) (since replaced disk and synced in the RAID). We moved the VM to another store (STORE2 \ another pair), we had an additional crash the following evening (ESX froze) so decided as this is a production VM best to move it to our secondary ESX server. We are concerned about moving the VM back to SERVER A until we know it is stable.

What we did:
Shut the VM down, copied the files to the backup VM server (which is the spec below). Powered on the VM but it is running extremely slow, the VM is Windows SBS 2011 and it took over an hour to boot to the Windows login, interactive with the VM from there on in is still very very slow. We have the client's OWA access to this box up but again access to that is very slow. We've granted the VM 8GB of RAM and 1*4 vCPU.

Backup VM Server Hardware:
Server B (COLOMG01)
ESXi 5.5 (not Update 2 as per Server A).
RAID5 (yes I know we should really not have R5, it's on the list of tasks to do)
Any other VM's on this box run fine.

Now for the big question:
On the backup VM server this VM is now running we have in vSphere a message stating "virtual machine disk consolidation is needed".
As this is an Exchange (well SBS 2011) box I'm concerned if I do the "consolidation" option under Snap Shot Manager that it will revert back to some old version of the VM or version of the virtual hard disk which wont contain their current Exchange database and respective email data. I'll attach a screenshot of the vhd's we can see for this VM once I can get onto the box (should be in the next hour). My other concern is with the speed that it's currently running at I dont want to kick off the consolidation and it take the machine offline (due to high disk activity) for days as it's barely accessible now through OWA with a lot of lag and time outs.

One thing that concerns me is that the VM has moved from a new version of ESX. Do we need to do "upgrade virtual hardware"? I've had bad experiences of VM's not booting after doing that. We're currently installing VMWare Tools onto the VM to see if that helps with the operating speed but my understanding is that is more for graphical interfacing with the VM, no?

Really stuck here and I'm due to travel tomorrow so I'm hoping some of you ESX \ VMWare guru's can get us out of this mess.

Regards,

Leigh.
ProtecITAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay lets' park the virtual machine hardware version for the moment, and deal with the snapshot issue at hand...

Can you post a screenshot of the datastore, and we will look at the VM and it's snapshot, and how we deal with that!
0
ProtecITAuthor Commented:
Thanks Andrew, I see your name popping up all over the place for VM bits so Im sure you'll be able to help.

I'll get onto that now, might be a while as server is sooooo slow.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If the VM is running on a snapshot, the OS will be slow.

I need a screenshot of the ESXi datastore folder.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

ProtecITAuthor Commented:
While I'm getting those screens, what do you make of this? It's from another forum I cam across.

"If you delete the snapshots you will go back to the original and lose all of that data changes from time x to time y. You want to consolidate or you will lose data"

The VM is running so slow it's pinging back at around 200-300ms on the LAN!
0
ProtecITAuthor Commented:
I'm not sure it make a difference but I believe Trilead caused these snap shots.

Still working on those screens for you....
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Please read my EE Article about Snapshots, I've written quite a lot on the subject....

HOW TO: VMware Snapshots :- Be Patient

Yes, ALL backup applications, can leave VMs running on a snapshot..... your Admin Daily duties to check this!!!
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'm actually dealing with another snapshot question here, also:-

http://www.experts-exchange.com/questions/28693265/disk-consolidation-vmware-5-0.html#a40857658

Please follow my instructions....otherwise, you could get into "Snapshot Hell!", VM data could be lost or at worst the VM will be corrupted.
0
ProtecITAuthor Commented:
Thanks, I'll have a read through and follow progress on that one as well.
0
ProtecITAuthor Commented:
Finally got the screen. Please note our clients name (the company we host the server for is called Delta, that's not a reference to any file types etc).
VM-StoreFiles.jpg
VM-In-Vsphere.jpg
0
ProtecITAuthor Commented:
the VM is using this for it's vHDD

[RAID5-3x1TB] DR BACKUPS FROM CLOUD1/DELTA-SERVER/DELTA - 2011 SBS STD (STORE1) - 192.168.30.)/DELTA - 2011 SBS STD (STORE_) - 192.168.30.__)-000003.vmdk
0
ProtecITAuthor Commented:
Not sure if this helps but here's the setup of the VM.
VM-Setup.jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, from what I can see there is a VM, which has two snapshots attached, e.g. 0002 and 0003.

these are approx...64GB and 33GB, a total of 100GB worth of changes which need to be merged into the Parent disk.

This merge is going to take several hours, it could take  seconds, minutes, hours, days or weeks to complete, this depends on the storage system, and how fast the storage is.

Once the merge has started IT CANNOT be stopped, any messing, cancelling, restarting the host, VM, can cause the VM to be corrupted, and trashed....

so, sit tight, grab a coffee, go and watch a movie etc

Can you tell me how much storage is free ? and we will continue, with the next steps you must perform and try...
0
ProtecITAuthor Commented:
Thanks Andrew.
 
Couple of questions.

1 - Will performing the merge revert them back to old Exchange data?
2 - Can the progress be seen in VSphere Client?
3 - Do we leave the VM machine (guest) on for this process or shut it down?

As for how long it will take, here's the spec of the server:
Server is a DL120 G7 will 3 x SATAIII Drives in RAID 5 on a Smart Array B110i RAID Controller. 2 other VM's running but neither will be being access \ written to (very low) (one is a Win 7 VM running Sage, the other a Windows 2008 R2 which is just an machine we use to RDP onto to then gain access to other VM's in the COLO).  You think it will be done by the morning if I kick it off now?

I assume you mean free disk space on the entire Store? It's a 3.64TB volume with 1.04TB free.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
1 - Will performing the merge revert them back to old Exchange data?

No, this only occurs, if you revert the snapshot, we are going to commit/merge the data from the snapshot child disk to the parent disk.

2 - Can the progress be seen in VSphere Client?

Yes, but do not be alarmed, if the progress sticks at a certain %, or sticks at 95% for hours, it's still working.

3 - Do we leave the VM machine (guest) on for this process or shut it down?

It's best to shut it down, because it will take much longer to merge if running, AND performance will be worse. Yes, this does mean downtime for a while.

As for how long it will take, here's the spec of the server:
Server is a DL120 G7 will 3 x SATAIII Drives in RAID 5 on a Smart Array B110i RAID Controller. 2 other VM's running but neither will be being access \ written to (very low) (one is a Win 7 VM running Sage, the other a Windows 2008 R2 which is just an machine we use to RDP onto to then gain access to other VM's in the COLO).  You think it will be done by the morning if I kick it off now?

I assume you mean free disk space on the entire Store? It's a 3.64TB volume with 1.04TB free.

okay, that's enough storage space, SATA disks do not perform very well, and our slow..... so this is going to take many hours.....

BUT be warned, if you leave the VM running on a snapshot, the snapshots will only get larger, and larger and take longer and longer to merge, and IF the datastore runs out of space, the VM, and all VMs on the datastore will stop....

Do you wish to continue...?
0
ProtecITAuthor Commented:
Thanks Andrew.

Ok last question then I guess. If we did the merge with the VM powered up would the client still be able to access it (even if it took 2-3 days to finish) or do you think we'd be better (and safer) shutting it down?
0
ProtecITAuthor Commented:
VM has just finished installing VM Tools (took over and hour!), it's asking to reboot the VM. I guess it would be a good idea to do this now or just go ahead with your merge instructions?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The problem is, that performance is worse in the VM, it takes longer to merge, and can cause the VM to freeze, so it's your call, as to how you wish to proceed.

We generally recommend, issuing EMERGENCY downtime, and turn off the VM, otherwise, you will just be getting calls about poor performance, and not available, and when is this going to be back up!

At this point

1. Either Shutdown and Power OFF.

2. Continue, with power on (we would not recommend this course of action!)
0
ProtecITAuthor Commented:
Ok, we'll go with power off (the client has intermittent access to OWA and it's got to be fixed long term) and proceed with the merge.

Did you see my comment ref a reboot due to VM Tools just finishing it's install on the VM?
0
ProtecITAuthor Commented:
Ok VM is now shutdown. Please recommend next steps.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Did you see my comment ref a reboot due to VM Tools just finishing it's install on the VM?

Power off will be fine.

Okay......sit tight, and I'll hold you hand.....be patient....

Make sure no backup is going to start...

0. Shutdown the VM, and ensure it's powered off.
1.Right Click the VM, select Snapshot > Take Snapshot (no need to tick memory or quicese) Enter a name e.g. test
2. Wait 60 seconds at least.
3.  Right Click the VM, select Snapshot > Snapshot Manager > select the snapshot >  DELETE ALL

This will remove and merge ALL snapshots (consolidate/committ) into the parent disk....

BE PATIENT and WAIT....
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ProtecITAuthor Commented:
Just double checking - When you say "select the snapshot" in step 3, I select the snap shot I just created as there is one above it from an earlier time?

Screen attached.
snapshot1.jpg
0
ProtecITAuthor Commented:
Ok, it's now running. Will let you know in a few hours where it's got to.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Very Top snapshot....DELETE ALL.
0
ProtecITAuthor Commented:
Oh, I select Fix01 then clicked "DELETE ALL". What's the implication of this action? Will it still do the same thing?

The way I read the steps was to create the snap shot then when I went back into Snap Shot manager it automatically highlighted "Fix01" so assumed that was the correct selection.
0
ProtecITAuthor Commented:
I am online now and will be for the evening if you are still around to respond. Are you in the US?
0
ProtecITAuthor Commented:
It's done 30% in 1 hour. As rough calcs I'm looking at 23:30 this evening for completion.

15 mins  - 8.75%
0.5 hour - 17.5%
1 hour   - 35%
2 hour   - 70%
3 hours  - 105%
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
by selecting the snapshot you did just deletes that snapshot, you need to delete them ALL..and delete is the wrong word here, really, it's delete and merge contents with parent.

I'm in the UK....
0
ProtecITAuthor Commented:
Hi Andrew. Thanks for responding late into the evening.

It's finished I think, the task at the bottom of VSphere client has disappeared.

Now I get this in Snap Shot Manager. I guess I can boot the vm up?

Here's the file store as well.
snap-shots.jpg
File-Store-after-snapshot-delete-all.jpg
0
ProtecITAuthor Commented:
Ignore the snap shot JPG file that's the old screen shot. Here's the new one after the merge.
after-snapshot-delete-all.jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Congratulations, you are Snapshot FREE!!!!

Check regularly!!!

Yes Power-Up you are done...
0
ProtecITAuthor Commented:
It's booting, got to login in about  2-3 minutes.....about 57 minutes quicker than before! :)

Testing for current mail and a few other bits....looking good so far...will keep you posted.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, VMs on Snapshots zaps performance.

After each backup, you should check the VM is not running on a snapshot.

the easiest method is to check the datastore, or VM disk, is not -0000X.vmdk
0
ProtecITAuthor Commented:
Thanks Andrew. Good work!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.