VM Dead!

My exchange box.  

Looks like its stuck since backup of the VM with acronis.
'estimate virtual machine disks consolidation space requirements' in the console.

Can do anything with the VM, shutdown or ether.

Help!
LVL 1
CHI-LTDAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CHI-LTDAuthor Commented:
checked the host and the vm folder has loads of virtual disks in there!!!
0
CHI-LTDAuthor Commented:
help.
exchange.jpg
0
CHI-LTDAuthor Commented:
no san.  local storage..
0
Zephyr ICTCloud ArchitectCommented:
Did you stop backup already (probably, just checking) ?

I'm assuming these are all delta files created by Acronis, to consolidate all these will be nearly impossible, check using this KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003638

I think the safest and "fastest" way to solve this (if consolidation is a no go) is to do a v2v of the Exchange VM so that you lose the delta files ... Info on v2v http://www.trainsignal.com/blog/vmware-v2v-migration

The v2v might take a while as well though.
0
CHI-LTDAuthor Commented:
no, it looks like acronis was backup up and not removing/consolidating the files...
0
CHI-LTDAuthor Commented:
we have had to reboot the host.
now i have a consolidation running on the vm.
loads of vmdks etc in the folder of the vm...
0
Zephyr ICTCloud ArchitectCommented:
Those vmdks are probably the deltas ... http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005049

Does the name of this vmdk have "delta" in it? ... Can't really tell from the screenshot.
0
CHI-LTDAuthor Commented:
Not for this VM.
But the acronis appliance VM does have delta listed under its folder.
e.g.vmdk.redo & delta.REDO

I cannot power on the acronis appliance either....
0
Zephyr ICTCloud ArchitectCommented:
Do you still have enough disk-space on the VMware server?
0
CHI-LTDAuthor Commented:
the ones in the picture (exchange vm) are like:
exchange2.jpg
0
CHI-LTDAuthor Commented:
yes, at least the host shows 2.* tb free.  The files in the vm folder total 774gb so should be fine, space wise..
0
Zephyr ICTCloud ArchitectCommented:
Yeah, those still look like snapshots to me...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
That server is an awful mess.

Many Snapshots, have not been committed.
0
CHI-LTDAuthor Commented:
manually delete the files?
0
CHI-LTDAuthor Commented:
yes i can see that...  Its consolidating the VM as i type...
0
Zephyr ICTCloud ArchitectCommented:
That server is an awful mess.

I concur, but didn't want to discourage too much ;)
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No, do not delete the files manually!

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would Be Patient, and wait and see what happens if anything after consolidation!

with that many snapshots, (too many snapshots 29+ and the VM will fail to start, I would recommend clone the existing disks to get a new virtual disk without any snapshots.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I wonder are you replicating the VM, every few minutes?

with that many snapshots performance will be affected in the VM.
0
CHI-LTDAuthor Commented:
it replicates daily (out of hours).
0
CHI-LTDAuthor Commented:
i did wonder why the VM was running slow, getting out of sync (time) emails regarding activesync...
0
CHI-LTDAuthor Commented:
How do i clone the VM?
I tried to use converter but it can't read the offline VM?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Performance of VMs on any snapshot, affects performance. High disk usage, affecting CPU.

How is it consolidating? before we move onto Cloning....
0
CHI-LTDAuthor Commented:
I have also noticed that another VM (bes) is of the same mess!!!
Loads of vmdks..
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Your Backup Product is not closing Snapshots correctly.
0
CHI-LTDAuthor Commented:
99% consolidating...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It could sit at 99% for a while...
0
CHI-LTDAuthor Commented:
Why hasn't the snapshot manager highlight/showed these files./.?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Snapshot Manager ONLY shows Snapshot's it's created!

These are created by the Storage API, that backup utilities use, I would recommend, using scripts or alarms to notify you of Snapshots, before you end up in Snapshot Hell!
0
CHI-LTDAuthor Commented:
Ok, should i start to see the duplicate vmdks be removed as it is at 99%?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Only at the end, when completed will you noticed if the snapshots have gone.

you may notice, some disappear, but it's like watching paint dry.

I would leave it a few hours, and come back.
0
CHI-LTDAuthor Commented:
Grr.

Not good.
0
CHI-LTDAuthor Commented:
Hancocka:
Quote mention: Try and Delete snapshots with the virtual machine powered off....?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
As I've written often, Be Patient, with Snapshot Deletion, Committing, Consolidating....and wait!
0
CHI-LTDAuthor Commented:
Sure.  Just wondered what your comment meant?  Delete them manually when VM off or the consolidation removes them?
0
CHI-LTDAuthor Commented:
I see acronis has created 3x additional drives...
acronis-cockup...jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'll explain, Power Off the VM, and Select Snapshot Manager - Delete (if they are present!)
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, these appliance, adds the disks of the VM to itself to backup!
0
CHI-LTDAuthor Commented:
Thanks.  

I can safely remove these from the appliance then?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
How is the snapshot progressing, do not touch anything whilst the merge is happening.
0
CHI-LTDAuthor Commented:
i have deleted the hard disks that attached to the vm disks and booted acronis.
exchange vm still @ 99%...  There was a lot of files there.
0
CHI-LTDAuthor Commented:
Thinking worst case, and having to delete the files...

Wouldn't the disk file within the hard disk 1 signify the live disk?  e.g my BES server:
[chi-host2_local_storage] chi-bessrvr_Jul 2, 2012/chi-bessrvr_Jul 2, 2012_1_B0A259C1-3782-4C1E-98A9-A73C410A96E8-000062.vmdk
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Worst case, we clone the disks, and the resulting cloned disk, will have no snapshots.
0
CHI-LTDAuthor Commented:
Using vcentre converter?
0
CHI-LTDAuthor Commented:
if so, 'unable to obtain hardware information for the selected machine'...
0
CHI-LTDAuthor Commented:
running v4.0.1
0
CHI-LTDAuthor Commented:
upgrading to v5
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
VMware Converter does not clone, it Converts!

Lets wait.....for the Snapshot to complete.

Please note, if you "mess" you increase the risk to damaging your virtual machine, resulting in corruption, and then you will have to Restore from Backup.
0
CHI-LTDAuthor Commented:
Sure.  Will wait.
How do you V2V?
0
CHI-LTDAuthor Commented:
We are only using vcentre client not server..
0
CHI-LTDAuthor Commented:
Yes 99..
0
CHI-LTDAuthor Commented:
So it will clone?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Possibly, you will need to wait, until it's finished.
0
CHI-LTDAuthor Commented:
Ok, it looks like its the replication side of things, not the backups...
0
CHI-LTDAuthor Commented:
and, while its consolidating i'm restoring it from backup (last nights) to the other host..
0
CHI-LTDAuthor Commented:
can i putty into the host and see whats happening exactly for the consolidation?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Not really, Be Patient!

you can tail logs, but you will not see a friendly progress indicator.....

how large are all the snapshots?
0
CHI-LTDAuthor Commented:
242 objects in the folder, total.
117 objects, about c380gb
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Personally, I would not have Consolidated, but too late now, you will have to wait, aborting can cause corruption.
0
CHI-LTDAuthor Commented:
What would have you done?

I think we will be using the restore from backup option!
0
CHI-LTDAuthor Commented:
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
CLONE-out the disk!

If you restore your Backup, will it not be out of date?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
that's a lot of snapshots, when was the last time the VM folder was checked?
0
CHI-LTDAuthor Commented:
it will be out of date, but as of 7.30 pm last night so we are happy...
a long time ago!
0
CHI-LTDAuthor Commented:
but we think our 2nd exchange box would have collated the inbound emails so should be fine...
4hrs to go on the restore!
0
CHI-LTDAuthor Commented:
Ok, i have another VM that is in the same predicament.
How should i clone it?  vcentre?
0
CHI-LTDAuthor Commented:
sorry, converter 5.1?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
vCenter is one option, but it can also be cloned at the command line via the console.

vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>


and WAIT, it could take a while to clone, and merge all the snapshots into a new virtual machine disk.
0
CHI-LTDAuthor Commented:
Have you ever come across my replication problem with acronis or veeam?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Many times with all products!
0
CHI-LTDAuthor Commented:
Hmm, i was hoping you would say 'use veeam vmware appliance'!?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Unfortunately, since the beginning, when the Storage API was created, third party backup utilities and VMware's own Backup products, can and leave snapshots!

It should be added to your daily VMware Admin routines, to check for snapshots, create email alerts and monitor.

I do not think it will be ever fixed.....
0
CHI-LTDAuthor Commented:
I think BE2010 VSS is also causing issues..
0
CHI-LTDAuthor Commented:
still @ 99%
0
CHI-LTDAuthor Commented:
ok, the other vm that had the same problem finally removed its snapshots after 4hrs.
there are still 2x vmdk files there though...

Consolidate?

exchange still restoring and consolidating...
0
CHI-LTDAuthor Commented:
here is the bes vm..
bes-vmdk...jpg
0
CHI-LTDAuthor Commented:
deleted the 2 old vmdk files as they were in use by acronis and linked to the appliance vm as hdds...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you can always check the VM if running on a snapshot...see my EE Article
0
CHI-LTDAuthor Commented:
finally looks like its worked...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If all the snapshots have gone, and checks confirm it's no longer on a snapshot disk.

You just need to be patient.....and wait for the process to complete.

In future, Check for Snapshots, e.g. part of your daily routine, or via scripts and alaerts!
0
CHI-LTDAuthor Commented:
Need to find out why this has happened now!
How best to set up alerts?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Enable the Snapshot Alert using vCenter Server.
0
CHI-LTDAuthor Commented:
We dont use vcentre server...
Any other way?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Scripting....(and a new question!),

..... and a good reason to use vCenter Server.
0
CHI-LTDAuthor Commented:
for two hosts?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.