Solved

Virtual Machine has gone back 5 days after removing snapshot

Posted on 2013-05-16
36
533 Views
Last Modified: 2013-06-05
I urgently need some advice. I have a client running ESXI 5 and has 3 virtual machines
running on the server. The Host is a IBM X3650 running in raid 5. The hardware is perfectly o.k but I had a situation where the main virtual machine running Windows Server 2003 bombed out as the virtual machine ran out of space. Upon checking I discovered 3 Snapshots taken that were taking up space that I think were taken by the backup VMX software.
I deleted each snapshot in order, this cleared space and the virtual machine was able to run but it has gone back 5 days. Is there anyway to get back to the present.

I have attached a picture of the virtual machines directory

Is there any hope or do I just have to wear it.
server.jpg
0
Comment
Question by:cdsaus
  • 17
  • 14
  • 3
  • +1
36 Comments
 
LVL 14

Assisted Solution

by:RickEpnet
RickEpnet earned 40 total points
ID: 39172137
How did you delete the Snapshots? Did you use the Snap Shot manager?

This is a good read on Snap Shots VMware Snapshots :- Be Patient
0
 
LVL 28

Assisted Solution

by:asavener
asavener earned 60 total points
ID: 39172417
Snapshots are awesome... except when they're not.

That image seems to indicate that you still have a snapshot out there.

What do you see when you go to snapshot manager for that device?
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39172696
Check my EE Article posted, and confirm if you are still running on a Snapshot.

If you have reverted to 5 days ago, something has gone wrong.

Check, if you are running on a Snapshot?

this last snapshot may be possible to merge this with parent to revert to present.

BUT, you will need to turn off machine now.

In the mean time, I would restore the VM as a standby, but do not turn on, until we confirm the worst.
0
 
LVL 28

Assisted Solution

by:asavener
asavener earned 60 total points
ID: 39172711
Right-click the VM, select Snapshot Manager, and look for the "you are here" message.  If there is a snapshot at the same level or below the "you are here" message, you might be able to switch to that snapshot.

Otherwise, you're stuck.
0
 

Author Comment

by:cdsaus
ID: 39173737
No I have no snapshots in the snapshot manager, I deleted them but I think the consolidation failed as there are what appears to be snapshots in the directory but missing the delta files.
I am trying to setup another ESXI server so I can copy the virtual machine to it so I can
attempt consolidation without the risk.
0
 
LVL 14

Accepted Solution

by:
RickEpnet earned 40 total points
ID: 39173761
You may be stuck I agree. However if you look at the date on the Snapshot files they are from 4 days ago. I think if it were me and this was valuable data I would open a ticket with VMware assuming you have current support.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39173915
Please check your VM, is it currently writing to a snapshot disk?

you may be able to clone your existing disk, and merge the remaing snapshot, to create a new disk. If you want to try this let me know, I can work with you to show you how.
0
 

Author Comment

by:cdsaus
ID: 39177554
Hi yes, I have discovered there are orphaned delta files, 1 is approx 3gb and the other is about 30gb. The snapshot manager must have not reconsolidated files due to lack of space.

I believe / hope this contains the data that  I need.

I have copied the virtual machine to another host and tried performing another snapshot
and then delete all in the hope that it would reconsolidate but no luck.

Just wondering how I can reconsolidate these orphaned Delta Files, will cloning as you suggest do this?

I would appreciate your advice.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39177561
Yes, it's possible to clone, using a snapshot and parent virtual machine disk.

however, if the chain is broken it may not be possible.

it has to be done from the console or remotely via ssh?

do you know how to access the console or remotely via ssh.

Also the VM must be off.

Can you check if the VM is currently not running on a snapshot, check the VM settings as per my article.

also can you upload the latest screens shot.

If you did not have enough disk space, it can cause a fail to merge!
0
 

Author Comment

by:cdsaus
ID: 39177821
Thanks for response.

I have checked VM is off

VM is not running on a snapshot

I have not used remote via ssh but want to learn.

I am not on site so Console at the moment is not a option

I have updated with latest screen shot of VM directory listing

You can see the orphaned delta files.

For some reason when I copied the files to a local share the files actually say Delta
but in the vm directory I dont see the Delta wording in the name.

I copied these files to a windows share as for some reason the vm copy stopped after
the vm files where copied. no other files where copied so I had to do it remotely as I could
not access the machine which ran the copy job. I used veeam.
VMDirectory.jpg
vmcontents.JPG
0
 

Author Comment

by:cdsaus
ID: 39177829
Just for you reference, I know why the problem happened.

about a week ago I use VMX backup software to do a backup. I cant remember the
exact reason but cancelled the backup. I was unaware that it had not deleted or cleaned up
its snapshot.

I had a call from client about 5 days later that machine was running very badly. upon checking
vm could not start due to lack of space. I discovered the snapshot and deleted them 1 by one
everything went ok to my knowledge but about 30 minutes later up getting the machine going I got a call from client saying there work was not there and had reverted to the 05/12/2013

I assume the snapshots did not reconsolidate properly thus leaving the orphaned snapshots in the directory. am I correct to assume this happend ?
0
 
LVL 28

Assisted Solution

by:asavener
asavener earned 60 total points
ID: 39177841
Did you click on "revert to current snapshot?"
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39177848
For some reason when I copied the files to a local share the files actually say Delta
but in the vm directory I dont see the Delta wording in the name.

this is normal, when moving files from ESXi (vmfs).

from what I can see there is 1 snapshot left, it's disassociated with the VM, your VM has TWO virtual disks attached - is this correct?
0
 

Author Comment

by:cdsaus
ID: 39177870
Yes correct, and i have setup remote ssh using putty.
0
 

Author Comment

by:cdsaus
ID: 39177951
Did you click on "revert to current snapshot?"

The option appears disabled so it must think there is no snapshot available.

I have checked and VM is not running on any snapshot
0
 

Author Comment

by:cdsaus
ID: 39177952
you may be able to clone your existing disk, and merge the remaing snapshot, to create a new disk. If you want to try this let me know, I can work with you to show you how.

Do you think I should try this?
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39178237
It will not do any harm, because it will create a new virtual disk, it very much depends if you have the time, and want to continue this.

Did you restore the VM, and is the client back up and running, and in business, with no loss of data?
0
 

Author Comment

by:cdsaus
ID: 39178241
I have time, you were going to show me how to do it through the console or remotely via ssh.

I have not restored or converted the virtual machine. I have made a copy of it to another host
just as a backup. The client is down at the moment and as mentioned 5 days of data missing due to the failed consolidation of snapshot.
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39178243
OKay, the first thing you need todo, is download PuTTy.

http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe

Download to your computer, run it and put the IP Address of your ESXi server.

you then need to Enable SSH on your ESXi server, if not already enabled.

see here for this

VMware KB: Using Tech Support Mode in ESXi 4.1 and ESXi 5.x
0
 

Author Comment

by:cdsaus
ID: 39178351
Got putty and enabled SSH, I am doing this remotely so I assume we are doing Remote Console.
0
 

Author Comment

by:cdsaus
ID: 39178420
Hi I have read the articl and followed the instructions, I am able to remote in now via ssh. not sure what your next recommend step is?

Thanks
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39178464
vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

you need run the above commands for both virtual disks.

so you need to change director (cd) to the vmfs directory where the files exist and type


vmkfstools -i server.workgroup.local-000004.vmdk /vmfs/volumes/<temp folder name>/server.workgroup.local.vmdk

make sure you create a temp new folder, so you create a new clone of the existing virtual machine disk, VM must be OFF.

you need to repeat the abvoe for second virtual machine disk

vmkfstools -i server.workgroup.local_1-000004.vmdk /vmfs/volumes/<temp folder name>/server.workgroup.local_1.vmdk

if you get an error running the above commands, it's possible that the Snapshot, has become disassociated with the parent disk, and we would need to re-create that association.
0
 

Author Comment

by:cdsaus
ID: 39178536
Sorry for my inexperience with command line arguements. I have cd to the directory where
the files are but I need to copy to another host as I dont have the space on this host.

the host is ip 10.0.0.238 and the datastore where the file vm is to to in is located in /vmfs/volumes/datastore3/temp

can you give me a example of how to structure the command.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39178551
okay, copy to another host

and then you need to type

cd /vmfs/volumes/datastore3/temp

but you will also need to create a new folder, called newvm (temp folder) in datastore3

command then is

vmkfstools -i server.workgroup.local_1-000004.vmdk /vmfs/volumes/newvm/server.workgroup.local_1.vmdk

the above command is executed in /vmfs/volumes/datastore3/temp
0
 

Author Comment

by:cdsaus
ID: 39178560
Okay, just so I get this right I need to copy the 2 Delta files (only) to the host and then execute
the command in the temp directory. Do I need to copy any other files or just the 2 snapshot files.

Sorry for all the questions.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39178633
you need to copy ALL the VM files, that were in the VM Folder.

do you have a backup of this VM to restore?
0
 

Author Comment

by:cdsaus
ID: 39184661
When trying to do the clone procedure I keep getting the followin error.


Failed to open 'server.workgroup.local-000004.vmdk': The parent of this virtual disk could not be opened (23).
/vmfs/volumes/519872be-01d542e0-1758-001a64c1311c/server.workgroup.local # Failed to open 'server.workgroup.local-0000
04.vmdk': The parent of this virtual disk could not be opened (23).
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39184696
okay, this is possibly because the snapshot is not linked to the parent virtual disk anymore, part of the previous consolidation/revert/merge, has removed the association.

could you paste the contents of the server.workgroup.local-000004.vmdk file (the small 1kb file, is a text file descriptor file)
0
 

Author Comment

by:cdsaus
ID: 39186014
This is what I have

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a2212306
parentCID=6b0ed997
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="server.workgroup.local-000003.vmdk"
# Extent description
RW 146806784 VMFSSPARSE "server.workgroup.local-000004-delta.vmdk"

# The Disk Data Base
#DDB

ddb.longContentID = "7b0f21f2d7547fdf4201094ea2212306"
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39186582
okay, this is what I suspected you can see the parent states server.workgroup.local-000003.vmdk which is no longer available or on the disk.
0
 

Author Comment

by:cdsaus
ID: 39186603
That's correct, it is no longer there.
Is it still possible to consolidate the
 snapshot ?
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39199716
the problem we have is:-

parent disk + snapshot1 + snapshot2 + snapshot3 + snapshot4

the above are all chained, for some reason, you have been left with only snapshot4, which is no longer chained to parent disk, because it would have been chained to snapshot3, which is chained to snapshot2, etc

we can alter the disks, and force snapshot4 to be chained with parent disk, and complete a clone, BUT it's possible we may end up with a corrupted parent disk of no use, OR a successful merge.
0
 

Author Comment

by:cdsaus
ID: 39200167
Willing to try anything at this point, I am up for it if you are.

Thanks

Jim
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39200390
okay, I'll need to look also at the original parent.vmdk file, the descriptor file. the small file.

if you could upload the contents.
0
 

Author Comment

by:cdsaus
ID: 39221692
I have not been able to find the descriptor file, but I have had some help from VWARE paid support. who managed to clone the file as hanccocka sugggested and was on the right track. They then attached the cloned drive to the running VM and I was able to view the directory structure and extract the file that  I needed. The snapshot was definately orphaned and they had to do some editing to bring the snapshot in line with the original VM. I am sure this was the road we were heading on with the advice from hanccocka and I would have loved to have seen the instructions he would have offered for future reference. Thank you all for your advice and especially to  hanccocka for his efforts, your advice was invaluable and a learning experience for me.

Jim
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 400 total points
ID: 39221710
Jim

Yes, this was the process, and always works, if you've got the time, unfortunately, VMware Support , have the advantage of providing Remote Support, here on EE we cannot!

Andrew
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now