Virtual Machine has gone back 5 days after removing snapshot

I urgently need some advice. I have a client running ESXI 5 and has 3 virtual machines
running on the server. The Host is a IBM X3650 running in raid 5. The hardware is perfectly o.k but I had a situation where the main virtual machine running Windows Server 2003 bombed out as the virtual machine ran out of space. Upon checking I discovered 3 Snapshots taken that were taking up space that I think were taken by the backup VMX software.
I deleted each snapshot in order, this cleared space and the virtual machine was able to run but it has gone back 5 days. Is there anyway to get back to the present.

I have attached a picture of the virtual machines directory

Is there any hope or do I just have to wear it.
server.jpg
cdsausAsked:
Who is Participating?
 
RickEpnetConnect With a Mentor Commented:
You may be stuck I agree. However if you look at the date on the Snapshot files they are from 4 days ago. I think if it were me and this was valuable data I would open a ticket with VMware assuming you have current support.
0
 
RickEpnetConnect With a Mentor Commented:
How did you delete the Snapshots? Did you use the Snap Shot manager?

This is a good read on Snap Shots VMware Snapshots :- Be Patient
0
 
asavenerConnect With a Mentor Commented:
Snapshots are awesome... except when they're not.

That image seems to indicate that you still have a snapshot out there.

What do you see when you go to snapshot manager for that device?
0
Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Check my EE Article posted, and confirm if you are still running on a Snapshot.

If you have reverted to 5 days ago, something has gone wrong.

Check, if you are running on a Snapshot?

this last snapshot may be possible to merge this with parent to revert to present.

BUT, you will need to turn off machine now.

In the mean time, I would restore the VM as a standby, but do not turn on, until we confirm the worst.
0
 
asavenerConnect With a Mentor Commented:
Right-click the VM, select Snapshot Manager, and look for the "you are here" message.  If there is a snapshot at the same level or below the "you are here" message, you might be able to switch to that snapshot.

Otherwise, you're stuck.
0
 
cdsausAuthor Commented:
No I have no snapshots in the snapshot manager, I deleted them but I think the consolidation failed as there are what appears to be snapshots in the directory but missing the delta files.
I am trying to setup another ESXI server so I can copy the virtual machine to it so I can
attempt consolidation without the risk.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Please check your VM, is it currently writing to a snapshot disk?

you may be able to clone your existing disk, and merge the remaing snapshot, to create a new disk. If you want to try this let me know, I can work with you to show you how.
0
 
cdsausAuthor Commented:
Hi yes, I have discovered there are orphaned delta files, 1 is approx 3gb and the other is about 30gb. The snapshot manager must have not reconsolidated files due to lack of space.

I believe / hope this contains the data that  I need.

I have copied the virtual machine to another host and tried performing another snapshot
and then delete all in the hope that it would reconsolidate but no luck.

Just wondering how I can reconsolidate these orphaned Delta Files, will cloning as you suggest do this?

I would appreciate your advice.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Yes, it's possible to clone, using a snapshot and parent virtual machine disk.

however, if the chain is broken it may not be possible.

it has to be done from the console or remotely via ssh?

do you know how to access the console or remotely via ssh.

Also the VM must be off.

Can you check if the VM is currently not running on a snapshot, check the VM settings as per my article.

also can you upload the latest screens shot.

If you did not have enough disk space, it can cause a fail to merge!
0
 
cdsausAuthor Commented:
Thanks for response.

I have checked VM is off

VM is not running on a snapshot

I have not used remote via ssh but want to learn.

I am not on site so Console at the moment is not a option

I have updated with latest screen shot of VM directory listing

You can see the orphaned delta files.

For some reason when I copied the files to a local share the files actually say Delta
but in the vm directory I dont see the Delta wording in the name.

I copied these files to a windows share as for some reason the vm copy stopped after
the vm files where copied. no other files where copied so I had to do it remotely as I could
not access the machine which ran the copy job. I used veeam.
VMDirectory.jpg
vmcontents.JPG
0
 
cdsausAuthor Commented:
Just for you reference, I know why the problem happened.

about a week ago I use VMX backup software to do a backup. I cant remember the
exact reason but cancelled the backup. I was unaware that it had not deleted or cleaned up
its snapshot.

I had a call from client about 5 days later that machine was running very badly. upon checking
vm could not start due to lack of space. I discovered the snapshot and deleted them 1 by one
everything went ok to my knowledge but about 30 minutes later up getting the machine going I got a call from client saying there work was not there and had reverted to the 05/12/2013

I assume the snapshots did not reconsolidate properly thus leaving the orphaned snapshots in the directory. am I correct to assume this happend ?
0
 
asavenerConnect With a Mentor Commented:
Did you click on "revert to current snapshot?"
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
For some reason when I copied the files to a local share the files actually say Delta
but in the vm directory I dont see the Delta wording in the name.

this is normal, when moving files from ESXi (vmfs).

from what I can see there is 1 snapshot left, it's disassociated with the VM, your VM has TWO virtual disks attached - is this correct?
0
 
cdsausAuthor Commented:
Yes correct, and i have setup remote ssh using putty.
0
 
cdsausAuthor Commented:
Did you click on "revert to current snapshot?"

The option appears disabled so it must think there is no snapshot available.

I have checked and VM is not running on any snapshot
0
 
cdsausAuthor Commented:
you may be able to clone your existing disk, and merge the remaing snapshot, to create a new disk. If you want to try this let me know, I can work with you to show you how.

Do you think I should try this?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
It will not do any harm, because it will create a new virtual disk, it very much depends if you have the time, and want to continue this.

Did you restore the VM, and is the client back up and running, and in business, with no loss of data?
0
 
cdsausAuthor Commented:
I have time, you were going to show me how to do it through the console or remotely via ssh.

I have not restored or converted the virtual machine. I have made a copy of it to another host
just as a backup. The client is down at the moment and as mentioned 5 days of data missing due to the failed consolidation of snapshot.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
OKay, the first thing you need todo, is download PuTTy.

http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe

Download to your computer, run it and put the IP Address of your ESXi server.

you then need to Enable SSH on your ESXi server, if not already enabled.

see here for this

VMware KB: Using Tech Support Mode in ESXi 4.1 and ESXi 5.x
0
 
cdsausAuthor Commented:
Got putty and enabled SSH, I am doing this remotely so I assume we are doing Remote Console.
0
 
cdsausAuthor Commented:
Hi I have read the articl and followed the instructions, I am able to remote in now via ssh. not sure what your next recommend step is?

Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

you need run the above commands for both virtual disks.

so you need to change director (cd) to the vmfs directory where the files exist and type


vmkfstools -i server.workgroup.local-000004.vmdk /vmfs/volumes/<temp folder name>/server.workgroup.local.vmdk

make sure you create a temp new folder, so you create a new clone of the existing virtual machine disk, VM must be OFF.

you need to repeat the abvoe for second virtual machine disk

vmkfstools -i server.workgroup.local_1-000004.vmdk /vmfs/volumes/<temp folder name>/server.workgroup.local_1.vmdk

if you get an error running the above commands, it's possible that the Snapshot, has become disassociated with the parent disk, and we would need to re-create that association.
0
 
cdsausAuthor Commented:
Sorry for my inexperience with command line arguements. I have cd to the directory where
the files are but I need to copy to another host as I dont have the space on this host.

the host is ip 10.0.0.238 and the datastore where the file vm is to to in is located in /vmfs/volumes/datastore3/temp

can you give me a example of how to structure the command.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
okay, copy to another host

and then you need to type

cd /vmfs/volumes/datastore3/temp

but you will also need to create a new folder, called newvm (temp folder) in datastore3

command then is

vmkfstools -i server.workgroup.local_1-000004.vmdk /vmfs/volumes/newvm/server.workgroup.local_1.vmdk

the above command is executed in /vmfs/volumes/datastore3/temp
0
 
cdsausAuthor Commented:
Okay, just so I get this right I need to copy the 2 Delta files (only) to the host and then execute
the command in the temp directory. Do I need to copy any other files or just the 2 snapshot files.

Sorry for all the questions.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
you need to copy ALL the VM files, that were in the VM Folder.

do you have a backup of this VM to restore?
0
 
cdsausAuthor Commented:
When trying to do the clone procedure I keep getting the followin error.


Failed to open 'server.workgroup.local-000004.vmdk': The parent of this virtual disk could not be opened (23).
/vmfs/volumes/519872be-01d542e0-1758-001a64c1311c/server.workgroup.local # Failed to open 'server.workgroup.local-0000
04.vmdk': The parent of this virtual disk could not be opened (23).
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
okay, this is possibly because the snapshot is not linked to the parent virtual disk anymore, part of the previous consolidation/revert/merge, has removed the association.

could you paste the contents of the server.workgroup.local-000004.vmdk file (the small 1kb file, is a text file descriptor file)
0
 
cdsausAuthor Commented:
This is what I have

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a2212306
parentCID=6b0ed997
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="server.workgroup.local-000003.vmdk"
# Extent description
RW 146806784 VMFSSPARSE "server.workgroup.local-000004-delta.vmdk"

# The Disk Data Base
#DDB

ddb.longContentID = "7b0f21f2d7547fdf4201094ea2212306"
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
okay, this is what I suspected you can see the parent states server.workgroup.local-000003.vmdk which is no longer available or on the disk.
0
 
cdsausAuthor Commented:
That's correct, it is no longer there.
Is it still possible to consolidate the
 snapshot ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
the problem we have is:-

parent disk + snapshot1 + snapshot2 + snapshot3 + snapshot4

the above are all chained, for some reason, you have been left with only snapshot4, which is no longer chained to parent disk, because it would have been chained to snapshot3, which is chained to snapshot2, etc

we can alter the disks, and force snapshot4 to be chained with parent disk, and complete a clone, BUT it's possible we may end up with a corrupted parent disk of no use, OR a successful merge.
0
 
cdsausAuthor Commented:
Willing to try anything at this point, I am up for it if you are.

Thanks

Jim
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
okay, I'll need to look also at the original parent.vmdk file, the descriptor file. the small file.

if you could upload the contents.
0
 
cdsausAuthor Commented:
I have not been able to find the descriptor file, but I have had some help from VWARE paid support. who managed to clone the file as hanccocka sugggested and was on the right track. They then attached the cloned drive to the running VM and I was able to view the directory structure and extract the file that  I needed. The snapshot was definately orphaned and they had to do some editing to bring the snapshot in line with the original VM. I am sure this was the road we were heading on with the advice from hanccocka and I would have loved to have seen the instructions he would have offered for future reference. Thank you all for your advice and especially to  hanccocka for his efforts, your advice was invaluable and a learning experience for me.

Jim
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Jim

Yes, this was the process, and always works, if you've got the time, unfortunately, VMware Support , have the advantage of providing Remote Support, here on EE we cannot!

Andrew
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.