In Need of VMware ESXi 5.1 Data Recovery Help.

We have a VM server that went down.  We were able to make repairs and have the server up and running but we have snap-shots that we are unable to commit hence we can't access tons of data.  This is a mission critical project.  Am I in the right place?
davidkaneperezAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You are in the right place.

Can you screenshot, the VM folder for me, so I can see the files?

Do you have a VM, which is running on a snapshot, that has failed, and the snapshots are corrupted (which happens), which is the danger of snapshots.

How did you get the VM up and running ?

I can work with you, with this issue...
Wayne88Commented:
Also, do you have any backup of the server?  Windows backup, Norton, Acronis, etc?
davidkaneperezAuthor Commented:
/vmfs/volumes/55ab2d88-490b96b1-afd2-002219aedc56 # ls -l BAE-CAD02
-rw-------    1 root     root       7864832 Jul 19 05:51 BAE-CAD02-000001-ctk.vmdk
-rw-------    1 root     root     109521915904 Jul 16 12:18 BAE-CAD02-000001-delta.vmdk
-rw-------    1 root     root           439 Jul 16 12:09 BAE-CAD02-000001.vmdk
-rw-------    1 root     root         29669 May  7 01:01 BAE-CAD02-Snapshot2241.vmsn
-rw-r--r--    1 root     root           132 Jul  1 19:57 BAE-CAD02-aux.xml
-rw-------    1 root     root       7864832 May  7 01:01 BAE-CAD02-ctk.vmdk
-rw-------    1 root     root     6442450944 Jul 13 22:59 BAE-CAD02-ec64b0f5.vswp
-rw-------    1 root     root     128849018880 May  7 01:01 BAE-CAD02-flat.vmdk
-rwx------    1 root     root           387 May  7 01:01 BAE-CAD02-vss_manifests2241.zip
-rw-------    1 root     root          8684 Jul 16 12:12 BAE-CAD02.nvram
-rw-------    1 root     root           608 Jul  1 08:02 BAE-CAD02.vmdk
-rw-r--r--    1 root     root           803 Jul 16 11:01 BAE-CAD02.vmsd
-rwxr-xr-x    1 root     root          4126 Jul 19 05:51 BAE-CAD02.vmx
-rw-r--r--    1 root     root          3040 Jul 19 05:50 BAE-CAD02.vmxf
-rw-------    1 root     root     536870912000 Nov 19  2012 BAE-CAD02_1-flat.vmdk
-rw-------    1 root     root           501 Nov 19  2012 BAE-CAD02_1.vmdk
-rw-------    1 root     root     26123153408 Jul 16 19:07 BAE-CAD02_2-000001-delta.vmdk
-rw-------    1 root     root           376 Jul 16 18:22 BAE-CAD02_2-000001.vmdk
-rw-------    1 root     root       8192512 Jul 22 17:25 BAE-CAD02_2-ctk.vmdk
-rw-------    1 root     root     536870912000 Jul 22 21:47 BAE-CAD02_2-flat.vmdk
-rw-------    1 root     root           587 Jul 22 17:25 BAE-CAD02_2.vmdk
-rw-------    1 root     root     548063940608 Jul 16 13:50 BAE-CAD02_3-000001-delta.vmdk
-rw-------    1 root     root           376 Jul 16 13:11 BAE-CAD02_3-000001.vmdk
-rw-------    1 root     root       5243392 Jul 22 17:25 BAE-CAD02_3-ctk.vmdk
-rw-------    1 root     root     1374389534720 Jul 22 21:47 BAE-CAD02_3-flat.vmdk
-rw-------    1 root     root           588 Jul 22 17:25 BAE-CAD02_3.vmdk
-rw-------    1 root     root       8192512 Jul 22 17:25 BAE-CAD02_4-ctk.vmdk
-rw-------    1 root     root     536870912000 Jul 22 17:59 BAE-CAD02_4-flat.vmdk
-rw-------    1 root     root           613 Jul 22 17:25 BAE-CAD02_4.vmdk
-rw-------    1 root     root       7864832 Jul 22 17:25 BAE-CAD02_repaired-ctk.vmdk
-rw-------    1 root     root     128849018880 Jul 22 22:05 BAE-CAD02_repaired-flat.vmdk
-rw-------    1 root     root           600 Jul 22 17:25 BAE-CAD02_repaired.vmdk
-rw-------    1 root     root      14122811 Jul 16 12:18 vmmcores-10.gz
-rw-------    1 root     root      15759794 Jul 16 10:29 vmmcores-5.gz
-rw-------    1 root     root      14748782 Jul 16 10:50 vmmcores-6.gz
-rw-------    1 root     root      14677246 Jul 16 11:11 vmmcores-7.gz
-rw-------    1 root     root      14016427 Jul 16 11:27 vmmcores-8.gz
-rw-------    1 root     root      14497354 Jul 16 11:51 vmmcores-9.gz
-rw-r--r--    1 root     root        171846 Jul 16 10:33 vmware-36.log
-rw-r--r--    1 root     root        214357 Jul 16 10:50 vmware-37.log
-rw-r--r--    1 root     root        154727 Jul 16 11:11 vmware-38.log
-rw-r--r--    1 root     root        154788 Jul 16 11:27 vmware-39.log
-rw-r--r--    1 root     root        154719 Jul 16 11:51 vmware-40.log
-rw-r--r--    1 root     root        154614 Jul 16 12:18 vmware-41.log
-rw-r--r--    1 root     root         59187 Jul 19 05:51 vmware.log
-rw-------    1 root     root      98566144 Jul 13 22:59 vmx-BAE-CAD02-3966021877-1.vswp
-r--------    1 root     root       5529600 Jul 16 11:51 vmx-zdump.000
-r--------    1 root     root       5550080 Jul 16 12:18 vmx-zdump.001
-r--------    1 root     root       5517312 Jul 16 11:27 vmx-zdump.003
/vmfs/volumes/55ab2d88-490b96b1-afd2-002219aedc56 #
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

davidkaneperezAuthor Commented:
This VM has 4 v drives.  We used vmkfstools -i to get the C:\ drive (120GB Drive) working.
davidkaneperezAuthor Commented:
I walked into this issue and they had no running backups.  I attempted Acronis, that failed, I also tried windows backups and that failed as well.
davidkaneperezAuthor Commented:
I also tried migrating the VM but of course that failed.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, no offence, but no more fiddling until we have a look at the damage....

by using vmkfstools -i, this is a clone command, depending upon how you used it, will merge the snapshots into the parent.

So you have a VM called  BAE-CAD02.

Before we start, do you have a budget for Disaster Recovery to approach Data Recovery Specialists Kroll Ontrack ?

Do you also haver a copy/backup of the current files.

Okay, so the files look like as follows:-


BAE-CAD02_repaired-flat.vmdk
BAE-CAD02_repaired.vmdk

missing VMDK here and snapshot?

BAE-CAD02-flat.vmdk
BAE-CAD02-000001-delta.vmdk
BAE-CAD02-000001.vmdk

looks okay

BAE-CAD02_1.vmdk
BAE-CAD02_1-flat.vmdk

snapshot attached

BAE-CAD02_2-flat.vmdk
BAE-CAD02_2.vmdk
BAE-CAD02_2-000001-delta.vmdk
BAE-CAD02_2-000001.vmdk
 
snapshot attached

BAE-CAD02_3-flat.vmdk
BAE-CAD02_3.vmdk
BAE-CAD02_3-000001-delta.vmdk
BAE-CAD02_3-000001.vmdk

looks okay

BAE-CAD02_4-flat.vmdk
BAE-CAD02_4.vmdk

 I'm assuming that the C: drive is the virtual disk

BAE-CAD02_repaired-flat.vmdk
BAE-CAD02_repaired.vmdk

and there are four other virtual disks? three of the four have snapshots?

is the missing data in these additional disks, which are not part of the VM at present ?

So what did you repair, and what virtual disks, are missing data ?

if may help if you screenshot, what the current VM disks settings in the VM settings look like we well...
davidkaneperezAuthor Commented:
The VM is currently working but the additional drives reverted to data as far as March.  I included an ls -l of the vmfs/volumes/Datastore... /BAE-CADD which are all of the files listed.  Please let me know if this is what you were looking for or if there is anything else.  Thanks.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay.....can you let me know, did you rebuild the virtual machine, and add the disks manually....it could be that....

I think you also need to shutdown the VM, because otherwise data will be lost, when we try to merge the data back?

has any data changed in the current VM if it is on?

We the OS disks also labelled:-

C:
D:
E:
F:

G: ?

Please look at ALL the disks settings, in the VM, and can you let me know if they are writing to the snapshot disk?

and do you know which virtual disk is which OS disk ?
davidkaneperezAuthor Commented:
I think you also need to shutdown the VM, because otherwise data will be lost, when we try to merge the data back?
---> Yes, I can now shut down the VM.

has any data changed in the current VM if it is on?
---> Yes it has since it is a production VM.

We the OS disks also labelled:-

C:
D:
E:
F:

G: ?
---> Yes, they were.

Please look at ALL the disks settings, in the VM, and can you let me know if they are writing to the snapshot disk?
--- >  Checking on this now.

and do you know which virtual disk is which OS disk ?
--->  Yes, it live on the 120GB drive BAE-CAD02.vmdk.
davidkaneperezAuthor Commented:
Getting the screen shot of the settings.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, those disks are they running on the snapshot disk, and they are the disks which are missing data ?

now we have an issue, because

1. do you want to lose the data which has been updated ? on these disks?

2. because if we merge the disks, this recent data will be lost?

what is this server, a file server with data and shares ?

Do you know how to check, if the disk is writing to a snapshot see my EE Article if unsure.

HOW TO: VMware Snapshots :- Be Patient
davidkaneperezAuthor Commented:
The server was brought back on site today.  We are ok if we lose that data.  We much rather have the data prior to today.  Also I checked the SnapShot Manager but there was nothing there.   Including screen shot.
Screenshot-from-2015-07-22-20-04-53.png
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Can you check the disk properties of the disks

D: 500GB BAE-CAD02_2-000001-delta.vmdk
E: 1.2TB BAE-CAD02_3-000001-delta.vmdk
F: 500GB

in VM settings are they writing to the current snapshot files

and are you missing data from which disks ?

D: and E:
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Sorry, I need to have screenshots showing the actual disk settings from the VM Settings and the paths to the disk in use.

Right Click the VM > Edit Settings > and then each disk > the file it's writing to, this was missing on the previous screenshots.
davidkaneperezAuthor Commented:
Looks like only the BAE-CAD02_{1,2,3,4}.vmdk are being modified.
Screenshot-from-2015-07-22-20-37-23.png
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay thanks for this....

okay currently this is what the VM looks lile

C: - BAE-CAD02_repaired.vmdk Hard Disk 1 - 120GB - no snapshot here - so we are okay...

D: BAE-CAD02_2.vmdk Hard Disk 2 - 500GB - snapshot here, so we need to deal with this....

E: BAE-CAD02_3.vmdk Hard Disk 3 - 1.25TB - snapshot here, so we need to deal with this...

F: BAE-CAD02_4.vmdk Hard Disk 5 - 500GB -  no snapshot here - so we are okay...

So do you agree with the above, I expect you are missing data, e.g. it's been "rolled back" on Drive D: and Drive F: ?

The plan, as you have powered on the VM, ALL the virtual disks, would have been updated, so the first thing you will have to do, is align the CIDs in the files

Align CIDS as follows:-

see here

http://kb.vmware.com/kb/1007969

You will need to do this for

D: BAE-CAD02_2.vmdk Hard Disk 2 - 500GB - snapshot here, so we need to deal with this....

E: BAE-CAD02_3.vmdk Hard Disk 3 - 1.25TB - snapshot here, so we need to deal with this...

Once, you have fixed the CID mismatch, you will the need to merge the snapshot disk with the parent, but we will use the CLONE command again, to create a new parent disk...

e.g.vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

this operation is going to take some time, to clone and merge and create a new disks.

when you have created the new repaired merged disks, remove the originals from the VM (do not delete from disk)

and add these new disks back to the VM. Note the Original SCSI ID values, so they appear as D: and E:.

Power ON the VM, and check the OS volumes.

I'm about to catch some Zzzzzzzzzz's here in the UK, I'll be back in approx 7 hours.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
davidkaneperezAuthor Commented:
Awesome.  I will try this.  This was my last shot.  I did not know where else to turn to.  Wow!!!!  I can now sleep at night.  I am so grateful for your time.  I will report back as soon as I am done.  Thanks.
davidkaneperezAuthor Commented:
I have been working on this for close to a 1.5 weeks.  I am so amazed that someone would take time to help me on this project.   I am again so grateful to have found help on this site.  Thanks a bunch.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We deal with snapshot issues like this every day of the week, with our clients, so we have many years of experience, helping Members of EE, put their servers back together!

All the best, and thanks for your kinds words, must get some Zzzzzzzsss now! ASAP!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Virtual Server

From novice to tech pro — start learning today.