Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

In Need of VMware ESXi 5.1 Data Recovery Help.

Posted on 2015-07-22
23
Medium Priority
?
933 Views
Last Modified: 2016-10-27
We have a VM server that went down.  We were able to make repairs and have the server up and running but we have snap-shots that we are unable to commit hence we can't access tons of data.  This is a mission critical project.  Am I in the right place?
0
Comment
Question by:davidkaneperez
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 14
  • 8
23 Comments
 
LVL 123
ID: 40894287
You are in the right place.

Can you screenshot, the VM folder for me, so I can see the files?

Do you have a VM, which is running on a snapshot, that has failed, and the snapshots are corrupted (which happens), which is the danger of snapshots.

How did you get the VM up and running ?

I can work with you, with this issue...
0
 
LVL 18

Expert Comment

by:Wayne88
ID: 40894290
Also, do you have any backup of the server?  Windows backup, Norton, Acronis, etc?
0
 

Author Comment

by:davidkaneperez
ID: 40894310
/vmfs/volumes/55ab2d88-490b96b1-afd2-002219aedc56 # ls -l BAE-CAD02
-rw-------    1 root     root       7864832 Jul 19 05:51 BAE-CAD02-000001-ctk.vmdk
-rw-------    1 root     root     109521915904 Jul 16 12:18 BAE-CAD02-000001-delta.vmdk
-rw-------    1 root     root           439 Jul 16 12:09 BAE-CAD02-000001.vmdk
-rw-------    1 root     root         29669 May  7 01:01 BAE-CAD02-Snapshot2241.vmsn
-rw-r--r--    1 root     root           132 Jul  1 19:57 BAE-CAD02-aux.xml
-rw-------    1 root     root       7864832 May  7 01:01 BAE-CAD02-ctk.vmdk
-rw-------    1 root     root     6442450944 Jul 13 22:59 BAE-CAD02-ec64b0f5.vswp
-rw-------    1 root     root     128849018880 May  7 01:01 BAE-CAD02-flat.vmdk
-rwx------    1 root     root           387 May  7 01:01 BAE-CAD02-vss_manifests2241.zip
-rw-------    1 root     root          8684 Jul 16 12:12 BAE-CAD02.nvram
-rw-------    1 root     root           608 Jul  1 08:02 BAE-CAD02.vmdk
-rw-r--r--    1 root     root           803 Jul 16 11:01 BAE-CAD02.vmsd
-rwxr-xr-x    1 root     root          4126 Jul 19 05:51 BAE-CAD02.vmx
-rw-r--r--    1 root     root          3040 Jul 19 05:50 BAE-CAD02.vmxf
-rw-------    1 root     root     536870912000 Nov 19  2012 BAE-CAD02_1-flat.vmdk
-rw-------    1 root     root           501 Nov 19  2012 BAE-CAD02_1.vmdk
-rw-------    1 root     root     26123153408 Jul 16 19:07 BAE-CAD02_2-000001-delta.vmdk
-rw-------    1 root     root           376 Jul 16 18:22 BAE-CAD02_2-000001.vmdk
-rw-------    1 root     root       8192512 Jul 22 17:25 BAE-CAD02_2-ctk.vmdk
-rw-------    1 root     root     536870912000 Jul 22 21:47 BAE-CAD02_2-flat.vmdk
-rw-------    1 root     root           587 Jul 22 17:25 BAE-CAD02_2.vmdk
-rw-------    1 root     root     548063940608 Jul 16 13:50 BAE-CAD02_3-000001-delta.vmdk
-rw-------    1 root     root           376 Jul 16 13:11 BAE-CAD02_3-000001.vmdk
-rw-------    1 root     root       5243392 Jul 22 17:25 BAE-CAD02_3-ctk.vmdk
-rw-------    1 root     root     1374389534720 Jul 22 21:47 BAE-CAD02_3-flat.vmdk
-rw-------    1 root     root           588 Jul 22 17:25 BAE-CAD02_3.vmdk
-rw-------    1 root     root       8192512 Jul 22 17:25 BAE-CAD02_4-ctk.vmdk
-rw-------    1 root     root     536870912000 Jul 22 17:59 BAE-CAD02_4-flat.vmdk
-rw-------    1 root     root           613 Jul 22 17:25 BAE-CAD02_4.vmdk
-rw-------    1 root     root       7864832 Jul 22 17:25 BAE-CAD02_repaired-ctk.vmdk
-rw-------    1 root     root     128849018880 Jul 22 22:05 BAE-CAD02_repaired-flat.vmdk
-rw-------    1 root     root           600 Jul 22 17:25 BAE-CAD02_repaired.vmdk
-rw-------    1 root     root      14122811 Jul 16 12:18 vmmcores-10.gz
-rw-------    1 root     root      15759794 Jul 16 10:29 vmmcores-5.gz
-rw-------    1 root     root      14748782 Jul 16 10:50 vmmcores-6.gz
-rw-------    1 root     root      14677246 Jul 16 11:11 vmmcores-7.gz
-rw-------    1 root     root      14016427 Jul 16 11:27 vmmcores-8.gz
-rw-------    1 root     root      14497354 Jul 16 11:51 vmmcores-9.gz
-rw-r--r--    1 root     root        171846 Jul 16 10:33 vmware-36.log
-rw-r--r--    1 root     root        214357 Jul 16 10:50 vmware-37.log
-rw-r--r--    1 root     root        154727 Jul 16 11:11 vmware-38.log
-rw-r--r--    1 root     root        154788 Jul 16 11:27 vmware-39.log
-rw-r--r--    1 root     root        154719 Jul 16 11:51 vmware-40.log
-rw-r--r--    1 root     root        154614 Jul 16 12:18 vmware-41.log
-rw-r--r--    1 root     root         59187 Jul 19 05:51 vmware.log
-rw-------    1 root     root      98566144 Jul 13 22:59 vmx-BAE-CAD02-3966021877-1.vswp
-r--------    1 root     root       5529600 Jul 16 11:51 vmx-zdump.000
-r--------    1 root     root       5550080 Jul 16 12:18 vmx-zdump.001
-r--------    1 root     root       5517312 Jul 16 11:27 vmx-zdump.003
/vmfs/volumes/55ab2d88-490b96b1-afd2-002219aedc56 #
0
Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

 

Author Comment

by:davidkaneperez
ID: 40894315
This VM has 4 v drives.  We used vmkfstools -i to get the C:\ drive (120GB Drive) working.
0
 

Author Comment

by:davidkaneperez
ID: 40894320
I walked into this issue and they had no running backups.  I attempted Acronis, that failed, I also tried windows backups and that failed as well.
0
 

Author Comment

by:davidkaneperez
ID: 40894322
I also tried migrating the VM but of course that failed.
0
 
LVL 123
ID: 40894338
Okay, no offence, but no more fiddling until we have a look at the damage....

by using vmkfstools -i, this is a clone command, depending upon how you used it, will merge the snapshots into the parent.

So you have a VM called  BAE-CAD02.

Before we start, do you have a budget for Disaster Recovery to approach Data Recovery Specialists Kroll Ontrack ?

Do you also haver a copy/backup of the current files.

Okay, so the files look like as follows:-


BAE-CAD02_repaired-flat.vmdk
BAE-CAD02_repaired.vmdk

missing VMDK here and snapshot?

BAE-CAD02-flat.vmdk
BAE-CAD02-000001-delta.vmdk
BAE-CAD02-000001.vmdk

looks okay

BAE-CAD02_1.vmdk
BAE-CAD02_1-flat.vmdk

snapshot attached

BAE-CAD02_2-flat.vmdk
BAE-CAD02_2.vmdk
BAE-CAD02_2-000001-delta.vmdk
BAE-CAD02_2-000001.vmdk
 
snapshot attached

BAE-CAD02_3-flat.vmdk
BAE-CAD02_3.vmdk
BAE-CAD02_3-000001-delta.vmdk
BAE-CAD02_3-000001.vmdk

looks okay

BAE-CAD02_4-flat.vmdk
BAE-CAD02_4.vmdk

 I'm assuming that the C: drive is the virtual disk

BAE-CAD02_repaired-flat.vmdk
BAE-CAD02_repaired.vmdk

and there are four other virtual disks? three of the four have snapshots?

is the missing data in these additional disks, which are not part of the VM at present ?

So what did you repair, and what virtual disks, are missing data ?

if may help if you screenshot, what the current VM disks settings in the VM settings look like we well...
0
 

Author Comment

by:davidkaneperez
ID: 40894373
The VM is currently working but the additional drives reverted to data as far as March.  I included an ls -l of the vmfs/volumes/Datastore... /BAE-CADD which are all of the files listed.  Please let me know if this is what you were looking for or if there is anything else.  Thanks.
0
 
LVL 123
ID: 40894381
Okay.....can you let me know, did you rebuild the virtual machine, and add the disks manually....it could be that....

I think you also need to shutdown the VM, because otherwise data will be lost, when we try to merge the data back?

has any data changed in the current VM if it is on?

We the OS disks also labelled:-

C:
D:
E:
F:

G: ?

Please look at ALL the disks settings, in the VM, and can you let me know if they are writing to the snapshot disk?

and do you know which virtual disk is which OS disk ?
0
 

Author Comment

by:davidkaneperez
ID: 40894429
I think you also need to shutdown the VM, because otherwise data will be lost, when we try to merge the data back?
---> Yes, I can now shut down the VM.

has any data changed in the current VM if it is on?
---> Yes it has since it is a production VM.

We the OS disks also labelled:-

C:
D:
E:
F:

G: ?
---> Yes, they were.

Please look at ALL the disks settings, in the VM, and can you let me know if they are writing to the snapshot disk?
--- >  Checking on this now.

and do you know which virtual disk is which OS disk ?
--->  Yes, it live on the 120GB drive BAE-CAD02.vmdk.
0
 

Author Comment

by:davidkaneperez
ID: 40894430
Getting the screen shot of the settings.
0
 
LVL 123
ID: 40894442
okay, those disks are they running on the snapshot disk, and they are the disks which are missing data ?

now we have an issue, because

1. do you want to lose the data which has been updated ? on these disks?

2. because if we merge the disks, this recent data will be lost?

what is this server, a file server with data and shares ?

Do you know how to check, if the disk is writing to a snapshot see my EE Article if unsure.

HOW TO: VMware Snapshots :- Be Patient
0
 

Author Comment

by:davidkaneperez
ID: 40894452
The server was brought back on site today.  We are ok if we lose that data.  We much rather have the data prior to today.  Also I checked the SnapShot Manager but there was nothing there.   Including screen shot.
Screenshot-from-2015-07-22-20-04-53.png
0
 
LVL 123
ID: 40894463
Can you check the disk properties of the disks

D: 500GB BAE-CAD02_2-000001-delta.vmdk
E: 1.2TB BAE-CAD02_3-000001-delta.vmdk
F: 500GB

in VM settings are they writing to the current snapshot files

and are you missing data from which disks ?

D: and E:
0
 
LVL 123
ID: 40894470
Sorry, I need to have screenshots showing the actual disk settings from the VM Settings and the paths to the disk in use.

Right Click the VM > Edit Settings > and then each disk > the file it's writing to, this was missing on the previous screenshots.
0
 

Author Comment

by:davidkaneperez
ID: 40894471
Looks like only the BAE-CAD02_{1,2,3,4}.vmdk are being modified.
Screenshot-from-2015-07-22-20-37-23.png
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 40894487
okay thanks for this....

okay currently this is what the VM looks lile

C: - BAE-CAD02_repaired.vmdk Hard Disk 1 - 120GB - no snapshot here - so we are okay...

D: BAE-CAD02_2.vmdk Hard Disk 2 - 500GB - snapshot here, so we need to deal with this....

E: BAE-CAD02_3.vmdk Hard Disk 3 - 1.25TB - snapshot here, so we need to deal with this...

F: BAE-CAD02_4.vmdk Hard Disk 5 - 500GB -  no snapshot here - so we are okay...

So do you agree with the above, I expect you are missing data, e.g. it's been "rolled back" on Drive D: and Drive F: ?

The plan, as you have powered on the VM, ALL the virtual disks, would have been updated, so the first thing you will have to do, is align the CIDs in the files

Align CIDS as follows:-

see here

http://kb.vmware.com/kb/1007969

You will need to do this for

D: BAE-CAD02_2.vmdk Hard Disk 2 - 500GB - snapshot here, so we need to deal with this....

E: BAE-CAD02_3.vmdk Hard Disk 3 - 1.25TB - snapshot here, so we need to deal with this...

Once, you have fixed the CID mismatch, you will the need to merge the snapshot disk with the parent, but we will use the CLONE command again, to create a new parent disk...

e.g.vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

this operation is going to take some time, to clone and merge and create a new disks.

when you have created the new repaired merged disks, remove the originals from the VM (do not delete from disk)

and add these new disks back to the VM. Note the Original SCSI ID values, so they appear as D: and E:.

Power ON the VM, and check the OS volumes.

I'm about to catch some Zzzzzzzzzz's here in the UK, I'll be back in approx 7 hours.
0
 

Author Comment

by:davidkaneperez
ID: 40894494
Awesome.  I will try this.  This was my last shot.  I did not know where else to turn to.  Wow!!!!  I can now sleep at night.  I am so grateful for your time.  I will report back as soon as I am done.  Thanks.
0
 

Author Closing Comment

by:davidkaneperez
ID: 40894496
I have been working on this for close to a 1.5 weeks.  I am so amazed that someone would take time to help me on this project.   I am again so grateful to have found help on this site.  Thanks a bunch.
0
 
LVL 123
ID: 40894500
We deal with snapshot issues like this every day of the week, with our clients, so we have many years of experience, helping Members of EE, put their servers back together!

All the best, and thanks for your kinds words, must get some Zzzzzzzsss now! ASAP!
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A new Acronis survey shows people often have conflicting thoughts about data protection.
When it comes to protecting Oracle Database servers and systems, there are a ton of myths out there. Here are the most common.
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question