ESXi 5.5 - Cannot boot any vms after crash

After a crash on local RAID5 storage (which it turned out is a known problem on this IBM), the vms will no longer boot. All are acting as if there is no storage to boot from.

This leads me to believe that the vms are listed in vsphere but that the storage has in fact become inaccessible.
Linux vmz only show Grub or the F2 Vmware bios screen pops up, goes away, pops up, goes away, and nothing else happens.

I cannot export because esxi says it cannot find the storage. I am able to copy the files from the datastore onto remote storage but the copies have the same problem once added back into inventory and trying to start them.

I checked the datastore using voma and some other commands and see 'Found Stale Lock' on the storage but then the report also says Total Errors Found: 0.

I'm not sure what caused the locks but at one point, I ran the same command and those were gone.  

I'm at a total loss on how to fix this problem and sure hope some can offer some insight so I can get these vms back up.

esxi 5.5
server; ibm 3655 with local SAS storage
projectsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Firstly, has there been any corruption to the RAID volume, which supports the datastore (VMFS), where the VMs are stored.

A few questions...

1. Where is ESXi installed ? Same disks RAID array as VMFS storage ?

2. Does ESXi boot?

3. Are VMs registered in the Inventory ?
0
projectsAuthor Commented:
1;
VMFS:  /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3

ls -la /vmfs/volumes/
 datastore1 -> 555d93fc-101c4070-ff0e-00145e5b014a

Strangely, df leads to this error, which it didn't earlier.
# df
Traceback (most recent call last):
  File "/bin/df", line 101, in <module>
    sys.exit(main(sys.argv))
  File "/bin/df", line 55, in main
    o = eval(output)
  File "<string>", line 1
    Errors:
          ^
SyntaxError: invalid syntax

2: Yes, esxi boots
3: yes, all VMs are registered and showing.

As mentioned, I can see the vms, I can see their data files in the storage but no matter local or copies, none boot.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
corruption in the VMFS datastore, caused by RAID or storage controller.

If all VMs are showing the same symptoms the virtual disks are corrupted, or data is missing.

I would suggest

1. Restore VMs from backup, after carefully looking at the server.

2. If no backups, consultant a Data Recovery Specialist ASAP, turn off the server to prevent further damage.

We can recommend Kroll Ontrack

http://www.krollontrack.com/
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

projectsAuthor Commented:
No support on this so we're on our own. Backups are too old since they had not done any during a storage migration.

Might you know is there is a way of doing the following.

For example, copying all of the files to another storage, already did that.
Replacing only the files which cause the vm to boot, keeping the main data files? Any chance of doing something like that?
0
projectsAuthor Commented:
before you say no... we've nothing to lose at this point.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
how are your VMs split ?

the data for the VM is contained in the VMDK, that is the only file you need.

but if it's not intact and corrupted the data is gone.

the VMDK can be added to another VM, of it can be mounted in Windows.
0
projectsAuthor Commented:
My thought it, import a working backup.
Then, shut it down, make a copy of what ever file I'm going to change, then copy the most current file in place.
See if it'll boot. If everything looks intact, then make a new backup asap.
If it doesn't work out, shut it down, overwrite the new file with the old backup and fire it back up.

This is the directory structure for this vm for example, on datastore1. In this case, the vm has two drives.
I've nothing to lose by copying which ever file I need.

zim-192-63_C624.vmx
2.87 KB
Virtual Machine
[datastore1] zim-192-63_C624
8/7/2015 12:46:53 PM

zim-192-63_C624.vmdk
12,582,910.00 KB
Virtual Disk
[datastore1] zim-192-63_C624
6/16/2015 4:28:55 PM

zim-192-63_C624_1.vmdk
16,777,220.00 KB
Virtual Disk
[datastore1] zim-192-63_C624
6/16/2015 4:29:05 PM

zim-192-63_C624.nvram
8.48 KB
Non-volatile memory file
[datastore1] zim-192-63_C624
8/7/2015 12:46:53 PM

vmware-8.log
28.74 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/6/2015 10:40:41 PM

vmware-7.log
198.36 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/6/2015 7:11:44 PM

vmware-10.log
1,767.88 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/7/2015 12:09:16 PM

zim-192-63_C624-000001.vmdk
1,262,592.00 KB
12,582,910.00 KB
Virtual Disk
[datastore1] zim-192-63_C624
8/7/2015 12:13:26 PM

vmware.log
240.38 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/7/2015 12:46:53 PM

zim-192-63_C624_1-000001.vmdk
1,655,808.00 KB
16,777,220.00 KB
Virtual Disk
[datastore1] zim-192-63_C624
8/7/2015 12:13:26 PM

zim-192-63_C624-Snapshot1.vmsn
4,195,572.00 KB
Snapshot file
[datastore1] zim-192-63_C624
7/29/2015 11:12:25 AM

vmware-6.log
125.12 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/6/2015 12:19:39 PM

vmware-5.log
109.34 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/5/2015 2:29:23 PM

vmware-9.log
762.22 KB
Virtual Machine log file
[datastore1] zim-192-63_C624
8/6/2015 11:17:02 PM

zim-192-63_C624.vmxf
0.36 KB
File
[datastore1] zim-192-63_C624
8/7/2015 12:13:26 PM

zim-192-63_C624.vmsd
0.48 KB
File
[datastore1] zim-192-63_C624
7/29/2015 11:09:21 AM
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
all you need is the VMDKs, they contain the data.

you can re-create the configuration file and add the disks.
0
projectsAuthor Commented:
Sorry?

>you can re-create the configuration file and add the disks.

Can you explain this a little more please? I'm not sure I understand.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Select New Virtual Machine, and create a new VM, but then add the existing disks you have.

your data is in the VMDKs, the other files are just configuration files, non-booting VMs is always because the VMDKs virtual disks are corrupted, otherwise, you would have no VMs start or registered with inventory.
1
projectsAuthor Commented:
For example, in my old backup, I've got;

05/26/2015  11:49 PM     4,499,921,408 zim-192-63_C624-disk1.vmdk
05/27/2015  12:49 AM     5,745,056,256 zim-192-63_C624-disk2.vmdk
05/27/2015  12:49 AM               221 zim-192-63_C624.mf
05/27/2015  12:49 AM             8,936 zim-192-63_C624.ovf

I was thinking of copying the most current vmdk files into this directory, then importing to see if it would boot and have the data. If it doesn't work, I'll copy the old vmdk files back.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
if those files are from an EXPORT, you will need to IMPORT them back and not just copy.

see my EE Article here

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE

also if your VMFS datastore is corrupted, you may not be able to write to it correctly, or import them.

e.g. it's not normal for df to crash!
0
projectsAuthor Commented:
Since they don't boot, there is little chance this will work then. I'll try, just for the heck of it, nothing to lose :)
0
projectsAuthor Commented:
PS, the files are not from an export, they were copied directly from the datastore to another location.
Therefore, if I understand, you're saying...

Copying them to an exported vm, then importing is not going to work.
Instead, create a new vm, stop it, copy the saved vmdk file over the new vmdk.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I just noticed two things with the files you listed....


there was an OVF file, an MF file, and the VMDKs are not correct, there should be two, per disks, and these files compared to your originals, are compressed.

just like it looks like and Export!

So to me, they look like you need to Import them, certainly as they are, they will not work....

I'm about to disappear to bed here in the UK, as time is 1.30am, I can continue with you over the weekend, I'll be around....
0
projectsAuthor Commented:
I see two.

zim-192-63_C624.vmx 2.87 KB Virtual Machine [datastore1] zim-192-63_C624 8/7/2015 12:46:53 PM
zim-192-63_C624.vmdk 12,582,910.00 KB Virtual Disk [datastore1] zim-192-63_C624 6/16/2015 4:28:55 PM
zim-192-63_C624_1.vmdk 16,777,220.00 KB Virtual Disk [datastore1] zim-192-63_C624 6/16/2015 4:29:05 PM
zim-192-63_C624.nvram 8.48 KB Non-volatile memory file [datastore1] zim-192-63_C624 8/7/2015 12:46:53 PM
vmware-8.log 28.74 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/6/2015 10:40:41 PM
vmware-7.log 198.36 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/6/2015 7:11:44 PM
vmware-10.log 1,767.88 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/7/2015 12:09:16 PM
zim-192-63_C624-000001.vmdk 1,262,592.00 KB 12,582,910.00 KB Virtual Disk [datastore1] zim-192-63_C624 8/7/2015 12:13:26 PM
vmware.log 240.38 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/7/2015 12:46:53 PM
zim-192-63_C624_1-000001.vmdk 1,655,808.00 KB 16,777,220.00 KB Virtual Disk [datastore1] zim-192-63_C624 8/7/2015 12:13:26 PM
zim-192-63_C624-Snapshot1.vmsn 4,195,572.00 KB Snapshot file [datastore1] zim-192-63_C624 7/29/2015 11:12:25 AM
vmware-6.log 125.12 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/6/2015 12:19:39 PM
vmware-5.log 109.34 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/5/2015 2:29:23 PM
vmware-9.log 762.22 KB Virtual Machine log file [datastore1] zim-192-63_C624 8/6/2015 11:17:02 PM
zim-192-63_C624.vmxf 0.36 KB File [datastore1] zim-192-63_C624 8/7/2015 12:13:26 PM
zim-192-63_C624.vmsd 0.48 KB

Open in new window


I tried as you suggested, creating a new vm, then copying the vmdk over the new one. Same deal, boot problems.
Can the vmdk files be opened and maybe some of the their contents saved assuming they aren't 100% trashed.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
0
projectsAuthor Commented:
Can't get it to work on windows but the files are accessible via Linux nfs shares so am trying from a centos box.

# kpartx -av dev-192-58-64bit-flat.vmdk; mount -o /dev/mapper/loop0p1 /mnt/vmdk/
mount: can't find /mnt/vmdk in /etc/fstab or /etc/mtab

# ls /dev/mapper/
control  VolGroup00-LogVol00  VolGroup00-LogVol01

Any idea how I can mount the image? Maybe I can get at some of the data this way.
It isn't 100% clear to me but I think that kpartx is when you have more than one partition in the vmdk. Since it's a Linux os, it definitely has multiples.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you should be able to mount the partitions to an existing mount point
0
projectsAuthor Commented:
Sorry? Not sure what the means. I'm trying to find the command/method of mounting the vmdk but have yet to find enough info or example which works.

I'm on a centos 6.x box with the vmdk file copied there.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
your error suggests you do not have the folder /mnt/vmware to mount to.
0
projectsAuthor Commented:
The folder is called /mnt/vmdk and yup, it exists.
You're saying this should simply mount? I've shown the errors, I can't get past them.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
no I was simply responding to the error, which says it cannot find /mnt/vmdk
0
projectsAuthor Commented:
I understand that you were saying the mount point was missing and I was confirming that it was not. That is not the cause of the error since the mount point exists. I created it before trying to mount the file. Something else is causing the error which is why I posted it because so far, I've not found the reason.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
corrupted VMDK, there is no error checking on the VMDK for the mount to succeed or fail.
0
projectsAuthor Commented:
That's what I figured sadly.
I've read lots of posts from people asking if there is a way of running a disk checker on the files but never read any definitive answers.

I've often been able to access messed up hardware disks but virtual files, I don't know if there are any recovery methods/programs.
0
projectsAuthor Commented:
That part that gave me hope is that all of the vms log, xml and other files can be read. They aren't trashed so I figured the vmdk files have a slight chance of being recoverable.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
but the vmdk, is also larger, compared to smaller files, so therefore, a larger file, has a far higher probability of being damaged, as it spans more clusters/sectors of the VMFS datastore, or underlying hardware RAID storage.
0
projectsAuthor Commented:
Yes, very true.
The thing is, here is the last log showing that the vm seemed to have been shut down gracefully, giving me hope that what ever happened, the vms seem to go down cleanly.

2015-08-07T23:51:37.775Z| vmx| I120: WORKER: asyncOps=2 maxActiveOps=1 maxPending=0 maxCompleted=1
2015-08-07T23:51:37.842Z| vmx| I120: Vix: [81477 mainDispatch.c:3870]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0
2015-08-07T23:51:37.842Z| vmx| I120: Vix: [81477 mainDispatch.c:3889]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2015-08-07T23:51:37.925Z| vmx| I120: Vix: [81477 mainDispatch.c:3870]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1 additionalError=0
2015-08-07T23:51:37.926Z| vmx| I120: Vix: [81477 mainDispatch.c:3889]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2015-08-07T23:51:37.926Z| vmx| I120: Transitioned vmx/execState/val to poweredOff
2015-08-07T23:51:37.926Z| vmx| I120: VMX idle exit
2015-08-07T23:51:37.926Z| vmx| I120: VMIOP: Exit
2015-08-07T23:51:37.954Z| vmx| I120: Vix: [81477 mainDispatch.c:861]: VMAutomation_LateShutdown()
2015-08-07T23:51:37.954Z| vmx| I120: Vix: [81477 mainDispatch.c:811]: VMAutomationCloseListenerSocket. Closing listener socket.
2015-08-07T23:51:37.961Z| vmx| I120: Flushing VMX VMDB connections
2015-08-07T23:51:37.962Z| vmx| I120: VmdbDbRemoveCnx: Removing Cnx from Db for '/db/connection/#1/'
2015-08-07T23:51:37.962Z| vmx| I120: VmdbCnxDisconnect: Disconnect: closed pipe for pub cnx '/db/connection/#1/' (0)
2015-08-07T23:51:37.968Z| vmx| I120: VMX exit (0).
2015-08-07T23:51:37.968Z| vmx| I120: AIOMGR-S : stat o=1 r=3 w=0 i=0 br=49152 bw=0
2015-08-07T23:51:37.968Z| vmx| I120: OBJLIB-LIB : ObjLib cleanup done.
2015-08-07T23:51:37.968Z| vmx| W110: VMX has left the building: 0.

Open in new window

0
projectsAuthor Commented:
Trying the vmware mount tool

C:\Program Files\VMware\VMware DiskMount Utility>vmware-mount.exe Z: dev58.vmdk
Unable to mount the virtual disk.  The disk may be in use by a virtual
machine or mounted under another drive letter.  If not, verify that the
disk is a virtual disk file, and that the disk file has not been corrupted.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
This error message occurs, if the vmdk is corrupted.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
projectsAuthor Commented:
Thanks. I wanted to make darn sure before having to agree.
Since they are all corrupted, I'll try some other tools since I have nothing to lose.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Sorry, about the outcome.
0
projectsAuthor Commented:
It's not over yet.
I found that you can log into the esxi host and run some local commands against the files.

http://blog.asiantuntijakaveri.fi/2012/01/fixing-broken-vmware-vsphere-5-vmdk.html

# vmkfstools --fix check dev-192-58-64bit.vmdk
Disk is error free

# vmkfstools -i dev-192-58-64bit.vmdk new-disk.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk 'dev-192-58-64bit.vmdk'...
Clone: 100% done.

I then removed the first virtual drive and added this one. Interestingly, this new, supposedly zero error file won't boot either, only shows GRUB at the boot.

so, next I'm making a copy of this working file over to another esxi host and will see if I can mount it as a second drive. I'll first add it into the vm  inventory, boot the vm and see if I can deal with it as any other drive.  Will update.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
the fact that you can clone it, is odd, because it would usually fail, if there was an error in the vmdk!

the vmdk may be error free, but is are contents inside correct as they should be ?

e.g. can you create a new Linux virtual machine, and add these disks, and mount and check the partitions ?
0
projectsAuthor Commented:
That's exactly what I'm trying to find out. If I can find a way of accessing the file, then I can find out how it's contents are. However, there is something about e very vmdk that is messed up so I've not been able to access.

The next thing I tried was to take this supposedly good copy of the file above and trying to add it as a second disk to a vm. I first tried simply pointing to the repaired, existing file but that didn't work. I then created a new disk, saved it, then copied my old file over the new one but that too didn't work, the vm complaining and not booting.

I'll next try what you mentioned, creating a new vm using the repaired disk.
0
projectsAuthor Commented:
So this is odd.
I used a different vm this time and added one of the vmdk's from the crashed host.
This time, the vm boots and I can see the second disk (sdb, sdb1 and sdb2) which is positive but trying to mount it leads to;

# mount -t ext3 /dev/sdb2 /mnt
mount: /dev/sdb2 already mounted or /mnt busy
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
some form of corruption have you check with OS tools.

Linux tools in the OS will have a much better chance of fixing the corruption if possible.
0
projectsAuthor Commented:
So far, the volume looks good.

]# lvdisplay /dev/VolGroup00
  --- Logical volume ---
  LV Path                /dev/VolGroup00/LogVol00
  LV Name                LogVol00
  VG Name                VolGroup00
  LV UUID                v0Y8nF-oujb-hBf2-frzZ-Jtxh-yP2U-YmE8wM
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                7.12 GiB
  Current LE             228
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/VolGroup00/LogVol01
  LV Name                LogVol01
  VG Name                VolGroup00
  LV UUID                rpj5Nk-4x5A-pZga-FHb3-Xslx-Wchl-ISONQx
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                768.00 MiB
  Current LE             24
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

As I understand it, all I need to do now is to mount a LVM volume? But, I can't seem to get past this now.

# mount /dev/VolGroup01/LogVol00 /mnt
mount: you must specify the filesystem type

# vgchange -ay VolGroup00
  2 logical volume(s) in volume group "VolGroup00" now active

# lvs
  LV       VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  LogVol00 VolGroup00 -wi-a-----   7.12g
  LogVol01 VolGroup00 -wi-a----- 768.00m
  lv_root  vg_stor65  -wi-ao----   3.57g
  lv_swap  vg_stor65  -wi-ao----   3.94g

# mount /dev/VolGroup00/LogVol00 /mnt -o ro,user
mount: wrong fs type, bad option, bad superblock on /dev/mapper/VolGroup00-LogVol00,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

EXT3-fs (sdb1): error: no journal found
EXT3-fs (dm-2): error: no journal found
EXT3-fs (dm-2): recovery required on readonly filesystem
EXT3-fs (dm-2): write access will be enabled during recovery
EXT3-fs (dm-2): error: no journal found
EXT3-fs (dm-2): recovery required on readonly filesystem
EXT3-fs (dm-2): write access will be enabled during recovery
EXT3-fs (dm-2): error: no journal found

# tune2fs -j /dev/VolGroup00/LogVol00
tune2fs 1.41.12 (17-May-2010)
The filesystem already has a journal.

# fsck /dev/VolGroup00/LogVol00
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Superblock has an invalid journal (inode 8).
0
projectsAuthor Commented:
Finally rebuilt the drive and found nothing useful on it. Too many directories and stuff to look through but it was a positive.

I'll post another question about where I'm at now and will be using one of the important vmdk files.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.