VMware VM fails to boot after Recovery from SAN disk-space "Full"

Akulsh
Akulsh used Ask the Experts™
on
Being out of town, I could not recover white-space in time, and all volumes were seen as full by Dell Storage SAN. All VMs froze. Eventually we could fix this problem and most VMs booted up OK. However, in case of one VM with many disks -- it is SQL server -- we see this error on trying to start it - "File system specific implementation of Lookup[file] failed".

Rebooting host ESXi server has not helped, neither VM migrating.

Please advise. Thanks.

AKK
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
I would highly recommend you have a backup of the VM then you can attempt the following:

To solve the issue you need to check the .vmdk file and repair it. If you need to repair it is good to have a backup of the file just in case.

1. Enable SSH access to your ESXi host and login
2. Check the main .vmdk file if it corrupted or not
vmkfstools -x check /path/to/your/machine.vmdk
3. To repair the file type:
vmkfstools -x repair /path/to/your/machine.vmdk
4. Start you VM!
In my case:
[root@esx02:/vmfs/volumes/5b3a61e9-248e71e2-e165-0cc47ae7cfd1/machine] vmkfstools -x check machine.vmdk
Disk needs repair.
[root@esx02:/vmfs/volumes/5b3a61e9-248e71e2-e165-0cc47ae7cfd1/machine] vmkfstools -x repair machine.vmdk
Disk was successfully repaired.


Reference : https://www.vionblog.com/vmware-6-5-cant-start-vm-host-crash/
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Can you screenshot the VM folder, and check if there are any snapshots ?

Author

Commented:
Tguy565,
Thanks for replying. Enclosed is the screenshot of all  VMDK files
xxx-sql_1.vmdk is probably corrupt, but it is 0 KB. Can it be repaired?

Andrew,
Thanks for replying. There were no screenshots.

AKK
VMDK-of-Non-bootable-VM.JPG
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
If it's zero the data has gone, and is beyond repair!

There would be zero bytes to repair!

I'm afraid it's a restore.

But you may get a better view at the shell prompt.
@Akulsh

I am afraid Andrew is correct, if the value shows 0 then it is most likely not repairable. I would recommend that you restore those VMDX from backup if you have one.
Your "One" chance might be the following application but even then it is a long shot:

https://www.nucleustechnologies.com/vmdk-recovery/

Author

Commented:
Thank you both for your expertise and time.

Author

Commented:
Luckily I did have a Veeam backup though Restore was failing. I could not restore *.vmdk files either since Veeam did not want to overwrite existing files, even after I renamed them. (Somehow *-flat.vmdk files had been generated.)

Finally I was able to restore all VMDK files to a different shutdown-VM. From there, I was able to move them to the folder of SQL VM, after renaming those -flat files.

The VM became healthy and could be rebooted. Thank you both.
@Akulsh, Glad you got your SQL svr back up.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial