Unable to start VM - file locked?

hongedit
hongedit used Ask the Experts™
on
Hi

I am unable to start one of my machines after powering it off. Error:

Unable to access file [Datastore] SQL/SQL.vmdk since it is locked.

See the error stack for details on the cause of this problem.

Reason: Failed to lock the file.
Cannot open the disk '/vmfs/volumes/UUID/SQL.vmdk or one of the snapshot disks it depends on.

I tried to clone the VM, but the clone also fails to start with error that the file is not a valid virtual machine.

The Datastore is an iSCSI SAN.

Any ideas?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
is there any *.LCK file or folder present in the folder with the VM files?

Author

Commented:
Not that I can see, no

Author

Commented:
I ran vmkfstools -D /vmfs/volumes/path/to/file via SSH and the output I got was:

login as: root
root@192.168.23.4's password:
You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

Tech Support Mode may be disabled by an administrative user.
Please consult the ESXi Configuration Guide for additional
important information.

~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL/Saints-SQL.vmdk
Lock [type 10c00001 offset 58759168 v 118, hb offset 3182592
gen 87, mode 0, owner 00000000-00000000-0000-000000000000 mtime 11611]
Addr <4, 125, 11>, gen 12, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 548, nb 1 tbz 0, cow 0, zla 2, bs 65536
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Danny McDanielClinical Systems Analyst

Commented:
try restarting management agents on all hosts that connect to that datastore to make sure that there are no other processes locking it like a failed vmotion.
Danny McDanielClinical Systems Analyst

Commented:
also try vmkfstools -D /vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL/Saints-SQL[b][i]-flat[/b][/i].vmdk

Author

Commented:
I have rebooted both the hosts, is that enough? This VM has never been vmotioned anywhere, its only a few days old.

Author

Commented:
Results:

~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL/Saints-SQL-flat.vmdk
Lock [type 10c00001 offset 58757120 v 61, hb offset 3440640
gen 87, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1651]
Addr <4, 125, 10>, gen 11, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 107374182400, nb 16135 tbz 0, cow 0, zla 3, bs 1048576

I dont even know what any of it means.
Top Expert 2010

Commented:
Do you have a VM backup solution? If so, has it failed on the VM in question?

~coolsport00

Author

Commented:
No VM Backup as of yet.

This is a new virtual infrastructure, just waiting on some parts to arrive to test out VDR.
The file that corresponds to the disk is locked. There must be a lck file with the VMWare image files that is preventing you to use it.
Top Expert 2010

Commented:
Look at this VMware KB on VM's not being able to be powered on due to being locked:
http://kb.vmware.com/kb/10051

(I think the KB site is down at the moment, but look when it's up) :)

~coolsport00
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Have you used vDR to backup this VM?

Check the VMWare vDR appliance does not have this disk attached to it?

Check the properties of the vDR appliance, edit the Virtual Machine properties, and check how many disks are currently attached, if this VMs disk is attached to vDR, it's either currenty backing up the VM, or Backup has failed.

Detact disk, by Delete the Disk from the VM, but not from the disk.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
that was Detach the disk, by Deleting the Disk from the VM, but not from the disk!

Author

Commented:
There is definately no lck file :/

I have not used VDR yet, so no chance of that locking anything. Literally just been deployed.
Danny McDanielClinical Systems Analyst

Commented:
yeah, reboots would take care of any process locks.

The vmkfstools -D command will return the UUID of the system that has the file lock, so when it comes back 0000's it can mean that the server the command was ran from has the lock or there's some other kind of locking going on.

Has the SAN or host(s) crashed or been powered off unexpectedly?
Top Expert 2010

Commented:
Look at the SCSI controller...is it IDE or LSI Logic?
Danny McDanielClinical Systems Analyst

Commented:
.lck's are used by NFS, so you shouldn't see them.  they are seen with 'ls -al' command

Author

Commented:
The SAN has not been powered off "unexpectedly" - only when all the VM's were shut down cleanly and hosts turned off.

SCSI Controller is LSI Logic

Author

Commented:
Ah. Here is the log:

/vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL # ls -al
drwxr-xr-x    1 root     root               2240 May 13 21:59 .
drwxr-xr-t    1 root     root               1120 May 13 21:59 ..
-rw-r--r--    1 root     root                  1 May 10 22:27 Saints-SQL-08744d6         f.hlog
-rw-------    1 root     root       107374182400 May 13 16:23 Saints-SQL-flat.vm         dk
-rw-------    1 root     root               8684 May 13 16:24 Saints-SQL.nvram
-rw-------    1 root     root                548 May 10 10:38 Saints-SQL.vmdk
-rw-r--r--    1 root     root                  0 May 10 11:35 Saints-SQL.vmsd
-rwxr-xr-x    1 root     root               3709 May 13 22:11 Saints-SQL.vmx
-rw-r--r--    1 root     root                265 May 13 22:11 Saints-SQL.vmxf
-rw-r--r--    1 root     root              38506 May 13 17:36 vmware-14.log
-rw-r--r--    1 root     root              38511 May 13 17:38 vmware-15.log
-rw-r--r--    1 root     root              38511 May 13 17:39 vmware-16.log
-rw-r--r--    1 root     root              38492 May 13 17:57 vmware-17.log
-rw-r--r--    1 root     root              38509 May 13 19:46 vmware-18.log
-rw-r--r--    1 root     root              38516 May 13 21:45 vmware-19.log
-rw-r--r--    1 root     root              38515 May 13 21:59 vmware.log
/vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL #
Top Expert 2010
Commented:
Try this...create a new VM withOUT creating a virtual disk with it. Remove the hard (virtual) disk from the current problem VM and attach it to your new VM. Try powering on the new VM and post back the results.

~coolsport00
Danny McDanielClinical Systems Analyst

Commented:
is that "Saints-SQL-flat.vm         dk" entry a typo or just the way it posted?
Top Expert 2010

Commented:
BTW...when you remove the disk from the current VM, do *NOT* select to delete from disk; ONLY DELETE/REMOVE from the VM.

Author

Commented:
Oo - that worked. It's powweing on.

Can I just use this as the VM from now on, and delete the other one?
Top Expert 2010

Commented:
Or, maybe another option is to rt-click on the VM and select 'Remove from Inventory'. Once removed, open the Datastore Browser the VM is on. Go into the VM's folder, rt-click the .vmx file and select 'Add to Inventory'.
Danny McDanielClinical Systems Analyst

Commented:
I'd start looking at logs or calling support...

grep Saints-SQL.vmdk /var/log/vmware/hostd.log
grep Saints-SQL.vmdk /var/log/vmkernel
grep Saints-SQL.vmdk /vmfs/volumes/4dc914f7-43e9978a-1c40-00259033dd4c/Saints-SQL/vmware.log
Top Expert 2010

Commented:
YES!..because the other VM doesn't have a disk associated with it. Awesome...glad you're back up :)
Top Expert 2010

Commented:
What probably happened was, somehow, your .vmx file got corrupted...not sure why or how. Creating a new VM creates a new .vmx file. We coulda spent time looking in the current .vmx file for what went wrong, but why when we can create a whole new VM in seconds :)
Danny McDanielClinical Systems Analyst

Commented:
nice catch, coolsport00.  I was thinking it might have been a .vmdk corruption...
Top Expert 2010

Commented:
well, as you know...a VM not powering on due to lock could be several different things. next up would certainly have been the vmdk :)

Author

Commented:
Um I think I misclicked for points, can a mod help please!
Danny McDanielClinical Systems Analyst

Commented:
I'll object for coolsport00  :)
Top Expert 2010

Commented:
Yeah...just click Request Attention :)

Author

Commented:
Thank you all
Top Expert 2010

Commented:
Ah...your the man "danm66" ha :)

Author

Commented:
Well just for a laugh I shut down the newly created VM, re-added the disk to the old VM and it booted up, lol.

Go figure.
Top Expert 2010

Commented:
:)

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial