hongedit
asked on
Unable to start VM - file locked?
Hi
I am unable to start one of my machines after powering it off. Error:
Unable to access file [Datastore] SQL/SQL.vmdk since it is locked.
See the error stack for details on the cause of this problem.
Reason: Failed to lock the file.
Cannot open the disk '/vmfs/volumes/UUID/SQL.vm dk or one of the snapshot disks it depends on.
I tried to clone the VM, but the clone also fails to start with error that the file is not a valid virtual machine.
The Datastore is an iSCSI SAN.
Any ideas?
I am unable to start one of my machines after powering it off. Error:
Unable to access file [Datastore] SQL/SQL.vmdk since it is locked.
See the error stack for details on the cause of this problem.
Reason: Failed to lock the file.
Cannot open the disk '/vmfs/volumes/UUID/SQL.vm
I tried to clone the VM, but the clone also fails to start with error that the file is not a valid virtual machine.
The Datastore is an iSCSI SAN.
Any ideas?
is there any *.LCK file or folder present in the folder with the VM files?
ASKER
Not that I can see, no
ASKER
I ran vmkfstools -D /vmfs/volumes/path/to/file via SSH and the output I got was:
login as: root
root@192.168.23.4's password:
You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.
VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.
Tech Support Mode may be disabled by an administrative user.
Please consult the ESXi Configuration Guide for additional
important information.
~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL/Saint s-SQL.vmdk
Lock [type 10c00001 offset 58759168 v 118, hb offset 3182592
gen 87, mode 0, owner 00000000-00000000-0000-000 000000000 mtime 11611]
Addr <4, 125, 11>, gen 12, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 548, nb 1 tbz 0, cow 0, zla 2, bs 65536
login as: root
root@192.168.23.4's password:
You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.
VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.
Tech Support Mode may be disabled by an administrative user.
Please consult the ESXi Configuration Guide for additional
important information.
~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e
Lock [type 10c00001 offset 58759168 v 118, hb offset 3182592
gen 87, mode 0, owner 00000000-00000000-0000-000
Addr <4, 125, 11>, gen 12, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 548, nb 1 tbz 0, cow 0, zla 2, bs 65536
try restarting management agents on all hosts that connect to that datastore to make sure that there are no other processes locking it like a failed vmotion.
also try vmkfstools -D /vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL/Saint s-SQL[b][i ]-flat[/b][/i].vmdk
ASKER
I have rebooted both the hosts, is that enough? This VM has never been vmotioned anywhere, its only a few days old.
ASKER
Results:
~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL/Saint s-SQL-flat .vmdk
Lock [type 10c00001 offset 58757120 v 61, hb offset 3440640
gen 87, mode 0, owner 00000000-00000000-0000-000 000000000 mtime 1651]
Addr <4, 125, 10>, gen 11, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 107374182400, nb 16135 tbz 0, cow 0, zla 3, bs 1048576
I dont even know what any of it means.
~ # vmkfstools -D /vmfs/volumes/4dc914f7-43e
Lock [type 10c00001 offset 58757120 v 61, hb offset 3440640
gen 87, mode 0, owner 00000000-00000000-0000-000
Addr <4, 125, 10>, gen 11, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 107374182400, nb 16135 tbz 0, cow 0, zla 3, bs 1048576
I dont even know what any of it means.
Do you have a VM backup solution? If so, has it failed on the VM in question?
~coolsport00
~coolsport00
ASKER
No VM Backup as of yet.
This is a new virtual infrastructure, just waiting on some parts to arrive to test out VDR.
This is a new virtual infrastructure, just waiting on some parts to arrive to test out VDR.
The file that corresponds to the disk is locked. There must be a lck file with the VMWare image files that is preventing you to use it.
Look at this VMware KB on VM's not being able to be powered on due to being locked:
http://kb.vmware.com/kb/10051
(I think the KB site is down at the moment, but look when it's up) :)
~coolsport00
http://kb.vmware.com/kb/10051
(I think the KB site is down at the moment, but look when it's up) :)
~coolsport00
Have you used vDR to backup this VM?
Check the VMWare vDR appliance does not have this disk attached to it?
Check the properties of the vDR appliance, edit the Virtual Machine properties, and check how many disks are currently attached, if this VMs disk is attached to vDR, it's either currenty backing up the VM, or Backup has failed.
Detact disk, by Delete the Disk from the VM, but not from the disk.
Check the VMWare vDR appliance does not have this disk attached to it?
Check the properties of the vDR appliance, edit the Virtual Machine properties, and check how many disks are currently attached, if this VMs disk is attached to vDR, it's either currenty backing up the VM, or Backup has failed.
Detact disk, by Delete the Disk from the VM, but not from the disk.
that was Detach the disk, by Deleting the Disk from the VM, but not from the disk!
ASKER
There is definately no lck file :/
I have not used VDR yet, so no chance of that locking anything. Literally just been deployed.
I have not used VDR yet, so no chance of that locking anything. Literally just been deployed.
yeah, reboots would take care of any process locks.
The vmkfstools -D command will return the UUID of the system that has the file lock, so when it comes back 0000's it can mean that the server the command was ran from has the lock or there's some other kind of locking going on.
Has the SAN or host(s) crashed or been powered off unexpectedly?
The vmkfstools -D command will return the UUID of the system that has the file lock, so when it comes back 0000's it can mean that the server the command was ran from has the lock or there's some other kind of locking going on.
Has the SAN or host(s) crashed or been powered off unexpectedly?
Look at the SCSI controller...is it IDE or LSI Logic?
.lck's are used by NFS, so you shouldn't see them. they are seen with 'ls -al' command
ASKER
The SAN has not been powered off "unexpectedly" - only when all the VM's were shut down cleanly and hosts turned off.
SCSI Controller is LSI Logic
SCSI Controller is LSI Logic
ASKER
Ah. Here is the log:
/vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL # ls -al
drwxr-xr-x 1 root root 2240 May 13 21:59 .
drwxr-xr-t 1 root root 1120 May 13 21:59 ..
-rw-r--r-- 1 root root 1 May 10 22:27 Saints-SQL-08744d6 f.hlog
-rw------- 1 root root 107374182400 May 13 16:23 Saints-SQL-flat.vm dk
-rw------- 1 root root 8684 May 13 16:24 Saints-SQL.nvram
-rw------- 1 root root 548 May 10 10:38 Saints-SQL.vmdk
-rw-r--r-- 1 root root 0 May 10 11:35 Saints-SQL.vmsd
-rwxr-xr-x 1 root root 3709 May 13 22:11 Saints-SQL.vmx
-rw-r--r-- 1 root root 265 May 13 22:11 Saints-SQL.vmxf
-rw-r--r-- 1 root root 38506 May 13 17:36 vmware-14.log
-rw-r--r-- 1 root root 38511 May 13 17:38 vmware-15.log
-rw-r--r-- 1 root root 38511 May 13 17:39 vmware-16.log
-rw-r--r-- 1 root root 38492 May 13 17:57 vmware-17.log
-rw-r--r-- 1 root root 38509 May 13 19:46 vmware-18.log
-rw-r--r-- 1 root root 38516 May 13 21:45 vmware-19.log
-rw-r--r-- 1 root root 38515 May 13 21:59 vmware.log
/vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL #
/vmfs/volumes/4dc914f7-43e
drwxr-xr-x 1 root root 2240 May 13 21:59 .
drwxr-xr-t 1 root root 1120 May 13 21:59 ..
-rw-r--r-- 1 root root 1 May 10 22:27 Saints-SQL-08744d6 f.hlog
-rw------- 1 root root 107374182400 May 13 16:23 Saints-SQL-flat.vm dk
-rw------- 1 root root 8684 May 13 16:24 Saints-SQL.nvram
-rw------- 1 root root 548 May 10 10:38 Saints-SQL.vmdk
-rw-r--r-- 1 root root 0 May 10 11:35 Saints-SQL.vmsd
-rwxr-xr-x 1 root root 3709 May 13 22:11 Saints-SQL.vmx
-rw-r--r-- 1 root root 265 May 13 22:11 Saints-SQL.vmxf
-rw-r--r-- 1 root root 38506 May 13 17:36 vmware-14.log
-rw-r--r-- 1 root root 38511 May 13 17:38 vmware-15.log
-rw-r--r-- 1 root root 38511 May 13 17:39 vmware-16.log
-rw-r--r-- 1 root root 38492 May 13 17:57 vmware-17.log
-rw-r--r-- 1 root root 38509 May 13 19:46 vmware-18.log
-rw-r--r-- 1 root root 38516 May 13 21:45 vmware-19.log
-rw-r--r-- 1 root root 38515 May 13 21:59 vmware.log
/vmfs/volumes/4dc914f7-43e
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
is that "Saints-SQL-flat.vm dk" entry a typo or just the way it posted?
BTW...when you remove the disk from the current VM, do *NOT* select to delete from disk; ONLY DELETE/REMOVE from the VM.
ASKER
Oo - that worked. It's powweing on.
Can I just use this as the VM from now on, and delete the other one?
Can I just use this as the VM from now on, and delete the other one?
Or, maybe another option is to rt-click on the VM and select 'Remove from Inventory'. Once removed, open the Datastore Browser the VM is on. Go into the VM's folder, rt-click the .vmx file and select 'Add to Inventory'.
I'd start looking at logs or calling support...
grep Saints-SQL.vmdk /var/log/vmware/hostd.log
grep Saints-SQL.vmdk /var/log/vmkernel
grep Saints-SQL.vmdk /vmfs/volumes/4dc914f7-43e 9978a-1c40 -00259033d d4c/Saints -SQL/vmwar e.log
grep Saints-SQL.vmdk /var/log/vmware/hostd.log
grep Saints-SQL.vmdk /var/log/vmkernel
grep Saints-SQL.vmdk /vmfs/volumes/4dc914f7-43e
YES!..because the other VM doesn't have a disk associated with it. Awesome...glad you're back up :)
What probably happened was, somehow, your .vmx file got corrupted...not sure why or how. Creating a new VM creates a new .vmx file. We coulda spent time looking in the current .vmx file for what went wrong, but why when we can create a whole new VM in seconds :)
nice catch, coolsport00. I was thinking it might have been a .vmdk corruption...
well, as you know...a VM not powering on due to lock could be several different things. next up would certainly have been the vmdk :)
ASKER
Um I think I misclicked for points, can a mod help please!
I'll object for coolsport00 :)
Yeah...just click Request Attention :)
ASKER
Thank you all
Ah...your the man "danm66" ha :)
ASKER
Well just for a laugh I shut down the newly created VM, re-added the disk to the old VM and it booted up, lol.
Go figure.
Go figure.
:)