ESXi 6.0.0 - vmdk file locked and cannot be moved after move operation
I was moving a vm fileset (all files in the vm storage directory) and everything was moving fine until the last file - it came back with a message "unable to access file /vmfs/volumes....../vmname
I had previously (prior to the move of files) removed the VM from inventory so it is not associated to the host.
This is an iscsi connected data source and destination.
I have rebooted not only the esx host but the storage array as well. I've done it in both orders - storage first, then host - host first, then storage.
How do I unlock this file so I can move it?
I had previously (prior to the move of files) removed the VM from inventory so it is not associated to the host.
This is an iscsi connected data source and destination.
I have rebooted not only the esx host but the storage array as well. I've done it in both orders - storage first, then host - host first, then storage.
How do I unlock this file so I can move it?
ASKER
It's not another host, thanks. I only have one host.
I ran vim-cmd vmsvc/getallvms - it does not show the vm registered to this host, just to make sure. Since there are no other hosts I'm not sure what to do. The articles only discuss being able to identify the lock by the host it's associated to, so that doesn't apply.
What's even more bizarre is that this lone file is actually a snapshot file, I think (but not sure). It's labeled as vmname-000006.vmdk.
I'm tempted to re-add the vm to the inventory from the new storage location without this file. Is that recommended?
I ran vim-cmd vmsvc/getallvms - it does not show the vm registered to this host, just to make sure. Since there are no other hosts I'm not sure what to do. The articles only discuss being able to identify the lock by the host it's associated to, so that doesn't apply.
What's even more bizarre is that this lone file is actually a snapshot file, I think (but not sure). It's labeled as vmname-000006.vmdk.
I'm tempted to re-add the vm to the inventory from the new storage location without this file. Is that recommended?
are there any more snapshots, other than this single file ?
ASKER
Yes, that is the only one that didn't move over.
ah, so there were other snapshots files ?
ASKER
Yes, 7 in all.
ASKER
I guess my other option is to move everything back to that store, re-add back into the inventory, make sure it boots correctly. If so, then remove the snapshots and re-migrate the storage.
What do you think?
What do you think?
Moving virtual machines with snapshots is not recommended.
You should remove the snapshot and then move the VM.
I would not add to inventory without all the files required.
It is a puzzle, that there is a lock on this file, and as a snapshot file, without it, your VM will be corrupted, So you need it!
You should remove the snapshot and then move the VM.
I would not add to inventory without all the files required.
It is a puzzle, that there is a lock on this file, and as a snapshot file, without it, your VM will be corrupted, So you need it!
ASKER
So I'm stuck either moving everything back and crossing my fingers, or finding out where the lock is coming from which I have not figured out, so if you can help with that please let me know.
ASKER
Ok if anyone can help me I would appreciate it! I attempted to move back all the files and it left one file again, a different one, locked and I now I'm stuck.
I need a way to unlock this file please help!
I need a way to unlock this file please help!
What is the SAN?
And is it VMware certified so you can raise an AR with VMware?
There are no issues with SAN?
And is it VMware certified so you can raise an AR with VMware?
There are no issues with SAN?
ASKER
This is a QNAP TS-831X FW 4.2.2. Eight Bay Storage array.
Originally, I was transferring the VM's to this via iSCSI. They were coming from a single, internally attached storage device (not an array, a single disk). I am trying to migrate off this ESX host so I can consolidate it into a smaller form factor, so I've been systematically moving VM's off of it to the QNAP and then reconnecting the VM's to the host.
All the other VM's that were on this single disk moved without a problem to the QNAP, this is the only one that ended up with a locked file message.
Originally, I was transferring the VM's to this via iSCSI. They were coming from a single, internally attached storage device (not an array, a single disk). I am trying to migrate off this ESX host so I can consolidate it into a smaller form factor, so I've been systematically moving VM's off of it to the QNAP and then reconnecting the VM's to the host.
All the other VM's that were on this single disk moved without a problem to the QNAP, this is the only one that ended up with a locked file message.
Have you looked at the Host logs?
You don't have any backup jobs running?
You don't have any backup jobs running?
ASKER
No backup jobs running, I'll look at the logs.
Please check with Qnap support because there have been issued with Qnap firmwares and ESXi in the past with previous EE members.
ASKER
Hmm ok thanks. However since this lock is occuring on a single disk inside the esx host, not on the qnap, I tend to think it's not related. I've rebooted the qnap and the esx host.
Okay I thought it was on the snap.
Is you host supported for VMware?
Are you licenses?
Is you host supported for VMware?
Are you licenses?
ASKER
It's a clone system that I built that has been running >2yrs so it's not something that I can say is "officially" supported - however the drive, controller and NIC are fully supported.
I'm on an eval license currently for 6.0, my 5.5 license is still valid but not under support.
I'm on an eval license currently for 6.0, my 5.5 license is still valid but not under support.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
That's a great idea I did not consider actually! I'll let you know.
I find it bizarre that you can't force an unlock on a file TBH. Orphaned locks happen in many situations and this is the first time I've been faced with one where I can't just go in and remove the lock manually.
I'll repost once I do the resinstall and let you know.
I find it bizarre that you can't force an unlock on a file TBH. Orphaned locks happen in many situations and this is the first time I've been faced with one where I can't just go in and remove the lock manually.
I'll repost once I do the resinstall and let you know.
It could be faulty disk?
ASKER
No, it's not faulty that was checked.
ASKER
Well now this is really getting interesting.
1) I installed on a fresh OS disk ESXi 6.0. Same behavior.
2) I can move files between directories within the same store (local disk or iSCSI store)
3) However, when I attempt to move one particular file from either store to the other store, I get the locked file message.
4) I've updated the firmware on the QNAP (just came out today) and that did nothing.
Hmm...
1) I installed on a fresh OS disk ESXi 6.0. Same behavior.
2) I can move files between directories within the same store (local disk or iSCSI store)
3) However, when I attempt to move one particular file from either store to the other store, I get the locked file message.
4) I've updated the firmware on the QNAP (just came out today) and that did nothing.
Hmm...
Check VMFS and disks it's corrupted it's not locked.
This confirms my suspicious as we have seen the locked message when datastore is corrupted
This confirms my suspicious as we have seen the locked message when datastore is corrupted
ASKER
Hey Andrew, this is what the result was on the QNAP volume in question. No errors:
[root@localhost:~] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------------- --------------------------
Software 55f8e33f-12962292-4759-540
datastore1 5800138b-80c5a3ce-03a8-540
datastore_replica 57538924-8ac661c0-dceb-680
QNAP-Store3 57f92747-83289064-e1ea-540
QNAP-Store1 57f3d30f-84581847-1f47-540
QNAP-Store2 57f66835-413a8ef4-1102-540
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/naa.6e
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
Detected VMFS file system (labeled:'QNAP-Store1') with UUID:57f3d30f-84581847-1f4
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.
Total Errors Found: 0
Same thing ran on the local disk:
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/t10.AT
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
Detected VMFS file system (labeled:'datastore_replic
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.
Total Errors Found: 0
So it's not a corruption.
[root@localhost:~] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------------- --------------------------
Software 55f8e33f-12962292-4759-540
datastore1 5800138b-80c5a3ce-03a8-540
datastore_replica 57538924-8ac661c0-dceb-680
QNAP-Store3 57f92747-83289064-e1ea-540
QNAP-Store1 57f3d30f-84581847-1f47-540
QNAP-Store2 57f66835-413a8ef4-1102-540
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/naa.6e
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
Detected VMFS file system (labeled:'QNAP-Store1') with UUID:57f3d30f-84581847-1f4
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.
Total Errors Found: 0
Same thing ran on the local disk:
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/t10.AT
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
Detected VMFS file system (labeled:'datastore_replic
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.
Total Errors Found: 0
So it's not a corruption.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
It's happening on both - the local disk originally, and also on the QNAP iSCSI.
Within the same disk or iSCIS store, I can transfer the files that report locked between directories on that store. It's only when I attempt to move the file between different volumes that I have the problem.
Within the same disk or iSCIS store, I can transfer the files that report locked between directories on that store. It's only when I attempt to move the file between different volumes that I have the problem.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
not sure what is going on with your host!
but at least you now have a workaround!
but at least you now have a workaround!
it could be another host.
see here
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051