Solved

ESXi 6.0.0 - vmdk file locked and cannot be moved after move operation

Posted on 2016-10-08
30
168 Views
Last Modified: 2016-10-31
I was moving a vm fileset (all files in the vm storage directory) and everything was moving fine until the last file - it came back with a message "unable to access file /vmfs/volumes....../vmname/vmfilename.vmdk since it is locked".

I had previously (prior to the move of files) removed the VM from inventory so it is not associated to the host.  

This is an iscsi connected data source and destination.

I have rebooted not only the esx host but the storage array as well. I've done it in both orders - storage first, then host - host first, then storage.

How do I unlock this file so I can move it?
0
Comment
Question by:netbones
  • 16
  • 14
30 Comments
 
LVL 119
ID: 41835138
0
 
LVL 5

Author Comment

by:netbones
ID: 41835169
It's not another host, thanks.  I only have one host.

I ran vim-cmd vmsvc/getallvms - it does not show the vm registered to this host, just to make sure.  Since there are no other hosts I'm not sure what to do. The articles only discuss being able to identify the lock by the host it's associated to, so that doesn't apply.

What's even more bizarre is that this lone file is actually a snapshot file, I think (but not sure). It's labeled as vmname-000006.vmdk.

I'm tempted to re-add the vm to the inventory from the new storage location without this file.  Is that recommended?
0
 
LVL 119
ID: 41835204
are there any more snapshots, other than this single file ?
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 5

Author Comment

by:netbones
ID: 41835205
Yes, that is the only one that didn't move over.
0
 
LVL 119
ID: 41835210
ah, so there were other snapshots files ?
0
 
LVL 5

Author Comment

by:netbones
ID: 41835211
Yes, 7 in all.
0
 
LVL 5

Author Comment

by:netbones
ID: 41835215
I guess my other option is to move everything back to that store, re-add back into the inventory, make sure it boots correctly.  If so, then remove the snapshots and re-migrate the storage.

What do you think?
0
 
LVL 119
ID: 41835219
Moving virtual machines with snapshots is not recommended.

You should remove the snapshot and then move the VM.

I would not add to inventory without all the files required.

It is a puzzle, that there is a lock on this file, and as a snapshot file, without it, your VM will be corrupted, So you need it!
0
 
LVL 5

Author Comment

by:netbones
ID: 41835221
So I'm stuck either moving everything back and crossing my fingers, or finding out where the lock is coming from which I have not figured out, so if you can help with that please let me know.
0
 
LVL 5

Author Comment

by:netbones
ID: 41836143
Ok if anyone can help me I would appreciate it! I attempted to move back all the files and it left one file again, a different one, locked and I now I'm stuck.

I need a way to unlock this file please help!
0
 
LVL 119
ID: 41836625
What is the SAN?

And is it VMware certified so you can raise an AR with VMware?

There are no issues with SAN?
0
 
LVL 5

Author Comment

by:netbones
ID: 41838818
This is a QNAP TS-831X FW 4.2.2.  Eight Bay Storage array.

Originally, I was transferring the VM's to this via iSCSI.  They were coming from a single, internally attached storage device (not an array, a single disk).  I am trying to migrate off this ESX host so I can consolidate it into a smaller form factor, so I've been systematically moving VM's off of it to the QNAP and then reconnecting the VM's to the host.  

All the other VM's that were on this single disk moved without a problem to the QNAP, this is the only one that ended up with a locked file message.
0
 
LVL 119
ID: 41839613
Have you looked at the Host logs?

You don't have any backup jobs running?
0
 
LVL 5

Author Comment

by:netbones
ID: 41840372
No backup jobs running, I'll look at the logs.
0
 
LVL 119
ID: 41840448
Please check with Qnap support because there have been issued with Qnap firmwares and ESXi in the past with previous EE members.
0
 
LVL 5

Author Comment

by:netbones
ID: 41840473
Hmm ok thanks.  However since this lock is occuring on a single disk inside the esx host, not on the qnap, I tend to think it's not related.  I've rebooted the qnap and the esx host.
0
 
LVL 119
ID: 41840902
Okay I thought it was on the snap.

Is you host supported for VMware?

Are you licenses?
0
 
LVL 5

Author Comment

by:netbones
ID: 41840982
It's a clone system that I built that has been running >2yrs so it's not something that I can say is "officially" supported - however the drive, controller and NIC are fully supported.

I'm on an eval license currently for 6.0, my 5.5 license is still valid but not under support.
0
 
LVL 119

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41841972
Okay well clone parts do not make a qualified system.

I think you would be better off recovering the file you need by completing a new installation of ESXi. Not an upgrade and not wiping or upgrading the datastore and then download this snapshot file you need or reinstate the rest of the files and merge the snapshot.

Unless the file is corrupt and the locked message is a red herring we've seen this before when VMFS or HDD had an error.
0
 
LVL 5

Author Comment

by:netbones
ID: 41842108
That's a great idea I did not consider actually!  I'll let you know.

I find it bizarre that you can't force an unlock on a file TBH. Orphaned locks happen in many situations and this is the first time I've been faced with one where I can't just go in and remove the lock manually.  

I'll repost once I do the resinstall and let you know.
0
 
LVL 119
ID: 41842294
It could be faulty disk?
0
 
LVL 5

Author Comment

by:netbones
ID: 41842397
No, it's not faulty that was checked.
0
 
LVL 119
ID: 41842580
Reinstall then.
0
 
LVL 5

Author Comment

by:netbones
ID: 41842883
Well now this is really getting interesting.

1) I installed on a fresh OS disk ESXi 6.0.  Same behavior.  
2) I can move files between directories within the same store (local disk or iSCSI store)
3) However, when I attempt to move one particular file from either store to the other store, I get the locked file message.
4) I've updated the firmware on the QNAP (just came out today) and that did nothing.

Hmm...
0
 
LVL 119
ID: 41843160
Check VMFS and disks it's corrupted it's not locked.

This confirms my suspicious as we have seen the locked message when datastore is corrupted
0
 
LVL 5

Author Comment

by:netbones
ID: 41843821
Hey Andrew, this is what the result was on the QNAP volume in question. No errors:

[root@localhost:~] esxcli storage vmfs extent list
Volume Name        VMFS UUID                            Extent Number  Device Name                                                                 Partition
-----------------  -----------------------------------  -------------  --------------------------------------------------------------------------  ---------
Software           55f8e33f-12962292-4759-5404a632b349              0  t10.ATA_____WDC_WD1003FZEX2D00MK2A0_______________________WD2DWMC3F1008456          1
datastore1         5800138b-80c5a3ce-03a8-5404a632b349              0  t10.ATA_____Samsung_SSD_840_PRO_Series______________S12PNEAD257661H_____            3
datastore_replica  57538924-8ac661c0-dceb-6805ca3ff14d              0  t10.ATA_____ST4000DM0002D1F2168__________________________________Z3008SLX           1
QNAP-Store3        57f92747-83289064-e1ea-5404a632b349              0  naa.6e843b67ac7d520dc688d45a6da346dd                                                1
QNAP-Store1        57f3d30f-84581847-1f47-5404a632b349              0  naa.6e843b6fb04aa86deffed449edbcbedf                                                1
QNAP-Store2        57f66835-413a8ef4-1102-5404a632b349              0  naa.6e843b6d645f386d1ba3d4fa6d9241d6                                                1
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/naa.6e843b6fb04aa86deffed449edbcbedf:1
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
   Detected VMFS file system (labeled:'QNAP-Store1') with UUID:57f3d30f-84581847-1f47-5404a632b349, Version 5:61
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0

Same thing ran on the local disk:
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/t10.ATA_____ST4000DM0002D1F2168__________________________________Z3008SLX:1
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
   Detected VMFS file system (labeled:'datastore_replica') with UUID:57538924-8ac661c0-dceb-6805ca3ff14d, Version 5:60
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0

So it's not a corruption.
0
 
LVL 119

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41844122
I thought the File is on local disk?

Or is it on Qnap iSCSI?

If it's on Qnap it 's their issue.

If everything has been restarted and you have new ESXi host software ESXi clearly does not like Qnap.

Which to be honest with you no surprise with recent issues we've seen.

Get this escalated to Qnap Support.
0
 
LVL 5

Author Comment

by:netbones
ID: 41844131
It's happening on both - the local disk originally, and also on the QNAP iSCSI.

Within the same disk or iSCIS store, I can transfer the files that report locked between directories on that store.  It's only when I attempt to move the file between different volumes that I have the problem.
0
 
LVL 5

Assisted Solution

by:netbones
netbones earned 0 total points
ID: 41845974
RESOLUTION:

The only way I could get this to work was to completely download to my PC the files in question, then upload them to the VMFS store to a different directory than where the VM originated, then moving the files between the two directories. I was able to do this for 2 of 3 VM's that experienced this problem.

I hate closing out a problem without a resolution as to the cause. I'm glad it's fixed, but only through excruciating download and upload times between the stores and my local system.

I'll leave this open in case anyone has any other ideas, but will close it this week and award points.
0
 
LVL 119
ID: 41845980
not sure what is going on with your host!

but at least you now have a workaround!
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

When converting a physical machine to a virtual machine using VMware vCenter Converter Standalone or vCenter Converter Enterprise, if an adapter type is not selected during the initial customization the resulting virtual machine may contain an IDE d…
When rebooting a vCenters 6.0 and try to connect using vSphere Client we get this issue "Invalid URL: The hostname could not parsed." When we get this error we need to do some changes in the vCenter advanced settings to fix the issue.
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
In this video tutorial I show you the main steps to install and configure  a VMware ESXi6.0 server. The video has my comments as text on the screen and you can pause anytime when needed. Hope this will be helpful. Verify that your hardware and BIO…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question