Solved

ESXi 6.0.0 - vmdk file locked and cannot be moved after move operation

Posted on 2016-10-08
30
81 Views
Last Modified: 2016-10-31
I was moving a vm fileset (all files in the vm storage directory) and everything was moving fine until the last file - it came back with a message "unable to access file /vmfs/volumes....../vmname/vmfilename.vmdk since it is locked".

I had previously (prior to the move of files) removed the VM from inventory so it is not associated to the host.  

This is an iscsi connected data source and destination.

I have rebooted not only the esx host but the storage array as well. I've done it in both orders - storage first, then host - host first, then storage.

How do I unlock this file so I can move it?
0
Comment
Question by:netbones
  • 16
  • 14
30 Comments
 
LVL 118
ID: 41835138
0
 
LVL 5

Author Comment

by:netbones
ID: 41835169
It's not another host, thanks.  I only have one host.

I ran vim-cmd vmsvc/getallvms - it does not show the vm registered to this host, just to make sure.  Since there are no other hosts I'm not sure what to do. The articles only discuss being able to identify the lock by the host it's associated to, so that doesn't apply.

What's even more bizarre is that this lone file is actually a snapshot file, I think (but not sure). It's labeled as vmname-000006.vmdk.

I'm tempted to re-add the vm to the inventory from the new storage location without this file.  Is that recommended?
0
 
LVL 118
ID: 41835204
are there any more snapshots, other than this single file ?
0
 
LVL 5

Author Comment

by:netbones
ID: 41835205
Yes, that is the only one that didn't move over.
0
 
LVL 118
ID: 41835210
ah, so there were other snapshots files ?
0
 
LVL 5

Author Comment

by:netbones
ID: 41835211
Yes, 7 in all.
0
 
LVL 5

Author Comment

by:netbones
ID: 41835215
I guess my other option is to move everything back to that store, re-add back into the inventory, make sure it boots correctly.  If so, then remove the snapshots and re-migrate the storage.

What do you think?
0
 
LVL 118
ID: 41835219
Moving virtual machines with snapshots is not recommended.

You should remove the snapshot and then move the VM.

I would not add to inventory without all the files required.

It is a puzzle, that there is a lock on this file, and as a snapshot file, without it, your VM will be corrupted, So you need it!
0
 
LVL 5

Author Comment

by:netbones
ID: 41835221
So I'm stuck either moving everything back and crossing my fingers, or finding out where the lock is coming from which I have not figured out, so if you can help with that please let me know.
0
 
LVL 5

Author Comment

by:netbones
ID: 41836143
Ok if anyone can help me I would appreciate it! I attempted to move back all the files and it left one file again, a different one, locked and I now I'm stuck.

I need a way to unlock this file please help!
0
 
LVL 118
ID: 41836625
What is the SAN?

And is it VMware certified so you can raise an AR with VMware?

There are no issues with SAN?
0
 
LVL 5

Author Comment

by:netbones
ID: 41838818
This is a QNAP TS-831X FW 4.2.2.  Eight Bay Storage array.

Originally, I was transferring the VM's to this via iSCSI.  They were coming from a single, internally attached storage device (not an array, a single disk).  I am trying to migrate off this ESX host so I can consolidate it into a smaller form factor, so I've been systematically moving VM's off of it to the QNAP and then reconnecting the VM's to the host.  

All the other VM's that were on this single disk moved without a problem to the QNAP, this is the only one that ended up with a locked file message.
0
 
LVL 118
ID: 41839613
Have you looked at the Host logs?

You don't have any backup jobs running?
0
 
LVL 5

Author Comment

by:netbones
ID: 41840372
No backup jobs running, I'll look at the logs.
0
 
LVL 118
ID: 41840448
Please check with Qnap support because there have been issued with Qnap firmwares and ESXi in the past with previous EE members.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 5

Author Comment

by:netbones
ID: 41840473
Hmm ok thanks.  However since this lock is occuring on a single disk inside the esx host, not on the qnap, I tend to think it's not related.  I've rebooted the qnap and the esx host.
0
 
LVL 118
ID: 41840902
Okay I thought it was on the snap.

Is you host supported for VMware?

Are you licenses?
0
 
LVL 5

Author Comment

by:netbones
ID: 41840982
It's a clone system that I built that has been running >2yrs so it's not something that I can say is "officially" supported - however the drive, controller and NIC are fully supported.

I'm on an eval license currently for 6.0, my 5.5 license is still valid but not under support.
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 41841972
Okay well clone parts do not make a qualified system.

I think you would be better off recovering the file you need by completing a new installation of ESXi. Not an upgrade and not wiping or upgrading the datastore and then download this snapshot file you need or reinstate the rest of the files and merge the snapshot.

Unless the file is corrupt and the locked message is a red herring we've seen this before when VMFS or HDD had an error.
0
 
LVL 5

Author Comment

by:netbones
ID: 41842108
That's a great idea I did not consider actually!  I'll let you know.

I find it bizarre that you can't force an unlock on a file TBH. Orphaned locks happen in many situations and this is the first time I've been faced with one where I can't just go in and remove the lock manually.  

I'll repost once I do the resinstall and let you know.
0
 
LVL 118
ID: 41842294
It could be faulty disk?
0
 
LVL 5

Author Comment

by:netbones
ID: 41842397
No, it's not faulty that was checked.
0
 
LVL 118
ID: 41842580
Reinstall then.
0
 
LVL 5

Author Comment

by:netbones
ID: 41842883
Well now this is really getting interesting.

1) I installed on a fresh OS disk ESXi 6.0.  Same behavior.  
2) I can move files between directories within the same store (local disk or iSCSI store)
3) However, when I attempt to move one particular file from either store to the other store, I get the locked file message.
4) I've updated the firmware on the QNAP (just came out today) and that did nothing.

Hmm...
0
 
LVL 118
ID: 41843160
Check VMFS and disks it's corrupted it's not locked.

This confirms my suspicious as we have seen the locked message when datastore is corrupted
0
 
LVL 5

Author Comment

by:netbones
ID: 41843821
Hey Andrew, this is what the result was on the QNAP volume in question. No errors:

[root@localhost:~] esxcli storage vmfs extent list
Volume Name        VMFS UUID                            Extent Number  Device Name                                                                 Partition
-----------------  -----------------------------------  -------------  --------------------------------------------------------------------------  ---------
Software           55f8e33f-12962292-4759-5404a632b349              0  t10.ATA_____WDC_WD1003FZEX2D00MK2A0_______________________WD2DWMC3F1008456          1
datastore1         5800138b-80c5a3ce-03a8-5404a632b349              0  t10.ATA_____Samsung_SSD_840_PRO_Series______________S12PNEAD257661H_____            3
datastore_replica  57538924-8ac661c0-dceb-6805ca3ff14d              0  t10.ATA_____ST4000DM0002D1F2168__________________________________Z3008SLX           1
QNAP-Store3        57f92747-83289064-e1ea-5404a632b349              0  naa.6e843b67ac7d520dc688d45a6da346dd                                                1
QNAP-Store1        57f3d30f-84581847-1f47-5404a632b349              0  naa.6e843b6fb04aa86deffed449edbcbedf                                                1
QNAP-Store2        57f66835-413a8ef4-1102-5404a632b349              0  naa.6e843b6d645f386d1ba3d4fa6d9241d6                                                1
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/naa.6e843b6fb04aa86deffed449edbcbedf:1
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
   Detected VMFS file system (labeled:'QNAP-Store1') with UUID:57f3d30f-84581847-1f47-5404a632b349, Version 5:61
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0

Same thing ran on the local disk:
[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/t10.ATA_____ST4000DM0002D1F2168__________________________________Z3008SLX:1
Checking if device is actively used by other hosts
Running VMFS Checker version 1.2 in check mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
   Detected VMFS file system (labeled:'datastore_replica') with UUID:57538924-8ac661c0-dceb-6805ca3ff14d, Version 5:60
Phase 2: Checking VMFS heartbeat region
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           0

So it's not a corruption.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 41844122
I thought the File is on local disk?

Or is it on Qnap iSCSI?

If it's on Qnap it 's their issue.

If everything has been restarted and you have new ESXi host software ESXi clearly does not like Qnap.

Which to be honest with you no surprise with recent issues we've seen.

Get this escalated to Qnap Support.
0
 
LVL 5

Author Comment

by:netbones
ID: 41844131
It's happening on both - the local disk originally, and also on the QNAP iSCSI.

Within the same disk or iSCIS store, I can transfer the files that report locked between directories on that store.  It's only when I attempt to move the file between different volumes that I have the problem.
0
 
LVL 5

Assisted Solution

by:netbones
netbones earned 0 total points
ID: 41845974
RESOLUTION:

The only way I could get this to work was to completely download to my PC the files in question, then upload them to the VMFS store to a different directory than where the VM originated, then moving the files between the two directories. I was able to do this for 2 of 3 VM's that experienced this problem.

I hate closing out a problem without a resolution as to the cause. I'm glad it's fixed, but only through excruciating download and upload times between the stores and my local system.

I'll leave this open in case anyone has any other ideas, but will close it this week and award points.
0
 
LVL 118
ID: 41845980
not sure what is going on with your host!

but at least you now have a workaround!
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
In this article, I show you step by step with screenshots to assist you - HOW TO: Deploy and Install the VMware vCenter Server Appliance 6.5 (VCSA 6.5), with some helpful tips along the way.
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now