?
Solved

Unable to remove file due to locked somwhere

Posted on 2016-08-09
12
Medium Priority
?
294 Views
Last Modified: 2016-08-11
trying to remove files not in use/orphaned  ,but no success  , as it seem those are locked somehow

I am able to rename n touch to change modify date ,,but trying to remove using rm -rf ..getting messages in logs

2016-08-10T04:01:46.360Z cpu11:108560647)Res3: 6166: Rank violation threshold reached: cid 0xc1d00002, resType 1, cnum 121 vol XYZ
2016-08-10T04:01:49.365Z cpu1:108560647)FS3: 1308: vol 'XYZ', lock at 25355264: [Req mode: 1] Checking liveness:
2016-08-10T04:01:49.365Z cpu1:108560647)[type 10c00002 offset 25355264 v 306, hb offset 3624960
gen 253, mode 1, owner 55a3843c-d8196114-c050-5cf3fc5b9b14 mtime 141955123

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?
0
Comment
Question by:patron
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
  • 3
12 Comments
 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 41749951
what is the name of the file.

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

locked files are difficult to isolate. shutting down the host is often easier.

Investigating virtual machine file locks on ESXi/ESX (10051)

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?

Yes.
0
 
LVL 1

Author Comment

by:patron
ID: 41749983
Lock [type 10c00001 offset 146208768 v 2360, hb offset 3289088
gen 7, mode 0, owner 00000000-00000000-0000-000000000000 mtime 214886984
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 315, 7>, gen 8, links 1, type reg, flags 0, uid 0, gid 0, mode 777
len 62277025792, nb 59392 tbz 28138, cow 0, newSinceEpoch 59392, zla 3, bs 1048576


unable to locate host or mac to look for affected host ?

VOMA can be used while vms are running there on my datastore ?
0
 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 41750004
what is the filename of the file you think is locked ?

All virtual machines on the affected datastore are powered off or migrated to another datastore.


As I wrote in my last post to your question keeping VMs stored on a datastore you believe to be corrupted is DANGEROUS! if the datastore corruption gets worse you will lose all your VMs. Act on these words and do something about it, before it's too late, and you lose all your VMs.

MIGRATE THEM OFF NOW!

Source
Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (2036767)
1
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 
LVL 1

Author Comment

by:patron
ID: 41750198
vmkfstools -D /vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk
Could not get the dump information for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk' (rv -1)
Could not dump metadata for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk': No such file or directory
Error: No such file or directory

tried to look more .no luck.

and most of files are vmdk

for voma , as per artcile we need to take that datastore into maintenance mode by moving all vms out , cant we do it with running vms and infra ?

i need to check if datasore is really corrupted as now 30 vms are running up & fine

Is there any case if any of  host in cluster may cause such lock due to some malfunction ,and how we can rectify it further ?
0
 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 41750235
Move all the VMs off. If you have Storage vMotion you can live migrate off the datastore.

then check entire datastore if any vmdks are remaining,

cross check, and check these are not associated with VMs.

then run voma.
1
 
LVL 11

Accepted Solution

by:
Richardson Porto earned 1000 total points
ID: 41751259
Based on the first message, the lock owner is the host with physical MAC address 5c:f3:fc:5b:9b:14 that refers to a MAC address of an IBM device.

So, go to Configuration tab, Network Adapters and search for the MAC posted above and you will find the host the owner the lock. After find the lock owner, you can try delete the files from an SSH session on that host.

Note that there is a known issue for ESXi 5.1 and 5.5, according with the following KB article: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
 
LVL 1

Author Comment

by:patron
ID: 41751853
Thanks Richardson,
i did checked for said mac - 5c:f3:fc:5b:9b:14 ,is of vmnic0 from one of my host but that perticuler nic is already down and not in use ,so why it is locked ?
-682BE5EA1E5EC557.jpg
0
 
LVL 11

Assisted Solution

by:Richardson Porto
Richardson Porto earned 1000 total points
ID: 41751938
No matter if the NIC is down, since there is no the NIC (or network) that is causing the lock. Who is causing the lock is some process on that host and the association between the NIC and MAC address is just to help identify the lock owner.

So, like I said before, no what you know which host is locking the file, try remove the files through that host.
1
 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 41751953
moved all your VMs off the datastore yet to get a clear picture of what files remain ?
0
 
LVL 1

Assisted Solution

by:patron
patron earned 0 total points
ID: 41752058
@Richardson,Thanks for all your help on this..please confirm if we can break/release lock forcefully with no interruption to running prod infra like other vms already there on same host and data store ?

any best practice we can follow ?

@Andrew ,Thanks for all your advise here, will proceed with same once approval in place

but one thing i found from article
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
that it could be issue due to host not patched since long time while we did vmfs and san firmware upgrade in b/w
current host patch level is u2 while it was know issue resolved further in u3
Storage Issues

 During an High Availability failover or a host crash, the .vswp files of powered ON VMs on that host might be left behind on the storageDuring a High Availability failover or host crash, the .vswp files of powered ON virtual machines on that host might be left behind on the storage. When many such failovers or crashes occur, the storage capacity might become full.
VMFS volume is locked
http://pubs.vmware.com/Release_Notes/en/vsphere/55/vsphere-esxi-55u3b-release-notes.html#whatsnew
VMFS volume on an ESXi host might remain locked due to failed metadata operations. An error message similar to the following is observed in vmkernel.log file:
WARNING: LVM: 12976: The volume on the device naa.50002ac002ba0956:1 locked, possibly because some remote host encountered an error during a volume operation and could not recover.
This issue is resolved in this release.


dont you think that host patching is must here as not updated after u2 ?
any major changes or concern we can observe in future if soon not updated from u2 to u3b ?
0
 
LVL 11

Assisted Solution

by:Richardson Porto
Richardson Porto earned 1000 total points
ID: 41752062
You can try delete the files from the lock owner without problem and without impact on other virtual machines running on the same host, BUT, if possible, put that host in Maintenance Mode and restart the host.

And of course, consider upgrade to latest version, like described here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
 
LVL 1

Author Closing Comment

by:patron
ID: 41753371
Thanks a Ton
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question