• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 550
  • Last Modified:

Unable to remove file due to locked somwhere

trying to remove files not in use/orphaned  ,but no success  , as it seem those are locked somehow

I am able to rename n touch to change modify date ,,but trying to remove using rm -rf ..getting messages in logs

2016-08-10T04:01:46.360Z cpu11:108560647)Res3: 6166: Rank violation threshold reached: cid 0xc1d00002, resType 1, cnum 121 vol XYZ
2016-08-10T04:01:49.365Z cpu1:108560647)FS3: 1308: vol 'XYZ', lock at 25355264: [Req mode: 1] Checking liveness:
2016-08-10T04:01:49.365Z cpu1:108560647)[type 10c00002 offset 25355264 v 306, hb offset 3624960
gen 253, mode 1, owner 55a3843c-d8196114-c050-5cf3fc5b9b14 mtime 141955123

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?
0
patron
Asked:
patron
  • 5
  • 4
  • 3
8 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what is the name of the file.

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

locked files are difficult to isolate. shutting down the host is often easier.

Investigating virtual machine file locks on ESXi/ESX (10051)

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?

Yes.
0
 
patronAuthor Commented:
Lock [type 10c00001 offset 146208768 v 2360, hb offset 3289088
gen 7, mode 0, owner 00000000-00000000-0000-000000000000 mtime 214886984
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 315, 7>, gen 8, links 1, type reg, flags 0, uid 0, gid 0, mode 777
len 62277025792, nb 59392 tbz 28138, cow 0, newSinceEpoch 59392, zla 3, bs 1048576


unable to locate host or mac to look for affected host ?

VOMA can be used while vms are running there on my datastore ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what is the filename of the file you think is locked ?

All virtual machines on the affected datastore are powered off or migrated to another datastore.


As I wrote in my last post to your question keeping VMs stored on a datastore you believe to be corrupted is DANGEROUS! if the datastore corruption gets worse you will lose all your VMs. Act on these words and do something about it, before it's too late, and you lose all your VMs.

MIGRATE THEM OFF NOW!

Source
Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (2036767)
1
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
patronAuthor Commented:
vmkfstools -D /vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk
Could not get the dump information for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk' (rv -1)
Could not dump metadata for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk': No such file or directory
Error: No such file or directory

tried to look more .no luck.

and most of files are vmdk

for voma , as per artcile we need to take that datastore into maintenance mode by moving all vms out , cant we do it with running vms and infra ?

i need to check if datasore is really corrupted as now 30 vms are running up & fine

Is there any case if any of  host in cluster may cause such lock due to some malfunction ,and how we can rectify it further ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Move all the VMs off. If you have Storage vMotion you can live migrate off the datastore.

then check entire datastore if any vmdks are remaining,

cross check, and check these are not associated with VMs.

then run voma.
1
 
Richardson PortoSenior Infrastructure SpecialistCommented:
Based on the first message, the lock owner is the host with physical MAC address 5c:f3:fc:5b:9b:14 that refers to a MAC address of an IBM device.

So, go to Configuration tab, Network Adapters and search for the MAC posted above and you will find the host the owner the lock. After find the lock owner, you can try delete the files from an SSH session on that host.

Note that there is a known issue for ESXi 5.1 and 5.5, according with the following KB article: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
 
patronAuthor Commented:
Thanks Richardson,
i did checked for said mac - 5c:f3:fc:5b:9b:14 ,is of vmnic0 from one of my host but that perticuler nic is already down and not in use ,so why it is locked ?
-682BE5EA1E5EC557.jpg
0
 
Richardson PortoSenior Infrastructure SpecialistCommented:
No matter if the NIC is down, since there is no the NIC (or network) that is causing the lock. Who is causing the lock is some process on that host and the association between the NIC and MAC address is just to help identify the lock owner.

So, like I said before, no what you know which host is locking the file, try remove the files through that host.
1
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
moved all your VMs off the datastore yet to get a clear picture of what files remain ?
0
 
patronAuthor Commented:
@Richardson,Thanks for all your help on this..please confirm if we can break/release lock forcefully with no interruption to running prod infra like other vms already there on same host and data store ?

any best practice we can follow ?

@Andrew ,Thanks for all your advise here, will proceed with same once approval in place

but one thing i found from article
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
that it could be issue due to host not patched since long time while we did vmfs and san firmware upgrade in b/w
current host patch level is u2 while it was know issue resolved further in u3
Storage Issues

 During an High Availability failover or a host crash, the .vswp files of powered ON VMs on that host might be left behind on the storageDuring a High Availability failover or host crash, the .vswp files of powered ON virtual machines on that host might be left behind on the storage. When many such failovers or crashes occur, the storage capacity might become full.
VMFS volume is locked
http://pubs.vmware.com/Release_Notes/en/vsphere/55/vsphere-esxi-55u3b-release-notes.html#whatsnew
VMFS volume on an ESXi host might remain locked due to failed metadata operations. An error message similar to the following is observed in vmkernel.log file:
WARNING: LVM: 12976: The volume on the device naa.50002ac002ba0956:1 locked, possibly because some remote host encountered an error during a volume operation and could not recover.
This issue is resolved in this release.


dont you think that host patching is must here as not updated after u2 ?
any major changes or concern we can observe in future if soon not updated from u2 to u3b ?
0
 
Richardson PortoSenior Infrastructure SpecialistCommented:
You can try delete the files from the lock owner without problem and without impact on other virtual machines running on the same host, BUT, if possible, put that host in Maintenance Mode and restart the host.

And of course, consider upgrade to latest version, like described here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
 
patronAuthor Commented:
Thanks a Ton
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 5
  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now