Solved

Unable to remove file due to locked somwhere

Posted on 2016-08-09
12
106 Views
Last Modified: 2016-08-11
trying to remove files not in use/orphaned  ,but no success  , as it seem those are locked somehow

I am able to rename n touch to change modify date ,,but trying to remove using rm -rf ..getting messages in logs

2016-08-10T04:01:46.360Z cpu11:108560647)Res3: 6166: Rank violation threshold reached: cid 0xc1d00002, resType 1, cnum 121 vol XYZ
2016-08-10T04:01:49.365Z cpu1:108560647)FS3: 1308: vol 'XYZ', lock at 25355264: [Req mode: 1] Checking liveness:
2016-08-10T04:01:49.365Z cpu1:108560647)[type 10c00002 offset 25355264 v 306, hb offset 3624960
gen 253, mode 1, owner 55a3843c-d8196114-c050-5cf3fc5b9b14 mtime 141955123

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?
0
Comment
Question by:patron
  • 5
  • 4
  • 3
12 Comments
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 41749951
what is the name of the file.

How to locate and release or break lock with no interruption to running infra?
will there be any risk if we release or break lock on running infra ?

locked files are difficult to isolate. shutting down the host is often easier.

Investigating virtual machine file locks on ESXi/ESX (10051)

Can we use Voma tool to check for vmfs volume's integrity  on running infra ?

Yes.
0
 
LVL 1

Author Comment

by:patron
ID: 41749983
Lock [type 10c00001 offset 146208768 v 2360, hb offset 3289088
gen 7, mode 0, owner 00000000-00000000-0000-000000000000 mtime 214886984
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 315, 7>, gen 8, links 1, type reg, flags 0, uid 0, gid 0, mode 777
len 62277025792, nb 59392 tbz 28138, cow 0, newSinceEpoch 59392, zla 3, bs 1048576


unable to locate host or mac to look for affected host ?

VOMA can be used while vms are running there on my datastore ?
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 41750004
what is the filename of the file you think is locked ?

All virtual machines on the affected datastore are powered off or migrated to another datastore.


As I wrote in my last post to your question keeping VMs stored on a datastore you believe to be corrupted is DANGEROUS! if the datastore corruption gets worse you will lose all your VMs. Act on these words and do something about it, before it's too late, and you lose all your VMs.

MIGRATE THEM OFF NOW!

Source
Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (2036767)
1
 
LVL 1

Author Comment

by:patron
ID: 41750198
vmkfstools -D /vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk
Could not get the dump information for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk' (rv -1)
Could not dump metadata for '/vmfs/volumes/XYZ/ABC_duplicate/ABC.vmdk': No such file or directory
Error: No such file or directory

tried to look more .no luck.

and most of files are vmdk

for voma , as per artcile we need to take that datastore into maintenance mode by moving all vms out , cant we do it with running vms and infra ?

i need to check if datasore is really corrupted as now 30 vms are running up & fine

Is there any case if any of  host in cluster may cause such lock due to some malfunction ,and how we can rectify it further ?
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 41750235
Move all the VMs off. If you have Storage vMotion you can live migrate off the datastore.

then check entire datastore if any vmdks are remaining,

cross check, and check these are not associated with VMs.

then run voma.
1
 
LVL 9

Accepted Solution

by:
Richardson Porto earned 250 total points
ID: 41751259
Based on the first message, the lock owner is the host with physical MAC address 5c:f3:fc:5b:9b:14 that refers to a MAC address of an IBM device.

So, go to Configuration tab, Network Adapters and search for the MAC posted above and you will find the host the owner the lock. After find the lock owner, you can try delete the files from an SSH session on that host.

Note that there is a known issue for ESXi 5.1 and 5.5, according with the following KB article: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 
LVL 1

Author Comment

by:patron
ID: 41751853
Thanks Richardson,
i did checked for said mac - 5c:f3:fc:5b:9b:14 ,is of vmnic0 from one of my host but that perticuler nic is already down and not in use ,so why it is locked ?
-682BE5EA1E5EC557.jpg
0
 
LVL 9

Assisted Solution

by:Richardson Porto
Richardson Porto earned 250 total points
ID: 41751938
No matter if the NIC is down, since there is no the NIC (or network) that is causing the lock. Who is causing the lock is some process on that host and the association between the NIC and MAC address is just to help identify the lock owner.

So, like I said before, no what you know which host is locking the file, try remove the files through that host.
1
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 41751953
moved all your VMs off the datastore yet to get a clear picture of what files remain ?
0
 
LVL 1

Assisted Solution

by:patron
patron earned 0 total points
ID: 41752058
@Richardson,Thanks for all your help on this..please confirm if we can break/release lock forcefully with no interruption to running prod infra like other vms already there on same host and data store ?

any best practice we can follow ?

@Andrew ,Thanks for all your advise here, will proceed with same once approval in place

but one thing i found from article
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
that it could be issue due to host not patched since long time while we did vmfs and san firmware upgrade in b/w
current host patch level is u2 while it was know issue resolved further in u3
Storage Issues

 During an High Availability failover or a host crash, the .vswp files of powered ON VMs on that host might be left behind on the storageDuring a High Availability failover or host crash, the .vswp files of powered ON virtual machines on that host might be left behind on the storage. When many such failovers or crashes occur, the storage capacity might become full.
VMFS volume is locked
http://pubs.vmware.com/Release_Notes/en/vsphere/55/vsphere-esxi-55u3b-release-notes.html#whatsnew
VMFS volume on an ESXi host might remain locked due to failed metadata operations. An error message similar to the following is observed in vmkernel.log file:
WARNING: LVM: 12976: The volume on the device naa.50002ac002ba0956:1 locked, possibly because some remote host encountered an error during a volume operation and could not recover.
This issue is resolved in this release.


dont you think that host patching is must here as not updated after u2 ?
any major changes or concern we can observe in future if soon not updated from u2 to u3b ?
0
 
LVL 9

Assisted Solution

by:Richardson Porto
Richardson Porto earned 250 total points
ID: 41752062
You can try delete the files from the lock owner without problem and without impact on other virtual machines running on the same host, BUT, if possible, put that host in Maintenance Mode and restart the host.

And of course, consider upgrade to latest version, like described here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113615
1
 
LVL 1

Author Closing Comment

by:patron
ID: 41753371
Thanks a Ton
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Microservice architecture adoption brings many advantages, but can add intricacy. Selecting the right orchestration tool is most important for business specific needs.
HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now