vCenter - can't delete a datastore, fails as "in use"
I've tried every suggestion out there, and still can't remove one datastore in my vCenter.
(I've not been able to reboot the hosts, or even put them in maintnenace - that's not an option at the moment).
2 hosts in a vCenter datacenter, not clustered.
I've been able to remove other datastores in this configuration, but cannot remove this particular datastore. I have 3 active datastores serving VM's other than this one (which has no VM's or files of any kind that I can see).
I've done the following:
- migrated all VM's.
- checked all vmx files for references to the datastore
- found 2 VM's not in inventory or powered on, which referenced the datastore - changed those
- deleted the .dvsData folder from the datastore
- browse datastore from vSphere client - no vm files
- SSH to hosts and "ls -l" the datastore - no files of any kind show (hidden files? don't know, I kid of assume there is, since it won't let me delete the datastore)
- Not used for HA (the 2 hosts are not in a cluster)
- made sure I/O Control is disabled on all hosts
- even followed this: VMware Front Experience: How to disable Storage I/O Control for an unavailable datastore
- no snapshots exist on any of the vm's that *were* on this datastore
- also checked for orphaned *-00000x.vmdk on migrated vm's - none exist
- unmounted the datastore
- this works from one host, but not the other.
- unmounting from host1 gives the red "X" on "no virtual machines exist (all others are green)
- unmounting from host2 unmounts the datastore on host2 but it still shows as mounted on host1
- deleting the datastore (whether mounted or not) fails with the "resource in use" error
(I've not been able to reboot the hosts, or even put them in maintnenace - that's not an option at the moment).
2 hosts in a vCenter datacenter, not clustered.
I've been able to remove other datastores in this configuration, but cannot remove this particular datastore. I have 3 active datastores serving VM's other than this one (which has no VM's or files of any kind that I can see).
I've done the following:
- migrated all VM's.
- checked all vmx files for references to the datastore
- found 2 VM's not in inventory or powered on, which referenced the datastore - changed those
- deleted the .dvsData folder from the datastore
- browse datastore from vSphere client - no vm files
- SSH to hosts and "ls -l" the datastore - no files of any kind show (hidden files? don't know, I kid of assume there is, since it won't let me delete the datastore)
- Not used for HA (the 2 hosts are not in a cluster)
- made sure I/O Control is disabled on all hosts
- even followed this: VMware Front Experience: How to disable Storage I/O Control for an unavailable datastore
- no snapshots exist on any of the vm's that *were* on this datastore
- also checked for orphaned *-00000x.vmdk on migrated vm's - none exist
- unmounted the datastore
- this works from one host, but not the other.
- unmounting from host1 gives the red "X" on "no virtual machines exist (all others are green)
- unmounting from host2 unmounts the datastore on host2 but it still shows as mounted on host1
- deleting the datastore (whether mounted or not) fails with the "resource in use" error
I had this problem recently and had to reboot each ESXi host that had connected to the datastore. Once each host had been rebooted, the offending datastore was gone.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I've requested that this question be closed as follows:
Accepted answer: 0 points for snowdog_2112's comment #a40448096
for the following reason:
Sorry Experts, I ended up "solving" this on my own.
Accepted answer: 0 points for snowdog_2112's comment #a40448096
for the following reason:
Sorry Experts, I ended up "solving" this on my own.
Oh, I mention the word "lock" in my opening post. You never came back and responded....other than to close with Sorry Experts, I ended up "solving" this on my own.!
see
http:#a40406976
have you checked for processes on the datastore?
We spent many weeks, with VMware Escalation Engineers, dumping the VMFS datastore to find the rouge process which had locked the VMFS file system.
So, if you don't mind, I'm Objecting to the Closure.
see
http:#a40406976
have you checked for processes on the datastore?
We spent many weeks, with VMware Escalation Engineers, dumping the VMFS datastore to find the rouge process which had locked the VMFS file system.
So, if you don't mind, I'm Objecting to the Closure.
ASKER
I went through all of those steps and none of the referenced articles led to the solution.
None of the links, suggestions, or references mentioned "host vMotion".
Based on the findings, I stand by my assessment.
You may be correct that a "process is locked", but the links and advice did not lead directly to my solution. I *DID* follow your suggestions and did a ton of investigating based on the links, and links from those links, and vmware community support.
It was, in fact, a last-effort desperation before forcing a maintenance window and host reboot that I vMotioned the 2 guests which seemed to be the source of the problem.
I am very appreciative of the suggestions, but I feel it could lead to a "false positive" for someone else searching for the issue.
If you want the points, I'll award them - as longs as people read the entire post and try a host vMotion first.
None of the links, suggestions, or references mentioned "host vMotion".
Based on the findings, I stand by my assessment.
You may be correct that a "process is locked", but the links and advice did not lead directly to my solution. I *DID* follow your suggestions and did a ton of investigating based on the links, and links from those links, and vmware community support.
It was, in fact, a last-effort desperation before forcing a maintenance window and host reboot that I vMotioned the 2 guests which seemed to be the source of the problem.
I am very appreciative of the suggestions, but I feel it could lead to a "false positive" for someone else searching for the issue.
If you want the points, I'll award them - as longs as people read the entire post and try a host vMotion first.
If my suggested lead to a solution.......
The solution provided ("reboot all hosts connected to the datastore") would clearly solve the issue.
Rebooting a host causes the guests to be vMotioned off. Not sure why you would need a maintenance window for rebooting the hosts, since you can clearly vMotion without impacting the users.
Rebooting a host causes the guests to be vMotioned off. Not sure why you would need a maintenance window for rebooting the hosts, since you can clearly vMotion without impacting the users.
ASKER
FYI - here is more detail on some of the items I went through.
https://communities.vmware.com/thread/493696
https://communities.vmware.com/thread/493696
We have also seen this strange behaviour of not being able to unmount a datastore, if it's the same issue.
have you checked for processes on the datastore?
We spent many weeks, with VMware Escalation Engineers, dumping the VMFS datastore to find the rouge process which had locked the VMFS file system.
We found the offending process, but could not kill the process, and in the end we had to power off the host.