Solved

Cannot browse datastore in vsphere 5.0

Posted on 2014-09-10
21
2,312 Views
Last Modified: 2014-09-19
We have a situation where we cannot browse the contents of a specific datastore, but multiple VM's are running from that datastore in good health.

I have troubleshot this as far as i can, i would really appreciate some expertise here, it feels like a super easy fix, but i can't find it.

environment:
vsphere server at 172.16.1.120, version 5.0.0
esxi physical server at 172.16.1.245, version 5.0
"broken" datastore is on an iSCSI target at 172.16.1.226

if i log into vsphere client directed at the vcenter manager 172.16.1.120, logging in with active directory (a domain admin) - i can see all of our physical esxi hosts, all of our VM's and everything is fine - except this one datastore.

on one esxi host, one VM isn't powered on.  Went to power it on, errored out as insufficient permission.

Weird, i though, so i went to browse the datastore it lives on (as seen from its esxi host) - the datastore appears blank (blank as in all i see is "\" and no folders or files)

in a panic, i check the health of the other vm's on this datastore - they're running just fine (but i can't browse the datastore or see their vmdk files)

so i try to make a new folder on the "blank" datastore, and receive:
"Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “servername.company.com” failed.

Google searched that, found 5-10 articles mentioning various things, but so far all i can find for sure is that if i use vsphere client to log DIRECTLY into the esx host itself as root (172.16.1.245), i can browse the same datastore just fine, see files, make/delete things, no problem

So the issue has to center around vsphere server - the problem is only present when logging in there

I've read that sometimes this happens and if you log into vcenter as root, you have proper permissions... but nobody knows the root password at this time (it isn't root, vmware, blank, or any of our prior passwords we like to use)

So, how can i make this datastore usable from vcenter again?
0
Comment
Question by:FocIS
  • 11
  • 9
21 Comments
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
vCenter just displays information from the Host ESXi server.

The Host is attached to the datastore.

Can you connect directly to the host using vSphere Client and root password, to check you can view the datastore correctly.

If you have access ESXi host via SSH, and can list the contents, I would suspect the datastore is okay, it's just an issue with vCenter Server, inventory service.
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
i can use the vsphere client and root password to connect directly to the esxi host, and the datastore appears and functions correctly

i also suspect it's an issue with vcenter server - what is this inventory service, what can i check?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
Oh the system service itself - i see it is running as localsystem, and is started in good health

i've rebooted the vcenter physical server (172.16.1.120) and the issue persists
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
There is no issue with the ESXi host connection to the datastore.

vCenter Server is just a management server, with the Hosts added.

What versions/builds of vCenter Server and ESXi are you using?

Are you running the latest and current versions of both?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
latest and current no...

vcenter:  5.0.0 build 455964
esxi: 5.0.0-469512-standard

this enterprise has been working perfectly fine for 3+ years without issue, the datastore issue just became apparent about 2 days ago, but i can't pinpoint anything specific that changed
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Yes, those builds are very old. There are original GA Builds from August 2011.

Latest is 1918656. (July 2014)

You do not have any special characters in foldernames ?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
we were always careful to not include spaces or special characters

back in the day i was reading how to upgrade from 5.0.x to 5.1 - and it was incredibly complicated or at least read that way...  i think the upgrade to 5.5 isn't as complicated though...

but on the other hand i wouldn't attempt an upgrade knowing that vcenter can't reliably see one datastore - with my luck it would just delete everything on it or something

what suggestions to make the contents of this datastore visible again as seen from vcenter (instead of just from the esx host)
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
its's either a bug, characters causing the issue, or permissions (which are different) when connecting via vCenter Server.

Do all users have the same issues from ALL workstations?

if you are on the vCenter Server, and connect back to the server via vSphere Client any different ?

Upgrading to 5.1 or 5.5 is the same! and can have complications because of new SSO, which is new to 5.1/5.5.

Updating a bug fix is different, and easier.

Are ALL the datastores on the same SAN ? (IP Address)
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
we only access vcenter thru the actual vcenter server (remote desktop into 172.16.1.120 and then run the client to connect to 172.16.1.120), this is where we can't see the contents of that datastore.

we have multiple other datastores:
- some on local hard drives of the esx host itself, accessed only by the respective esx host
- two other san's (three total san's, different physical MSA cabinets with unique ip addresses, all iSCSI)
- one NAS server used for archival storage, connected to some esx hosts

at least 3 esx hosts have connections to the datastore in question, this datastore is one of two on the iscsi msa cabinet.  if i try to browse the datastore from either of the other hosts (still inside vcenter) the result is the same, it looks blank.  

if i log into vcenter as domain\administrator or domain\my-username (both domain admins) the result is the same.  i haven't tried to log in as root since nobody knows the root password anymore

if i log into the vcenter by way of vmware web client, as a domain admin, i can navigate to where the datastore is, and see the performance info, hosts attached to it, vm's on it... but there's no link to 'browse datastore' from the web client
0
 
LVL 11

Expert Comment

by:Mr Tortur
Comment Utility
Hi,
I would check the permission side if I were you, knowing that you had an "insufficient permission" message starting a VM beside your browsing datastore issue.

Just check, when you log in vCenter vsphere client using domain admin account, selecting the vCenter server at the top of the inventory on the left side, and then going to "permission" tab on the right side, that you see your domain user or group with an administrator role in Vmware.
I don't really think your issue is only a permission one but I would check that to be sure.

Also I understand that your vCenter is a Windows machine (and not the vCenter Linux appliance from Vmware, which is the other possibility) so you do not have any root account on this server, but a local windows administrator account and domain accounts.

As Andrew Hancock said I would also check that this error is not coming of a particular vSphere client : I suggest to try to install/run the vSphere client on another workstation on your Lan and to test loggin in vCenter from this workstation.

And as Andrew Hancock too I would recommend to patch your ESXi servers (very simple using update manager for example). But this is a little more impacting, and I understand your fear. Don't be angry, I did not say you are a coward  ; )

You can also upgrade your vCenter version to a nearly but more up-to-date version, but this is again more and more intrusive for your infrastructure.
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 2

Author Comment

by:FocIS
Comment Utility
In the permission tab, everything appears to be correct - and my account doesn't seem to have any other restriction anywhere else

I was able to start the VM by logging into the esx server directly using the vsphere client, and browse the data store

I will install the vsphere client on another unrelated workstation and see how it looks from there
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
to  narrow down....

which datastore on which host is causing the issue?

and is it the same datastore on ALL hosts?

how are you selecting and looking at the datastore ? e.g. do you go to the summary of each host in vCenter ?

So you cannot start the VM from vCenter Server?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

any error message ?

do the VMs appear, or do they state unknown, inaccessible?

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

okay, interesting, if you select the datastore, from Inventory, Datastores and Datastores Cluster Tab can you browse?

What is this iSCSI SAN ?

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

viewing through vCenter Server is a different process.

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server

why is the datastore not presented to all hosts?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
i don't know how to reply quoted, so i'll reply in points

of the three VM's on this datastore (each with its own host), only one was off.  in vcenter server it appeared as a normal powered-off VM.  i could right-click edit it, change settings, view the console (black screen since it was off), edit memory/cpu, etc.  powering it on is what led to the permissions issue being stated.  it didn't specifically mention the datastore, but troubleshooting led us to check there.

-----------

this datastore isn't presented to all hosts mainly because we don't vmotion, and the storage is pretty much provisioned out for other things, other hosts probably wouldn't be able to find enough useful storage space on it, they use other MSA devices

if i select the datastore from inventory > datastores:
summary tab:  vmfs, 6 hosts, 5 vm's.  capacity 2.73tb, free space 1.04tb
vm's tab: lists the 5 vm's, their space, host cpu, mem, all powered on
hosts tab lists the 6 hosts, memory size, cpu, nic, uptime, etc
configuration tab, properties - lists the name, capacity, extend device, partitions, etc
alarms: none
permisisons:  administrators, role administrator, defined in the vcenter server
but back on 'getting started' for this datastore, clicking BROWSE DATASTORE = no error message, but just shows "/" in folders, no other folders or files

in there, if i click new folder, and hit ok, i get:
Call "FileManager.MakeDirectory" for object "FileManager" on vCenter Server "vs01.domain.com" failed.

-----------

yes, this is an iscsi san device - an hp MSA2324i with (23) 300gb sas drives in two vdisks

----------
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Are you multipathing to this datastore ?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
yes, there are two 4 nic's on each host, two nic's on each of two controllers in the MSA itself
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
and setup correctly for multi path?

active i/o multi paths to the controllers ?

do you have any other datastores on this MSA with the same issue ?
0
 
LVL 2

Author Comment

by:FocIS
Comment Utility
well, i can't say for sure if multipathing is set up exactly correctly, it's a mix of people who've worked on things over the years

each host that can see this datastore, has an iscsi card with multiple nics, and is configured to talk to all 4 of the iscsi servers ip addresses - the "a" side nics are on one switch, the "b" side nics are on another switch

there is another datastore on this msa but i can freely browse that, edit, create, delete, no problem from vcenter server, from any host that's connected to it
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
I would look at updating

1. VMware vSphere vCenter Server
2. VMware vSphere ESXi 5.0

or

.....move the VMs, destroy the datastore, and re-create a new datastore and move the VMs back.

or

Contact VMware Support.
0
 
LVL 2

Author Closing Comment

by:FocIS
Comment Utility
I ended up moving the vm's off the datastore, deleting it from vsphere, deleting it on the iscsi san itself, and recreating it from scratch
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Limit view on Vsphere Web Client home page 4 34
Netapp Snapshots 3 44
VMware vswitch shared gateway 5 45
VMWare iSCSI Issues 7 47
One of the new features of a version 7.0 or later virtual machine, supported in VMware vSphere 4.1, 5.0 or the VMware vSphere Hypervisor ESXi 4.1, ESXi 5.0 often overlooked by VMware Administrators is the ability to add and connect USB devices conne…
HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now