Cannot browse datastore in vsphere 5.0

We have a situation where we cannot browse the contents of a specific datastore, but multiple VM's are running from that datastore in good health.

I have troubleshot this as far as i can, i would really appreciate some expertise here, it feels like a super easy fix, but i can't find it.

environment:
vsphere server at 172.16.1.120, version 5.0.0
esxi physical server at 172.16.1.245, version 5.0
"broken" datastore is on an iSCSI target at 172.16.1.226

if i log into vsphere client directed at the vcenter manager 172.16.1.120, logging in with active directory (a domain admin) - i can see all of our physical esxi hosts, all of our VM's and everything is fine - except this one datastore.

on one esxi host, one VM isn't powered on.  Went to power it on, errored out as insufficient permission.

Weird, i though, so i went to browse the datastore it lives on (as seen from its esxi host) - the datastore appears blank (blank as in all i see is "\" and no folders or files)

in a panic, i check the health of the other vm's on this datastore - they're running just fine (but i can't browse the datastore or see their vmdk files)

so i try to make a new folder on the "blank" datastore, and receive:
"Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “servername.company.com” failed.

Google searched that, found 5-10 articles mentioning various things, but so far all i can find for sure is that if i use vsphere client to log DIRECTLY into the esx host itself as root (172.16.1.245), i can browse the same datastore just fine, see files, make/delete things, no problem

So the issue has to center around vsphere server - the problem is only present when logging in there

I've read that sometimes this happens and if you log into vcenter as root, you have proper permissions... but nobody knows the root password at this time (it isn't root, vmware, blank, or any of our prior passwords we like to use)

So, how can i make this datastore usable from vcenter again?
LVL 2
FocISAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
vCenter just displays information from the Host ESXi server.

The Host is attached to the datastore.

Can you connect directly to the host using vSphere Client and root password, to check you can view the datastore correctly.

If you have access ESXi host via SSH, and can list the contents, I would suspect the datastore is okay, it's just an issue with vCenter Server, inventory service.
0
FocISAuthor Commented:
i can use the vsphere client and root password to connect directly to the esxi host, and the datastore appears and functions correctly

i also suspect it's an issue with vcenter server - what is this inventory service, what can i check?
0
FocISAuthor Commented:
Oh the system service itself - i see it is running as localsystem, and is started in good health

i've rebooted the vcenter physical server (172.16.1.120) and the issue persists
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
There is no issue with the ESXi host connection to the datastore.

vCenter Server is just a management server, with the Hosts added.

What versions/builds of vCenter Server and ESXi are you using?

Are you running the latest and current versions of both?
0
FocISAuthor Commented:
latest and current no...

vcenter:  5.0.0 build 455964
esxi: 5.0.0-469512-standard

this enterprise has been working perfectly fine for 3+ years without issue, the datastore issue just became apparent about 2 days ago, but i can't pinpoint anything specific that changed
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, those builds are very old. There are original GA Builds from August 2011.

Latest is 1918656. (July 2014)

You do not have any special characters in foldernames ?
0
FocISAuthor Commented:
we were always careful to not include spaces or special characters

back in the day i was reading how to upgrade from 5.0.x to 5.1 - and it was incredibly complicated or at least read that way...  i think the upgrade to 5.5 isn't as complicated though...

but on the other hand i wouldn't attempt an upgrade knowing that vcenter can't reliably see one datastore - with my luck it would just delete everything on it or something

what suggestions to make the contents of this datastore visible again as seen from vcenter (instead of just from the esx host)
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
its's either a bug, characters causing the issue, or permissions (which are different) when connecting via vCenter Server.

Do all users have the same issues from ALL workstations?

if you are on the vCenter Server, and connect back to the server via vSphere Client any different ?

Upgrading to 5.1 or 5.5 is the same! and can have complications because of new SSO, which is new to 5.1/5.5.

Updating a bug fix is different, and easier.

Are ALL the datastores on the same SAN ? (IP Address)
0
FocISAuthor Commented:
we only access vcenter thru the actual vcenter server (remote desktop into 172.16.1.120 and then run the client to connect to 172.16.1.120), this is where we can't see the contents of that datastore.

we have multiple other datastores:
- some on local hard drives of the esx host itself, accessed only by the respective esx host
- two other san's (three total san's, different physical MSA cabinets with unique ip addresses, all iSCSI)
- one NAS server used for archival storage, connected to some esx hosts

at least 3 esx hosts have connections to the datastore in question, this datastore is one of two on the iscsi msa cabinet.  if i try to browse the datastore from either of the other hosts (still inside vcenter) the result is the same, it looks blank.  

if i log into vcenter as domain\administrator or domain\my-username (both domain admins) the result is the same.  i haven't tried to log in as root since nobody knows the root password anymore

if i log into the vcenter by way of vmware web client, as a domain admin, i can navigate to where the datastore is, and see the performance info, hosts attached to it, vm's on it... but there's no link to 'browse datastore' from the web client
0
Mr TorturSystem EngineerCommented:
Hi,
I would check the permission side if I were you, knowing that you had an "insufficient permission" message starting a VM beside your browsing datastore issue.

Just check, when you log in vCenter vsphere client using domain admin account, selecting the vCenter server at the top of the inventory on the left side, and then going to "permission" tab on the right side, that you see your domain user or group with an administrator role in Vmware.
I don't really think your issue is only a permission one but I would check that to be sure.

Also I understand that your vCenter is a Windows machine (and not the vCenter Linux appliance from Vmware, which is the other possibility) so you do not have any root account on this server, but a local windows administrator account and domain accounts.

As Andrew Hancock said I would also check that this error is not coming of a particular vSphere client : I suggest to try to install/run the vSphere client on another workstation on your Lan and to test loggin in vCenter from this workstation.

And as Andrew Hancock too I would recommend to patch your ESXi servers (very simple using update manager for example). But this is a little more impacting, and I understand your fear. Don't be angry, I did not say you are a coward  ; )

You can also upgrade your vCenter version to a nearly but more up-to-date version, but this is again more and more intrusive for your infrastructure.
0
FocISAuthor Commented:
In the permission tab, everything appears to be correct - and my account doesn't seem to have any other restriction anywhere else

I was able to start the VM by logging into the esx server directly using the vsphere client, and browse the data store

I will install the vsphere client on another unrelated workstation and see how it looks from there
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
to  narrow down....

which datastore on which host is causing the issue?

and is it the same datastore on ALL hosts?

how are you selecting and looking at the datastore ? e.g. do you go to the summary of each host in vCenter ?

So you cannot start the VM from vCenter Server?
0
FocISAuthor Commented:
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

any error message ?

do the VMs appear, or do they state unknown, inaccessible?

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

okay, interesting, if you select the datastore, from Inventory, Datastores and Datastores Cluster Tab can you browse?

What is this iSCSI SAN ?

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

viewing through vCenter Server is a different process.

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server

why is the datastore not presented to all hosts?
0
FocISAuthor Commented:
i don't know how to reply quoted, so i'll reply in points

of the three VM's on this datastore (each with its own host), only one was off.  in vcenter server it appeared as a normal powered-off VM.  i could right-click edit it, change settings, view the console (black screen since it was off), edit memory/cpu, etc.  powering it on is what led to the permissions issue being stated.  it didn't specifically mention the datastore, but troubleshooting led us to check there.

-----------

this datastore isn't presented to all hosts mainly because we don't vmotion, and the storage is pretty much provisioned out for other things, other hosts probably wouldn't be able to find enough useful storage space on it, they use other MSA devices

if i select the datastore from inventory > datastores:
summary tab:  vmfs, 6 hosts, 5 vm's.  capacity 2.73tb, free space 1.04tb
vm's tab: lists the 5 vm's, their space, host cpu, mem, all powered on
hosts tab lists the 6 hosts, memory size, cpu, nic, uptime, etc
configuration tab, properties - lists the name, capacity, extend device, partitions, etc
alarms: none
permisisons:  administrators, role administrator, defined in the vcenter server
but back on 'getting started' for this datastore, clicking BROWSE DATASTORE = no error message, but just shows "/" in folders, no other folders or files

in there, if i click new folder, and hit ok, i get:
Call "FileManager.MakeDirectory" for object "FileManager" on vCenter Server "vs01.domain.com" failed.

-----------

yes, this is an iscsi san device - an hp MSA2324i with (23) 300gb sas drives in two vdisks

----------
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Are you multipathing to this datastore ?
0
FocISAuthor Commented:
yes, there are two 4 nic's on each host, two nic's on each of two controllers in the MSA itself
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
and setup correctly for multi path?

active i/o multi paths to the controllers ?

do you have any other datastores on this MSA with the same issue ?
0
FocISAuthor Commented:
well, i can't say for sure if multipathing is set up exactly correctly, it's a mix of people who've worked on things over the years

each host that can see this datastore, has an iscsi card with multiple nics, and is configured to talk to all 4 of the iscsi servers ip addresses - the "a" side nics are on one switch, the "b" side nics are on another switch

there is another datastore on this msa but i can freely browse that, edit, create, delete, no problem from vcenter server, from any host that's connected to it
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would look at updating

1. VMware vSphere vCenter Server
2. VMware vSphere ESXi 5.0

or

.....move the VMs, destroy the datastore, and re-create a new datastore and move the VMs back.

or

Contact VMware Support.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
FocISAuthor Commented:
I ended up moving the vm's off the datastore, deleting it from vsphere, deleting it on the iscsi san itself, and recreating it from scratch
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.