Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Cannot browse datastore in vsphere 5.0

Posted on 2014-09-10
21
Medium Priority
?
2,818 Views
Last Modified: 2014-09-19
We have a situation where we cannot browse the contents of a specific datastore, but multiple VM's are running from that datastore in good health.

I have troubleshot this as far as i can, i would really appreciate some expertise here, it feels like a super easy fix, but i can't find it.

environment:
vsphere server at 172.16.1.120, version 5.0.0
esxi physical server at 172.16.1.245, version 5.0
"broken" datastore is on an iSCSI target at 172.16.1.226

if i log into vsphere client directed at the vcenter manager 172.16.1.120, logging in with active directory (a domain admin) - i can see all of our physical esxi hosts, all of our VM's and everything is fine - except this one datastore.

on one esxi host, one VM isn't powered on.  Went to power it on, errored out as insufficient permission.

Weird, i though, so i went to browse the datastore it lives on (as seen from its esxi host) - the datastore appears blank (blank as in all i see is "\" and no folders or files)

in a panic, i check the health of the other vm's on this datastore - they're running just fine (but i can't browse the datastore or see their vmdk files)

so i try to make a new folder on the "blank" datastore, and receive:
"Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “servername.company.com” failed.

Google searched that, found 5-10 articles mentioning various things, but so far all i can find for sure is that if i use vsphere client to log DIRECTLY into the esx host itself as root (172.16.1.245), i can browse the same datastore just fine, see files, make/delete things, no problem

So the issue has to center around vsphere server - the problem is only present when logging in there

I've read that sometimes this happens and if you log into vcenter as root, you have proper permissions... but nobody knows the root password at this time (it isn't root, vmware, blank, or any of our prior passwords we like to use)

So, how can i make this datastore usable from vcenter again?
0
Comment
Question by:FocIS
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 11
  • 9
21 Comments
 
LVL 123
ID: 40315054
vCenter just displays information from the Host ESXi server.

The Host is attached to the datastore.

Can you connect directly to the host using vSphere Client and root password, to check you can view the datastore correctly.

If you have access ESXi host via SSH, and can list the contents, I would suspect the datastore is okay, it's just an issue with vCenter Server, inventory service.
0
 
LVL 2

Author Comment

by:FocIS
ID: 40315078
i can use the vsphere client and root password to connect directly to the esxi host, and the datastore appears and functions correctly

i also suspect it's an issue with vcenter server - what is this inventory service, what can i check?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40315081
Oh the system service itself - i see it is running as localsystem, and is started in good health

i've rebooted the vcenter physical server (172.16.1.120) and the issue persists
0
 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks

 
LVL 123
ID: 40315189
There is no issue with the ESXi host connection to the datastore.

vCenter Server is just a management server, with the Hosts added.

What versions/builds of vCenter Server and ESXi are you using?

Are you running the latest and current versions of both?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40315210
latest and current no...

vcenter:  5.0.0 build 455964
esxi: 5.0.0-469512-standard

this enterprise has been working perfectly fine for 3+ years without issue, the datastore issue just became apparent about 2 days ago, but i can't pinpoint anything specific that changed
0
 
LVL 123
ID: 40315288
Yes, those builds are very old. There are original GA Builds from August 2011.

Latest is 1918656. (July 2014)

You do not have any special characters in foldernames ?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40315306
we were always careful to not include spaces or special characters

back in the day i was reading how to upgrade from 5.0.x to 5.1 - and it was incredibly complicated or at least read that way...  i think the upgrade to 5.5 isn't as complicated though...

but on the other hand i wouldn't attempt an upgrade knowing that vcenter can't reliably see one datastore - with my luck it would just delete everything on it or something

what suggestions to make the contents of this datastore visible again as seen from vcenter (instead of just from the esx host)
0
 
LVL 123
ID: 40315368
its's either a bug, characters causing the issue, or permissions (which are different) when connecting via vCenter Server.

Do all users have the same issues from ALL workstations?

if you are on the vCenter Server, and connect back to the server via vSphere Client any different ?

Upgrading to 5.1 or 5.5 is the same! and can have complications because of new SSO, which is new to 5.1/5.5.

Updating a bug fix is different, and easier.

Are ALL the datastores on the same SAN ? (IP Address)
0
 
LVL 2

Author Comment

by:FocIS
ID: 40315408
we only access vcenter thru the actual vcenter server (remote desktop into 172.16.1.120 and then run the client to connect to 172.16.1.120), this is where we can't see the contents of that datastore.

we have multiple other datastores:
- some on local hard drives of the esx host itself, accessed only by the respective esx host
- two other san's (three total san's, different physical MSA cabinets with unique ip addresses, all iSCSI)
- one NAS server used for archival storage, connected to some esx hosts

at least 3 esx hosts have connections to the datastore in question, this datastore is one of two on the iscsi msa cabinet.  if i try to browse the datastore from either of the other hosts (still inside vcenter) the result is the same, it looks blank.  

if i log into vcenter as domain\administrator or domain\my-username (both domain admins) the result is the same.  i haven't tried to log in as root since nobody knows the root password anymore

if i log into the vcenter by way of vmware web client, as a domain admin, i can navigate to where the datastore is, and see the performance info, hosts attached to it, vm's on it... but there's no link to 'browse datastore' from the web client
0
 
LVL 12

Expert Comment

by:Mr Tortur
ID: 40317461
Hi,
I would check the permission side if I were you, knowing that you had an "insufficient permission" message starting a VM beside your browsing datastore issue.

Just check, when you log in vCenter vsphere client using domain admin account, selecting the vCenter server at the top of the inventory on the left side, and then going to "permission" tab on the right side, that you see your domain user or group with an administrator role in Vmware.
I don't really think your issue is only a permission one but I would check that to be sure.

Also I understand that your vCenter is a Windows machine (and not the vCenter Linux appliance from Vmware, which is the other possibility) so you do not have any root account on this server, but a local windows administrator account and domain accounts.

As Andrew Hancock said I would also check that this error is not coming of a particular vSphere client : I suggest to try to install/run the vSphere client on another workstation on your Lan and to test loggin in vCenter from this workstation.

And as Andrew Hancock too I would recommend to patch your ESXi servers (very simple using update manager for example). But this is a little more impacting, and I understand your fear. Don't be angry, I did not say you are a coward  ; )

You can also upgrade your vCenter version to a nearly but more up-to-date version, but this is again more and more intrusive for your infrastructure.
0
 
LVL 2

Author Comment

by:FocIS
ID: 40317483
In the permission tab, everything appears to be correct - and my account doesn't seem to have any other restriction anywhere else

I was able to start the VM by logging into the esx server directly using the vsphere client, and browse the data store

I will install the vsphere client on another unrelated workstation and see how it looks from there
0
 
LVL 123
ID: 40317644
to  narrow down....

which datastore on which host is causing the issue?

and is it the same datastore on ALL hosts?

how are you selecting and looking at the datastore ? e.g. do you go to the summary of each host in vCenter ?

So you cannot start the VM from vCenter Server?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40317685
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server
0
 
LVL 123
ID: 40317767
right, i couldn't start the VM from vcenter server, i had to start it from vcenter client attached directly to the esxi host

any error message ?

do the VMs appear, or do they state unknown, inaccessible?

the datastore we call msa3-vol1 is the one we can't browse or control from vcenter server, weather or not we try from host1, or host2, or host3 'summary' tab

okay, interesting, if you select the datastore, from Inventory, Datastores and Datastores Cluster Tab can you browse?

What is this iSCSI SAN ?

the datastore is listed in the summary tab of 3 hosts, and all three hosts if you right-click on the datastore, and say 'browse datastore', it comes up with an empty "/" root folder with no ability to create a new folder/file - even though the same exact thing as seen from vcenter client attached directly to any of the esxi hosts (who have this datastore) can see and manipulate the files that are there

viewing through vCenter Server is a different process.

the datastore is not attached to all hosts (15 something hosts) but 3 of them - there are 3 VM's that live on this datastore, they are currently running just fine with users logged into them and everything.  i suppose if they reboot, we won't be able to start them from vcenter server

why is the datastore not presented to all hosts?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40317833
i don't know how to reply quoted, so i'll reply in points

of the three VM's on this datastore (each with its own host), only one was off.  in vcenter server it appeared as a normal powered-off VM.  i could right-click edit it, change settings, view the console (black screen since it was off), edit memory/cpu, etc.  powering it on is what led to the permissions issue being stated.  it didn't specifically mention the datastore, but troubleshooting led us to check there.

-----------

this datastore isn't presented to all hosts mainly because we don't vmotion, and the storage is pretty much provisioned out for other things, other hosts probably wouldn't be able to find enough useful storage space on it, they use other MSA devices

if i select the datastore from inventory > datastores:
summary tab:  vmfs, 6 hosts, 5 vm's.  capacity 2.73tb, free space 1.04tb
vm's tab: lists the 5 vm's, their space, host cpu, mem, all powered on
hosts tab lists the 6 hosts, memory size, cpu, nic, uptime, etc
configuration tab, properties - lists the name, capacity, extend device, partitions, etc
alarms: none
permisisons:  administrators, role administrator, defined in the vcenter server
but back on 'getting started' for this datastore, clicking BROWSE DATASTORE = no error message, but just shows "/" in folders, no other folders or files

in there, if i click new folder, and hit ok, i get:
Call "FileManager.MakeDirectory" for object "FileManager" on vCenter Server "vs01.domain.com" failed.

-----------

yes, this is an iscsi san device - an hp MSA2324i with (23) 300gb sas drives in two vdisks

----------
0
 
LVL 123
ID: 40317841
Are you multipathing to this datastore ?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40317848
yes, there are two 4 nic's on each host, two nic's on each of two controllers in the MSA itself
0
 
LVL 123
ID: 40317865
and setup correctly for multi path?

active i/o multi paths to the controllers ?

do you have any other datastores on this MSA with the same issue ?
0
 
LVL 2

Author Comment

by:FocIS
ID: 40317918
well, i can't say for sure if multipathing is set up exactly correctly, it's a mix of people who've worked on things over the years

each host that can see this datastore, has an iscsi card with multiple nics, and is configured to talk to all 4 of the iscsi servers ip addresses - the "a" side nics are on one switch, the "b" side nics are on another switch

there is another datastore on this msa but i can freely browse that, edit, create, delete, no problem from vcenter server, from any host that's connected to it
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 40317939
I would look at updating

1. VMware vSphere vCenter Server
2. VMware vSphere ESXi 5.0

or

.....move the VMs, destroy the datastore, and re-create a new datastore and move the VMs back.

or

Contact VMware Support.
0
 
LVL 2

Author Closing Comment

by:FocIS
ID: 40333122
I ended up moving the vm's off the datastore, deleting it from vsphere, deleting it on the iscsi san itself, and recreating it from scratch
0

Featured Post

Simple, centralized multimedia control

Watch and learn to see how ATEN provided an easy and effective way for three jointly-owned pubs to control the 60 televisions located across their three venues utilizing the ATEN Control System, Modular Matrix Switch and HDBaseT extenders.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
Teach the user how to delpoy the vCenter Server Appliance and how to configure its network settings Deploy OVF: Open VM console and configure networking:
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…

664 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question