Solved

Datastore disappeared from ESXi host but VMs still running

Posted on 2016-09-16
7
135 Views
Last Modified: 2016-09-19
Hi all

Got a very strange one here.

A client has a standalone ESXi 5.5 host, with a local datastore which started as being on one virtual disk, then being extended onto a second virtual disk.  Both of these VDs are in RAID 5, not that I think it matters.

Anyway all has been well for well over a year.  Today, an external NFS datastore was created on a SAN, and the required storage adapters added to the host so that it could see the NFS datastore.  This mounted, and all continued to be well.

However, the local datastore then suddenly disappeared.  Errors on the host at the time said (earliest first):

No datastores have been configured on the host
Device or filesystem with identifier naa.(long number) has entered the All Paths Down state
Device naa.(long number) has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.
Permanently inaccessible device naa.(long number) has no more opens. It is now safe to unmount datastores (if any)Unknown and delete the device.

In the inventory, I can see the three running VMs at the top, but below them, I see

Unknown (inaccessible)
Unknown 1 (inaccessible)
Unknown 2 (inaccessible)
Unknown 3 (inaccessible)

Bear in mind that the three running VMs are quite happy and working fine.

I have tried connecting to it using vcenter to do a storage vmotion to the newly connected NFS datastore but that moans about a reference to the datastore being present but there being no datastore.  Clearly the datastore is there though as otherwise these VMs wouldn't be running.

I don't want to do anything drastic to the host like reboot it as if it can't see the datastore, I am certain these VMs would not come up again.

Could anyone point me in the right direction to get this datastore back and recognised by the host please?  Reminder - it's a datastore that has been extended over two virtual disks so not sure that needs special treatment to re-recognise it.

Thanks in advance,

Chris
0
Comment
Question by:chris3879
  • 4
  • 3
7 Comments
 
LVL 118
ID: 41802119
First thing, is to restart the host, and check there are no errors on the RAID set created for datastores.

also is the hardware certified for ESXi and on the HCL ?

also update firmware and drivers for storage controller.
0
 

Author Comment

by:chris3879
ID: 41802672
Thanks but if I restart the host and it can't see the data store when it reboots, these VMs won't come back online again. It's a production environment and that would cause a nightmare. I can't even back them up using veeam so there is real risk of data loss.

Re: raid array, looking at it, one of the disks in one of the raid 5 arrays says it has failed. But the virtual disk is still presenting itself fine to esxi- I can see it in the storage devices list and all paths to it are ok.

Also yes the hardware is on the HCL - it's a Dell R620 with PERC.

Any other suggestions please?

Thanks
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 41802702
The fact that a disk has failed in the RAID set, has caused this issue.

You should already have backups, that you can restore in the event of data loss, also RAID 5 is no longer suitable for an enterprise environment because it can only support a single failed disk.

If the VMs are currently running, you could use VMware Converter, to obtain a current backup.

Sooner or later, you will have to reboot the host to troubleshoot, extending datastores over two seperate LUNs, is dangerous. (and not best practice).
1
Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

 

Author Comment

by:chris3879
ID: 41802705
Thanks very much Andrew.  I'll give VMware Converter a go to get an up to date backup and then reboot the host to troubleshoot.

Will post back here when I have more updates.

Thanks again
0
 
LVL 118
ID: 41802711
if you do lose the entire LUN (VD), this would be an ideal opportunity to change to RAID 6, and a single VD (or LUN)
0
 

Author Comment

by:chris3879
ID: 41804576
Just wanted to say thanks very much Andrew, the vcenter converter let me convert the powered on VM to another on the same host but on the new datastore.  Just as well as as soon as I powered off the original VMs, they became invisible and inaccessible to the host.

Next step will be to rebuild the RAID as RAID 6 as you suggest, to prevent a similar recurrence in future.

Thanks again for your help.
0
 
LVL 118
ID: 41804670
no problems, VMware Converter, is a handy tool to have around.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Is your company's data protection keeping pace with virtualization? Here are 7 dynamic ways to adapt to rapid breakthroughs in technology.
In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
In this video tutorial I show you the main steps to install and configure  a VMware ESXi6.0 server. The video has my comments as text on the screen and you can pause anytime when needed. Hope this will be helpful. Verify that your hardware and BIO…

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now