Solved

Datastore disappeared from ESXi host but VMs still running

Posted on 2016-09-16
7
105 Views
Last Modified: 2016-09-19
Hi all

Got a very strange one here.

A client has a standalone ESXi 5.5 host, with a local datastore which started as being on one virtual disk, then being extended onto a second virtual disk.  Both of these VDs are in RAID 5, not that I think it matters.

Anyway all has been well for well over a year.  Today, an external NFS datastore was created on a SAN, and the required storage adapters added to the host so that it could see the NFS datastore.  This mounted, and all continued to be well.

However, the local datastore then suddenly disappeared.  Errors on the host at the time said (earliest first):

No datastores have been configured on the host
Device or filesystem with identifier naa.(long number) has entered the All Paths Down state
Device naa.(long number) has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.
Permanently inaccessible device naa.(long number) has no more opens. It is now safe to unmount datastores (if any)Unknown and delete the device.

In the inventory, I can see the three running VMs at the top, but below them, I see

Unknown (inaccessible)
Unknown 1 (inaccessible)
Unknown 2 (inaccessible)
Unknown 3 (inaccessible)

Bear in mind that the three running VMs are quite happy and working fine.

I have tried connecting to it using vcenter to do a storage vmotion to the newly connected NFS datastore but that moans about a reference to the datastore being present but there being no datastore.  Clearly the datastore is there though as otherwise these VMs wouldn't be running.

I don't want to do anything drastic to the host like reboot it as if it can't see the datastore, I am certain these VMs would not come up again.

Could anyone point me in the right direction to get this datastore back and recognised by the host please?  Reminder - it's a datastore that has been extended over two virtual disks so not sure that needs special treatment to re-recognise it.

Thanks in advance,

Chris
0
Comment
Question by:chris3879
  • 4
  • 3
7 Comments
 
LVL 117
ID: 41802119
First thing, is to restart the host, and check there are no errors on the RAID set created for datastores.

also is the hardware certified for ESXi and on the HCL ?

also update firmware and drivers for storage controller.
0
 

Author Comment

by:chris3879
ID: 41802672
Thanks but if I restart the host and it can't see the data store when it reboots, these VMs won't come back online again. It's a production environment and that would cause a nightmare. I can't even back them up using veeam so there is real risk of data loss.

Re: raid array, looking at it, one of the disks in one of the raid 5 arrays says it has failed. But the virtual disk is still presenting itself fine to esxi- I can see it in the storage devices list and all paths to it are ok.

Also yes the hardware is on the HCL - it's a Dell R620 with PERC.

Any other suggestions please?

Thanks
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 41802702
The fact that a disk has failed in the RAID set, has caused this issue.

You should already have backups, that you can restore in the event of data loss, also RAID 5 is no longer suitable for an enterprise environment because it can only support a single failed disk.

If the VMs are currently running, you could use VMware Converter, to obtain a current backup.

Sooner or later, you will have to reboot the host to troubleshoot, extending datastores over two seperate LUNs, is dangerous. (and not best practice).
1
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 

Author Comment

by:chris3879
ID: 41802705
Thanks very much Andrew.  I'll give VMware Converter a go to get an up to date backup and then reboot the host to troubleshoot.

Will post back here when I have more updates.

Thanks again
0
 
LVL 117
ID: 41802711
if you do lose the entire LUN (VD), this would be an ideal opportunity to change to RAID 6, and a single VD (or LUN)
0
 

Author Comment

by:chris3879
ID: 41804576
Just wanted to say thanks very much Andrew, the vcenter converter let me convert the powered on VM to another on the same host but on the new datastore.  Just as well as as soon as I powered off the original VMs, they became invisible and inaccessible to the host.

Next step will be to rebuild the RAID as RAID 6 as you suggest, to prevent a similar recurrence in future.

Thanks again for your help.
0
 
LVL 117
ID: 41804670
no problems, VMware Converter, is a handy tool to have around.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

David Varnum recently wrote up his impressions of PRTG, based on a presentation by my colleague Christian at Tech Field Day at VMworld in Barcelona. Thanks David, for your detailed and honest evaluation!
Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now