• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 641
  • Last Modified:

Datastore disappeared from ESXi host but VMs still running

Hi all

Got a very strange one here.

A client has a standalone ESXi 5.5 host, with a local datastore which started as being on one virtual disk, then being extended onto a second virtual disk.  Both of these VDs are in RAID 5, not that I think it matters.

Anyway all has been well for well over a year.  Today, an external NFS datastore was created on a SAN, and the required storage adapters added to the host so that it could see the NFS datastore.  This mounted, and all continued to be well.

However, the local datastore then suddenly disappeared.  Errors on the host at the time said (earliest first):

No datastores have been configured on the host
Device or filesystem with identifier naa.(long number) has entered the All Paths Down state
Device naa.(long number) has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.
Permanently inaccessible device naa.(long number) has no more opens. It is now safe to unmount datastores (if any)Unknown and delete the device.

In the inventory, I can see the three running VMs at the top, but below them, I see

Unknown (inaccessible)
Unknown 1 (inaccessible)
Unknown 2 (inaccessible)
Unknown 3 (inaccessible)

Bear in mind that the three running VMs are quite happy and working fine.

I have tried connecting to it using vcenter to do a storage vmotion to the newly connected NFS datastore but that moans about a reference to the datastore being present but there being no datastore.  Clearly the datastore is there though as otherwise these VMs wouldn't be running.

I don't want to do anything drastic to the host like reboot it as if it can't see the datastore, I am certain these VMs would not come up again.

Could anyone point me in the right direction to get this datastore back and recognised by the host please?  Reminder - it's a datastore that has been extended over two virtual disks so not sure that needs special treatment to re-recognise it.

Thanks in advance,

Chris
0
chris3879
Asked:
chris3879
  • 4
  • 3
1 Solution
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
First thing, is to restart the host, and check there are no errors on the RAID set created for datastores.

also is the hardware certified for ESXi and on the HCL ?

also update firmware and drivers for storage controller.
0
 
chris3879Author Commented:
Thanks but if I restart the host and it can't see the data store when it reboots, these VMs won't come back online again. It's a production environment and that would cause a nightmare. I can't even back them up using veeam so there is real risk of data loss.

Re: raid array, looking at it, one of the disks in one of the raid 5 arrays says it has failed. But the virtual disk is still presenting itself fine to esxi- I can see it in the storage devices list and all paths to it are ok.

Also yes the hardware is on the HCL - it's a Dell R620 with PERC.

Any other suggestions please?

Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The fact that a disk has failed in the RAID set, has caused this issue.

You should already have backups, that you can restore in the event of data loss, also RAID 5 is no longer suitable for an enterprise environment because it can only support a single failed disk.

If the VMs are currently running, you could use VMware Converter, to obtain a current backup.

Sooner or later, you will have to reboot the host to troubleshoot, extending datastores over two seperate LUNs, is dangerous. (and not best practice).
1
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
chris3879Author Commented:
Thanks very much Andrew.  I'll give VMware Converter a go to get an up to date backup and then reboot the host to troubleshoot.

Will post back here when I have more updates.

Thanks again
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
if you do lose the entire LUN (VD), this would be an ideal opportunity to change to RAID 6, and a single VD (or LUN)
0
 
chris3879Author Commented:
Just wanted to say thanks very much Andrew, the vcenter converter let me convert the powered on VM to another on the same host but on the new datastore.  Just as well as as soon as I powered off the original VMs, they became invisible and inaccessible to the host.

Next step will be to rebuild the RAID as RAID 6 as you suggest, to prevent a similar recurrence in future.

Thanks again for your help.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
no problems, VMware Converter, is a handy tool to have around.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now