?
Solved

ESXi partition in need of recovery

Posted on 2016-07-29
29
Medium Priority
?
40 Views
Last Modified: 2016-08-03
I recently had an issue where I lost some network connectivity with my iSCSI NAS where my ESXi datastores are.  After reconnecting ESXi shows it mounts the target however the datastore does not appear.  When I go to add new storage it presents the LUN and shows there is a partition there (see attached pic) however I can't seem to figure out how to use that partition without wiping it out.  I am on ESXi 6.  Any ideas?
0
Comment
Question by:huntson
  • 16
  • 12
29 Comments
 
LVL 125
ID: 41734676
you only use the ADD Storage to ADD NEW Storage, this is not new storage!

Okay, can you get to the console, or remote via SSH ?
0
 

Author Comment

by:huntson
ID: 41734694
I just did it to verify that it can see the partition. Yes I can.
0
 
LVL 125

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 41734699
okay, I need you to type the following...

esxcfg-volume -l 

Open in new window


at the bash prompt.

this should show something like below

~ # esxcfg-volume -l
VMFS3 UUID/label: 4cc89503-f054d0d4-2865-00221958e471/nas3-vmfs
Can mount: Yes
Can resignature: No (the volume is being actively used)
Extent name: naa.600140588e9bab7d2b7ed4b53dadc1de:1     range: 0 - 1048319 (MB)

and then do the following

esxcfg-volume -M 4cc89503-f054d0d4-2865-00221958e471

Open in new window


or

esxcfg-volume -M nas3-vmfs

Open in new window


(obvsiously your GUID or friendly datastore name)
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:huntson
ID: 41734703
That first command just gives me a carriage return.  I am at the SSH prompt if that makes a snip-5.PNGdifference.
0
 
LVL 125
ID: 41734710
that's very unusual, but if you click Add Storage it finds the LUNs ?

and wants to overwrite them ?

can you check the LUNs, are present, but datastores are not ?

have you tried re-scanning the storage adaptor ?

failing this, you will have to check the logs /var/logs/vmkernel.log

check why they are not being mounted
1
 

Author Comment

by:huntson
ID: 41734735
What's the best way to access those logs as that's the next step?
0
 
LVL 12

Expert Comment

by:Mr Tortur
ID: 41734758
Hi,
view them at console or ssh session
or
get them using Winscp
0
 

Author Comment

by:huntson
ID: 41734759
Here we go.  I know the device I'm looking for is naa.6001405136dec50da155d4eb5d9574df

Any clue?
vmkernel.log
0
 
LVL 125
ID: 41734787
can you confirm, if the LUNS exists ?
0
 

Author Comment

by:huntson
ID: 41734797
It does indeed.
snip-6.PNG
0
 
LVL 125
ID: 41734870
whats the dead or error ?
0
 

Author Comment

by:huntson
ID: 41734880
I rebooted and that disappeared.  It was a previously mounted volume that no longer exists.  Any idea what to do next?  I don't have any support contract or anything with vmware - is there a way I can get a per-incident case?  Anywhere else I can turn?
0
 
LVL 125
ID: 41734912
see my other post about weirdness with Synology NAS and volumes and LUNs.

hey stay tuned, we will try and get this resolved.

I'm here for you man!
0
 

Author Comment

by:huntson
ID: 41734961
Can yo post a link please?

Thanks for your help!!
0
 

Author Comment

by:huntson
ID: 41734995
I see it's on my other question.  Thanks for checking into it.
0
 

Author Comment

by:huntson
ID: 41735009
I see although that's not necessarily the same here.  I was not close to running out of space before and certainly am not now (even though I understand I provisioned the entirety of the volume).  

You do bring up a good point in your discussion with that other user - is there any way to access the volume via iSCSI from Windows?  Off the top of my head I'd say that Windows/Mac doesn't understand VMFS.
0
 
LVL 125
ID: 41735040
Windows/Mac cannot read and write VMFS (well, unless you use a third party driver!) which is read only.

Okay, I can reproduced the issue here, using Synlogy NAS.

I created a 100GB iSCSI Thin LUN on a 200GB free Volume.

Presented to VMware ESXi, added a datastore, added a virtual machine.

Increase the size of the LUN on Synology 10GB at a time, increase space on datastore. all worked correctly.

As soon, as I increased the size of the LUN, so the volume space was 0.

datastore and LUN, on ESXi became unavailable.

I think you need to refer to Synology support.
0
 

Author Comment

by:huntson
ID: 41735045
Synology support?  They're a joke.

Thank you though for investigating.
0
 

Author Comment

by:huntson
ID: 41735046
This read-only concept sounds good though.  At least I can get all my machines off the volume assuming I am able to get it to mount.
0
 
LVL 125
ID: 41735090
We only use NFS for Backups.
0
 

Author Comment

by:huntson
ID: 41735188
What version of DSM are you using?  I'm going to call Synology and say I and someone else with x version had this issue.
0
 

Accepted Solution

by:
huntson earned 0 total points
ID: 41735190
FYI I'm on the phone with VMware (I purchased one of their per-indicdent support packages) and they have to do an experimental procedure to try to get my data back.  They say the 90meg VMFS header is damaged.  They want to try and copy the header from another volume with the same size onto this one and see what happens.
0
 
LVL 125
ID: 41735205
that would be why it's not being mounted and recognised.

DSM 5.x the latest edition , we've not upgraded to 6.x yet.

No backups ?
0
 

Author Comment

by:huntson
ID: 41735213
It was in the works to get some backups of our machines.  That'll be another question I post - best backup solution for budget vmware users.
0
 
LVL 125
ID: 41735215
Oh very dear......
0
 

Author Comment

by:huntson
ID: 41735255
They did it.  You should have seen the stuff they did.  Quite impressive and for a measly $300 or so.  Thanks for all your help here.

Since the solution was found externally what's the best way to mark a solution here?  I want to ensure you get points.
0
 
LVL 125
ID: 41735265
I would be more concerned as to why it got corrupted!

select your own Answer.

https:#a41735190
0
 

Author Closing Comment

by:huntson
ID: 41740324
I wound up calling support as this was quite a complicated corruption issue.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
In this article will go through how to backup a vPostgres DB from a broken vCenter Appliance and restore to a new vCenter Appliance.
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
Suggested Courses

621 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question