Solved

ESXi iSCSI attached store

Posted on 2013-01-19
23
606 Views
Last Modified: 2013-01-27
Hello all,
 I recently has an issue with a Windows Server 2008 Storage Server Standard Edition that forced me to do a Windows repair. In that process I lost some configuration. This Windows server is the data store for 2 ESXi (ESXi 4.1.0) servers. The interface between the 3 servers is iSCSI over ethernet.  After I reconfigured the interfaces back to the original configs everything was fine until I restarted on of the ESXi servers. When the ESXi server came back up it could not see the data store any longer. The second ESXi server is still communicating with the NAS and it still sees the file store it needs. I did not reboot the second ESXi server. Here are the details.

1. The interfaces on the ESXi and NAS are configured as follows.
   A. Iface 1
       * NAS, local lan subnet
       * ESXi, local LAN subnet (management network)

   B. Iface 2
      *NAS, iSCSi -1  10.10.10.1
      *ESXi, iSCSI on vSwitch 10.10.10.11

   C. Iface 3
       *NAS, iSCSI -2 10.10.20.1
       *ESXi, iSCSI on vSwitch 10.10.20.11
   

2. The ESXi server can ping the NAS on all 3 interfaces that are configured.
3. The NAS server can ping the ESXi server on all 3 interfaces that are configured.
4. No configuration changes have been made on the ESXi server.
5. The ESXi server can not auto discover the data store on the NAS.

How can I get this communication back between these two servers to bring this data store back online?
0
Comment
Question by:Brocklv6
  • 13
  • 9
23 Comments
 
LVL 117
ID: 38797148
does the ESXi server connect to the iSCSI server?
0
 

Author Comment

by:Brocklv6
ID: 38797155
The ESXi server connects to the NAS over 2 iSCSI over Ethernet interfaces.
0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 167 total points
ID: 38797163
Remove the store on esxi and readd it. Haven't found a better way
0
 
LVL 117
ID: 38797169
so, a single esxi server is missing a datastore?

when you rescan storage devices, is the lun displayed?
0
 

Author Comment

by:Brocklv6
ID: 38797187
This is what I see for devices.

ESXi LUN
0
 
LVL 117
ID: 38797191
under storage adapters
0
 

Author Comment

by:Brocklv6
ID: 38797196
iSCSI LUN
0
 

Author Comment

by:Brocklv6
ID: 38797199
Would it be something with the iSCSI Initiator on the Windows NAS?
0
 
LVL 117
ID: 38797203
check the properties of the iSCSI hba that it has the IP address of the iSCSI server, and if you are using chap settings check these are also correct.

rescan storage device

also check access is correctly allocated to this ESXi server iqn, permissions are correct on lun
0
 

Author Comment

by:Brocklv6
ID: 38797219
Another thing, t2here has been no changes made to the ESXi server. This has to be an issue with a config Windows NAS.
0
 
LVL 117
ID: 38797235
check iSCSI nas configs between servers, it would seem this server does not have access
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:Brocklv6
ID: 38797340
Seems that he Windows NAS is not listening on port 3260. But I cant find the service to start that uses that port.
0
 

Author Comment

by:Brocklv6
ID: 38797775
Ok, here is the latest update. I was able to re-install a iSCSI Target application on the NAS server, so now the ESXi host can see the storage on the NAS over iSCSI. For some reason I can not ad it as a datastore. Should the ESXi server see this storage now that auto discovery is working? Or do I need to actually add this as a datastore? Attached is the error I am getting. error
 Note: The VM data on this NAS drive is two virtual hard drive that contain all of the data for this ESXi host,  12 virtual machines. drive
0
 
LVL 117
ID: 38798173
that error above is a Windows sharing error.

so you can see the LUN now, but no datastore is visible?
0
 

Author Comment

by:Brocklv6
ID: 38798799
That is correct. When I start to add the datastore is when the error is thrown. I also do not want to format the data store, I want to still attach the existing VMs.
0
 
LVL 117
ID: 38799085
you will need to connect the Host ESX server using SSH, or run these commands at the console

type the following

esxcfg-volume -l

you should get output similar to the following:-

~ # esxcfg-volume -l
VMFS3 UUID/label: 4cc89503-f054d0d4-2865-00221958e471/nas3-vmfs
Can mount: Yes
Can resignature: No (the volume is being actively used)
Extent name: naa.600140588e9bab7d2b7ed4b53dadc1de:1     range: 0 - 1048319 (MB)

and then type

esxcfg-volume -m 4cc89503-f054d0d4-2865-00221958e471

or

esxcfg-volume -m nas3-vmfs

Refresh the displays on vSphere Client
0
 

Author Comment

by:Brocklv6
ID: 38799388
I attached the wrong error screen, this is the correct one. errorAlso, I changed the target connector to attach the partition instead of the physical drive. This is how it was configured before. Would this change the command you posted?3target
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 333 total points
ID: 38799416
iSCSI disk looks blank?

overwritten?

check using command line
0
 

Author Comment

by:Brocklv6
ID: 38799442
No it is not blank. DiskI do not have physical access to the ESXi host and I can not SSH into it because is is not enabled. How can I enable SSH with the vSphere client? Is it enable  VMkernel.Boot.techSupportMode?
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 333 total points
ID: 38799465
when you connected above it states the disk is blank?

there is no vmfs?

SSH and ESXi shell are enabled through the vSphere Client

see here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017910
0
 

Author Comment

by:Brocklv6
ID: 38799599
Yes it does show that the disk is blank but that is after it it errors. I know the disk is not blank. The 4 files are the only ones that have ever been present in the partition .vhd and .cbm
0
 

Assisted Solution

by:Brocklv6
Brocklv6 earned 0 total points
ID: 38806530
Solved.!!!
The ESXi server connects to the iSCSI targets on the NAS. After the recovery from the NAS crash the host could see but not mount the data stores. I fixed this by shutting down the iSCSI initiator in the host and renaming the files in /etc/vmware/vmkiscsid to .old. Reboot the host, and restarted the iSCSI initiator. It seemed that the .vhd files were corrupt for some reason. So thankfully I have a decent DR plan in place so I recovered my last backups of the .vhd files. I then rebuilt the iSCSI targets to mount the .vhd files instead of the NAS drive or partition. The host was then able to see and mount the data stores. All of the VMs came up and started successfully. I am having a small issue with the host, if I try to shut down or restart a VM the process will hang on 95%. It almost seems like the swap is full or there is an excessive memory usage. So I have to manually reboot the host to get out of the jam.
0
 

Author Closing Comment

by:Brocklv6
ID: 38823802
With the help form the people at experts-exchange, a VM specialist here locally, and some persistence and research on my part the problem was solved. Than you to all that helped.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Will try to explain how to use the VMware feature TAGs in the VMs and create Veeam Backup Jobs using TAGs. Since this article is too long, I will create second article for the Veeam tasks.
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now