Solved

Windows Storage Server 2003 R2 iSCSI target losing connection

Posted on 2011-02-14
13
2,633 Views
Last Modified: 2012-05-11
We have a Windows 2003 R2 Storage Server hosting shares in our network. It was also configured as an iSCSI target for a VMware ESXi 4.0.0 server and several VMs were stored on it.

A couple of weeks ago the VMs became unavailable and we haven't been able to get them running since. As soon as I try to do anything with them (start the VMs, mount the drives in a different VM, download the files, etc) the vSphere Client crashes and it loses its connection to the iSCSI target. I have to go to the SAN and restart the "Microsoft iSCSI Software Target" service, then rescan the Storage Adapters in VMware.

Once I do that it reconnects to the storage and I can see the files stored in it. I'm able to Browse the datastore, see the VMs powered down, and even add the virtual drives to other VMs, but as soon as I try to actually do anything with them it crashes again.

Restarting the ESXi host does not help, only restarting the iSCSI service on the SAN so I'm thinking that the problem is with Microsoft, but aren't really sure.

We've added enough local storage to the ESXi host to move the VMs there, so all I want to do is access the data and get it off the SAN. I don't need to get it working permanently, just long enough to pull the virtual disks to different storage.

I've tried mounting the .VHD that Windows SS created in a windows 2008 server, but it says that the drive has to be NTFS to mount. I've also tried using the Microsoft iSCSI Software Target MMC to mount the drive, but can't give it a drive letter and see the data. I'm guessing it's because the drive was formatted by VMWare, but again I'm taking a guess.

Does anyone have any thoughts on how to get access to these files long enough to move them to better storage?
0
Comment
Question by:sfcanderson
  • 8
  • 5
13 Comments
 
LVL 118
ID: 34892692
Microsoft Windows Storage Server 2003 R2 iSCSI is not on the VMware Hardware Compatibility list for VMware ESXi 4.0.0 Server. (so you run the risk of issues, anyway, that out of the way).

Is the Microsoft iSCSI Software Target service crashing on the Storage Server?

0
 
LVL 118
ID: 34892700
Okay, ignoring the vSphere Client have you tried, using the ESX console, and re-scanning for the volumes at the command line, and then using cp command to copy the VMs off the iSCSI LUNs to local datastore.
0
 
LVL 118
ID: 34892712
e.g. at console esxcfg-rescan vmhba33 (normally iSCSI software hba)
0
 
LVL 118
ID: 34892731
I think what we need to try and work out, is what is causing the service to fail, is it lack of resources on the Windows server, so you may have to shutdown, and stop all shares or activity etc

0
 

Author Comment

by:sfcanderson
ID: 34892735
Ugh, I didn't even think to check the VMware HCL! This was set up by a previous IT contact, so I'm trying to clean things up.

The Software Target service on the Storage Server is not crashing, It stays running. The only way I've found to regain access to the data, though, is to restart that service.
0
 
LVL 118
ID: 34892812
okay, that's good.

forget the vsphere client.

login to console of esx server, rescan for uba, and then check the vmfs luns, and manually copy to local data store, and see if it stays up long enough for copy.

let me know, got a few more ideas, see how that goes.
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 

Author Comment

by:sfcanderson
ID: 34893106
This is ESXi, so I don't think I can access the command line from the console. I'll try the Remote CLI a little later, but every time I've tried to copy the data through the vSphere Client things lock up.

This is also the main storage server for the network, so people are accessing it's shares all day. I've cleared it with management to reboot the server at 8:00 tonight (CST) and spend a couple of hours troubleshooting it. I'll provide more info when I get a chance to do those.
0
 
LVL 118
ID: 34893186
ESXi - yes you can get to the command line.

The fastest way to copy data off an iSCSI datastore, is at the command line (console).

from datastore to local datastore. (provided your iSCSI store stays up long enough), but we will see if this is the issue, if there is a timeout when the server tries to copy data off the iSCSI datastore to local datastore (local disk), we will have to think how we can beef-up fast access.
0
 

Author Comment

by:sfcanderson
ID: 34893473
OK, I'd thought this server had an ILO card, but I guess it doesn't so I can't get on the console. I've installed the remote CLI and am trying figure out how to run the commands through there. I'll let you know how it goes in a bit.

thanks for the help so far.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 34893593
can you ssh to the server?

what you could do is, use the vpshere client to connect to the server, create a user account that has shell access.

use that user account to access the shell using ssh, and whilst logged in with that user, then su to root.

so no need for ilo.

whilst then logged, esxcfg-rescan vmbha33

check iSCSI datastores, and then copy vms and folders to local disk (datastores)
0
 

Author Comment

by:sfcanderson
ID: 34893854
OK, so that worked, but now I have a new problem :(
It said it copied all of the files successfully, but one out of the three VMs is missing two files: the .VMX and .VSWP. The other two VMs had all of their files successfully and I was able to start them up.

The files are no longer in the old location and didn't show up in the local datastore either. My guess is some kind of corruption from all the iSCSI problems?

The .VMDK is there, so is there any way to get the VM rebuilt?
0
 

Author Closing Comment

by:sfcanderson
ID: 34893942
Thanks for all the help!
I created a new VM and just attached it to the .VMDK and now all three machines are on the local storage and running normally. Thanks again!
0
 
LVL 118
ID: 34895001
Glad you got it sorted. You may want to test it carefully! (iSCSI Storage).

NFS works well on Storage Server for ESX.
0

Featured Post

Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

919 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now