Link to home
Start Free TrialLog in
Avatar of sfcanderson
sfcandersonFlag for United States of America

asked on

Windows Storage Server 2003 R2 iSCSI target losing connection

We have a Windows 2003 R2 Storage Server hosting shares in our network. It was also configured as an iSCSI target for a VMware ESXi 4.0.0 server and several VMs were stored on it.

A couple of weeks ago the VMs became unavailable and we haven't been able to get them running since. As soon as I try to do anything with them (start the VMs, mount the drives in a different VM, download the files, etc) the vSphere Client crashes and it loses its connection to the iSCSI target. I have to go to the SAN and restart the "Microsoft iSCSI Software Target" service, then rescan the Storage Adapters in VMware.

Once I do that it reconnects to the storage and I can see the files stored in it. I'm able to Browse the datastore, see the VMs powered down, and even add the virtual drives to other VMs, but as soon as I try to actually do anything with them it crashes again.

Restarting the ESXi host does not help, only restarting the iSCSI service on the SAN so I'm thinking that the problem is with Microsoft, but aren't really sure.

We've added enough local storage to the ESXi host to move the VMs there, so all I want to do is access the data and get it off the SAN. I don't need to get it working permanently, just long enough to pull the virtual disks to different storage.

I've tried mounting the .VHD that Windows SS created in a windows 2008 server, but it says that the drive has to be NTFS to mount. I've also tried using the Microsoft iSCSI Software Target MMC to mount the drive, but can't give it a drive letter and see the data. I'm guessing it's because the drive was formatted by VMWare, but again I'm taking a guess.

Does anyone have any thoughts on how to get access to these files long enough to move them to better storage?
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Microsoft Windows Storage Server 2003 R2 iSCSI is not on the VMware Hardware Compatibility list for VMware ESXi 4.0.0 Server. (so you run the risk of issues, anyway, that out of the way).

Is the Microsoft iSCSI Software Target service crashing on the Storage Server?

Okay, ignoring the vSphere Client have you tried, using the ESX console, and re-scanning for the volumes at the command line, and then using cp command to copy the VMs off the iSCSI LUNs to local datastore.
e.g. at console esxcfg-rescan vmhba33 (normally iSCSI software hba)
I think what we need to try and work out, is what is causing the service to fail, is it lack of resources on the Windows server, so you may have to shutdown, and stop all shares or activity etc

Avatar of sfcanderson

ASKER

Ugh, I didn't even think to check the VMware HCL! This was set up by a previous IT contact, so I'm trying to clean things up.

The Software Target service on the Storage Server is not crashing, It stays running. The only way I've found to regain access to the data, though, is to restart that service.
okay, that's good.

forget the vsphere client.

login to console of esx server, rescan for uba, and then check the vmfs luns, and manually copy to local data store, and see if it stays up long enough for copy.

let me know, got a few more ideas, see how that goes.
This is ESXi, so I don't think I can access the command line from the console. I'll try the Remote CLI a little later, but every time I've tried to copy the data through the vSphere Client things lock up.

This is also the main storage server for the network, so people are accessing it's shares all day. I've cleared it with management to reboot the server at 8:00 tonight (CST) and spend a couple of hours troubleshooting it. I'll provide more info when I get a chance to do those.
ESXi - yes you can get to the command line.

The fastest way to copy data off an iSCSI datastore, is at the command line (console).

from datastore to local datastore. (provided your iSCSI store stays up long enough), but we will see if this is the issue, if there is a timeout when the server tries to copy data off the iSCSI datastore to local datastore (local disk), we will have to think how we can beef-up fast access.
OK, I'd thought this server had an ILO card, but I guess it doesn't so I can't get on the console. I've installed the remote CLI and am trying figure out how to run the commands through there. I'll let you know how it goes in a bit.

thanks for the help so far.
ASKER CERTIFIED SOLUTION
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
OK, so that worked, but now I have a new problem :(
It said it copied all of the files successfully, but one out of the three VMs is missing two files: the .VMX and .VSWP. The other two VMs had all of their files successfully and I was able to start them up.

The files are no longer in the old location and didn't show up in the local datastore either. My guess is some kind of corruption from all the iSCSI problems?

The .VMDK is there, so is there any way to get the VM rebuilt?
Thanks for all the help!
I created a new VM and just attached it to the .VMDK and now all three machines are on the local storage and running normally. Thanks again!
Glad you got it sorted. You may want to test it carefully! (iSCSI Storage).

NFS works well on Storage Server for ESX.