Solved

Windows Storage Server 2003 R2 iSCSI target losing connection

Posted on 2011-02-14
13
2,628 Views
Last Modified: 2012-05-11
We have a Windows 2003 R2 Storage Server hosting shares in our network. It was also configured as an iSCSI target for a VMware ESXi 4.0.0 server and several VMs were stored on it.

A couple of weeks ago the VMs became unavailable and we haven't been able to get them running since. As soon as I try to do anything with them (start the VMs, mount the drives in a different VM, download the files, etc) the vSphere Client crashes and it loses its connection to the iSCSI target. I have to go to the SAN and restart the "Microsoft iSCSI Software Target" service, then rescan the Storage Adapters in VMware.

Once I do that it reconnects to the storage and I can see the files stored in it. I'm able to Browse the datastore, see the VMs powered down, and even add the virtual drives to other VMs, but as soon as I try to actually do anything with them it crashes again.

Restarting the ESXi host does not help, only restarting the iSCSI service on the SAN so I'm thinking that the problem is with Microsoft, but aren't really sure.

We've added enough local storage to the ESXi host to move the VMs there, so all I want to do is access the data and get it off the SAN. I don't need to get it working permanently, just long enough to pull the virtual disks to different storage.

I've tried mounting the .VHD that Windows SS created in a windows 2008 server, but it says that the drive has to be NTFS to mount. I've also tried using the Microsoft iSCSI Software Target MMC to mount the drive, but can't give it a drive letter and see the data. I'm guessing it's because the drive was formatted by VMWare, but again I'm taking a guess.

Does anyone have any thoughts on how to get access to these files long enough to move them to better storage?
0
Comment
Question by:sfcanderson
  • 8
  • 5
13 Comments
 
LVL 118
ID: 34892692
Microsoft Windows Storage Server 2003 R2 iSCSI is not on the VMware Hardware Compatibility list for VMware ESXi 4.0.0 Server. (so you run the risk of issues, anyway, that out of the way).

Is the Microsoft iSCSI Software Target service crashing on the Storage Server?

0
 
LVL 118
ID: 34892700
Okay, ignoring the vSphere Client have you tried, using the ESX console, and re-scanning for the volumes at the command line, and then using cp command to copy the VMs off the iSCSI LUNs to local datastore.
0
 
LVL 118
ID: 34892712
e.g. at console esxcfg-rescan vmhba33 (normally iSCSI software hba)
0
 
LVL 118
ID: 34892731
I think what we need to try and work out, is what is causing the service to fail, is it lack of resources on the Windows server, so you may have to shutdown, and stop all shares or activity etc

0
 

Author Comment

by:sfcanderson
ID: 34892735
Ugh, I didn't even think to check the VMware HCL! This was set up by a previous IT contact, so I'm trying to clean things up.

The Software Target service on the Storage Server is not crashing, It stays running. The only way I've found to regain access to the data, though, is to restart that service.
0
 
LVL 118
ID: 34892812
okay, that's good.

forget the vsphere client.

login to console of esx server, rescan for uba, and then check the vmfs luns, and manually copy to local data store, and see if it stays up long enough for copy.

let me know, got a few more ideas, see how that goes.
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:sfcanderson
ID: 34893106
This is ESXi, so I don't think I can access the command line from the console. I'll try the Remote CLI a little later, but every time I've tried to copy the data through the vSphere Client things lock up.

This is also the main storage server for the network, so people are accessing it's shares all day. I've cleared it with management to reboot the server at 8:00 tonight (CST) and spend a couple of hours troubleshooting it. I'll provide more info when I get a chance to do those.
0
 
LVL 118
ID: 34893186
ESXi - yes you can get to the command line.

The fastest way to copy data off an iSCSI datastore, is at the command line (console).

from datastore to local datastore. (provided your iSCSI store stays up long enough), but we will see if this is the issue, if there is a timeout when the server tries to copy data off the iSCSI datastore to local datastore (local disk), we will have to think how we can beef-up fast access.
0
 

Author Comment

by:sfcanderson
ID: 34893473
OK, I'd thought this server had an ILO card, but I guess it doesn't so I can't get on the console. I've installed the remote CLI and am trying figure out how to run the commands through there. I'll let you know how it goes in a bit.

thanks for the help so far.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 34893593
can you ssh to the server?

what you could do is, use the vpshere client to connect to the server, create a user account that has shell access.

use that user account to access the shell using ssh, and whilst logged in with that user, then su to root.

so no need for ilo.

whilst then logged, esxcfg-rescan vmbha33

check iSCSI datastores, and then copy vms and folders to local disk (datastores)
0
 

Author Comment

by:sfcanderson
ID: 34893854
OK, so that worked, but now I have a new problem :(
It said it copied all of the files successfully, but one out of the three VMs is missing two files: the .VMX and .VSWP. The other two VMs had all of their files successfully and I was able to start them up.

The files are no longer in the old location and didn't show up in the local datastore either. My guess is some kind of corruption from all the iSCSI problems?

The .VMDK is there, so is there any way to get the VM rebuilt?
0
 

Author Closing Comment

by:sfcanderson
ID: 34893942
Thanks for all the help!
I created a new VM and just attached it to the .VMDK and now all three machines are on the local storage and running normally. Thanks again!
0
 
LVL 118
ID: 34895001
Glad you got it sorted. You may want to test it carefully! (iSCSI Storage).

NFS works well on Storage Server for ESX.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Teach the user how to convert virtaul disk file formats and how to rename virtual machine files on datastores. Open vSphere Web Client: Review VM disk settings: Migrate VM to new datastore with a thick provisioned (lazy zeroed) disk format: Rename a…
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now