Solved

Windows Storage Server 2003 R2 iSCSI target losing connection

Posted on 2011-02-14
13
2,643 Views
Last Modified: 2012-05-11
We have a Windows 2003 R2 Storage Server hosting shares in our network. It was also configured as an iSCSI target for a VMware ESXi 4.0.0 server and several VMs were stored on it.

A couple of weeks ago the VMs became unavailable and we haven't been able to get them running since. As soon as I try to do anything with them (start the VMs, mount the drives in a different VM, download the files, etc) the vSphere Client crashes and it loses its connection to the iSCSI target. I have to go to the SAN and restart the "Microsoft iSCSI Software Target" service, then rescan the Storage Adapters in VMware.

Once I do that it reconnects to the storage and I can see the files stored in it. I'm able to Browse the datastore, see the VMs powered down, and even add the virtual drives to other VMs, but as soon as I try to actually do anything with them it crashes again.

Restarting the ESXi host does not help, only restarting the iSCSI service on the SAN so I'm thinking that the problem is with Microsoft, but aren't really sure.

We've added enough local storage to the ESXi host to move the VMs there, so all I want to do is access the data and get it off the SAN. I don't need to get it working permanently, just long enough to pull the virtual disks to different storage.

I've tried mounting the .VHD that Windows SS created in a windows 2008 server, but it says that the drive has to be NTFS to mount. I've also tried using the Microsoft iSCSI Software Target MMC to mount the drive, but can't give it a drive letter and see the data. I'm guessing it's because the drive was formatted by VMWare, but again I'm taking a guess.

Does anyone have any thoughts on how to get access to these files long enough to move them to better storage?
0
Comment
Question by:sfcanderson
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 5
13 Comments
 
LVL 121
ID: 34892692
Microsoft Windows Storage Server 2003 R2 iSCSI is not on the VMware Hardware Compatibility list for VMware ESXi 4.0.0 Server. (so you run the risk of issues, anyway, that out of the way).

Is the Microsoft iSCSI Software Target service crashing on the Storage Server?

0
 
LVL 121
ID: 34892700
Okay, ignoring the vSphere Client have you tried, using the ESX console, and re-scanning for the volumes at the command line, and then using cp command to copy the VMs off the iSCSI LUNs to local datastore.
0
 
LVL 121
ID: 34892712
e.g. at console esxcfg-rescan vmhba33 (normally iSCSI software hba)
0
Turn your laptop into a mobile console!

The CV211 Laptop USB Console Adapter provides a direct Laptop-to-Computer connection for fast and easy remote desktop access with no software to install.

 
LVL 121
ID: 34892731
I think what we need to try and work out, is what is causing the service to fail, is it lack of resources on the Windows server, so you may have to shutdown, and stop all shares or activity etc

0
 

Author Comment

by:sfcanderson
ID: 34892735
Ugh, I didn't even think to check the VMware HCL! This was set up by a previous IT contact, so I'm trying to clean things up.

The Software Target service on the Storage Server is not crashing, It stays running. The only way I've found to regain access to the data, though, is to restart that service.
0
 
LVL 121
ID: 34892812
okay, that's good.

forget the vsphere client.

login to console of esx server, rescan for uba, and then check the vmfs luns, and manually copy to local data store, and see if it stays up long enough for copy.

let me know, got a few more ideas, see how that goes.
0
 

Author Comment

by:sfcanderson
ID: 34893106
This is ESXi, so I don't think I can access the command line from the console. I'll try the Remote CLI a little later, but every time I've tried to copy the data through the vSphere Client things lock up.

This is also the main storage server for the network, so people are accessing it's shares all day. I've cleared it with management to reboot the server at 8:00 tonight (CST) and spend a couple of hours troubleshooting it. I'll provide more info when I get a chance to do those.
0
 
LVL 121
ID: 34893186
ESXi - yes you can get to the command line.

The fastest way to copy data off an iSCSI datastore, is at the command line (console).

from datastore to local datastore. (provided your iSCSI store stays up long enough), but we will see if this is the issue, if there is a timeout when the server tries to copy data off the iSCSI datastore to local datastore (local disk), we will have to think how we can beef-up fast access.
0
 

Author Comment

by:sfcanderson
ID: 34893473
OK, I'd thought this server had an ILO card, but I guess it doesn't so I can't get on the console. I've installed the remote CLI and am trying figure out how to run the commands through there. I'll let you know how it goes in a bit.

thanks for the help so far.
0
 
LVL 121

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 34893593
can you ssh to the server?

what you could do is, use the vpshere client to connect to the server, create a user account that has shell access.

use that user account to access the shell using ssh, and whilst logged in with that user, then su to root.

so no need for ilo.

whilst then logged, esxcfg-rescan vmbha33

check iSCSI datastores, and then copy vms and folders to local disk (datastores)
0
 

Author Comment

by:sfcanderson
ID: 34893854
OK, so that worked, but now I have a new problem :(
It said it copied all of the files successfully, but one out of the three VMs is missing two files: the .VMX and .VSWP. The other two VMs had all of their files successfully and I was able to start them up.

The files are no longer in the old location and didn't show up in the local datastore either. My guess is some kind of corruption from all the iSCSI problems?

The .VMDK is there, so is there any way to get the VM rebuilt?
0
 

Author Closing Comment

by:sfcanderson
ID: 34893942
Thanks for all the help!
I created a new VM and just attached it to the .VMDK and now all three machines are on the local storage and running normally. Thanks again!
0
 
LVL 121
ID: 34895001
Glad you got it sorted. You may want to test it carefully! (iSCSI Storage).

NFS works well on Storage Server for ESX.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…
Suggested Courses

622 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question