Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Windows Storage Server 2003 R2 iSCSI target losing connection

Posted on 2011-02-14
13
Medium Priority
?
2,650 Views
Last Modified: 2012-05-11
We have a Windows 2003 R2 Storage Server hosting shares in our network. It was also configured as an iSCSI target for a VMware ESXi 4.0.0 server and several VMs were stored on it.

A couple of weeks ago the VMs became unavailable and we haven't been able to get them running since. As soon as I try to do anything with them (start the VMs, mount the drives in a different VM, download the files, etc) the vSphere Client crashes and it loses its connection to the iSCSI target. I have to go to the SAN and restart the "Microsoft iSCSI Software Target" service, then rescan the Storage Adapters in VMware.

Once I do that it reconnects to the storage and I can see the files stored in it. I'm able to Browse the datastore, see the VMs powered down, and even add the virtual drives to other VMs, but as soon as I try to actually do anything with them it crashes again.

Restarting the ESXi host does not help, only restarting the iSCSI service on the SAN so I'm thinking that the problem is with Microsoft, but aren't really sure.

We've added enough local storage to the ESXi host to move the VMs there, so all I want to do is access the data and get it off the SAN. I don't need to get it working permanently, just long enough to pull the virtual disks to different storage.

I've tried mounting the .VHD that Windows SS created in a windows 2008 server, but it says that the drive has to be NTFS to mount. I've also tried using the Microsoft iSCSI Software Target MMC to mount the drive, but can't give it a drive letter and see the data. I'm guessing it's because the drive was formatted by VMWare, but again I'm taking a guess.

Does anyone have any thoughts on how to get access to these files long enough to move them to better storage?
0
Comment
Question by:sfcanderson
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 5
13 Comments
 
LVL 123
ID: 34892692
Microsoft Windows Storage Server 2003 R2 iSCSI is not on the VMware Hardware Compatibility list for VMware ESXi 4.0.0 Server. (so you run the risk of issues, anyway, that out of the way).

Is the Microsoft iSCSI Software Target service crashing on the Storage Server?

0
 
LVL 123
ID: 34892700
Okay, ignoring the vSphere Client have you tried, using the ESX console, and re-scanning for the volumes at the command line, and then using cp command to copy the VMs off the iSCSI LUNs to local datastore.
0
 
LVL 123
ID: 34892712
e.g. at console esxcfg-rescan vmhba33 (normally iSCSI software hba)
0
NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

 
LVL 123
ID: 34892731
I think what we need to try and work out, is what is causing the service to fail, is it lack of resources on the Windows server, so you may have to shutdown, and stop all shares or activity etc

0
 

Author Comment

by:sfcanderson
ID: 34892735
Ugh, I didn't even think to check the VMware HCL! This was set up by a previous IT contact, so I'm trying to clean things up.

The Software Target service on the Storage Server is not crashing, It stays running. The only way I've found to regain access to the data, though, is to restart that service.
0
 
LVL 123
ID: 34892812
okay, that's good.

forget the vsphere client.

login to console of esx server, rescan for uba, and then check the vmfs luns, and manually copy to local data store, and see if it stays up long enough for copy.

let me know, got a few more ideas, see how that goes.
0
 

Author Comment

by:sfcanderson
ID: 34893106
This is ESXi, so I don't think I can access the command line from the console. I'll try the Remote CLI a little later, but every time I've tried to copy the data through the vSphere Client things lock up.

This is also the main storage server for the network, so people are accessing it's shares all day. I've cleared it with management to reboot the server at 8:00 tonight (CST) and spend a couple of hours troubleshooting it. I'll provide more info when I get a chance to do those.
0
 
LVL 123
ID: 34893186
ESXi - yes you can get to the command line.

The fastest way to copy data off an iSCSI datastore, is at the command line (console).

from datastore to local datastore. (provided your iSCSI store stays up long enough), but we will see if this is the issue, if there is a timeout when the server tries to copy data off the iSCSI datastore to local datastore (local disk), we will have to think how we can beef-up fast access.
0
 

Author Comment

by:sfcanderson
ID: 34893473
OK, I'd thought this server had an ILO card, but I guess it doesn't so I can't get on the console. I've installed the remote CLI and am trying figure out how to run the commands through there. I'll let you know how it goes in a bit.

thanks for the help so far.
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 34893593
can you ssh to the server?

what you could do is, use the vpshere client to connect to the server, create a user account that has shell access.

use that user account to access the shell using ssh, and whilst logged in with that user, then su to root.

so no need for ilo.

whilst then logged, esxcfg-rescan vmbha33

check iSCSI datastores, and then copy vms and folders to local disk (datastores)
0
 

Author Comment

by:sfcanderson
ID: 34893854
OK, so that worked, but now I have a new problem :(
It said it copied all of the files successfully, but one out of the three VMs is missing two files: the .VMX and .VSWP. The other two VMs had all of their files successfully and I was able to start them up.

The files are no longer in the old location and didn't show up in the local datastore either. My guess is some kind of corruption from all the iSCSI problems?

The .VMDK is there, so is there any way to get the VM rebuilt?
0
 

Author Closing Comment

by:sfcanderson
ID: 34893942
Thanks for all the help!
I created a new VM and just attached it to the .VMDK and now all three machines are on the local storage and running normally. Thanks again!
0
 
LVL 123
ID: 34895001
Glad you got it sorted. You may want to test it carefully! (iSCSI Storage).

NFS works well on Storage Server for ESX.
0

Featured Post

The Ideal Solution for Multi-Display Applications

Check out ATEN’s VS1912 12-Port DP Video Wall Media Player at InfoComm 2017. Kerri describes how easy it is to design creative video walls in asymmetric layouts and schedule detailed playlists ahead of time with its advanced scheduling feature.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

While rebooting windows server 2003 server , it's showing "active directory rebuilding indices please wait" at startup. It took a little while for this process to complete and once we logged on not all the services were started so another reboot is …
Each year, investment in cloud platforms grows more than 20% (https://www.immun.io/hubfs/Immunio_2016/Content/Marketing/Cloud-Security-Report-2016.pdf?submissionGuid=a8d80a00-6fee-4b85-81db-a4e28f681762) as an increasing number of companies begin to…
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question