?
Solved

We cant create new guest machine using iSCSI storage

Posted on 2008-11-13
14
Medium Priority
?
750 Views
Last Modified: 2012-06-22
We are having two vm hosts running on VMWARE ESX 3.0.2 on HP DL servers. We connected iSCSI storage using drive QLAN 4050c. Its connected through hardware initiator using HP MSA i1510. We installed few machines using this storage and everything works fine. But now , we noticed that neither we can create a new machine nor create new folder through SSH in that storage. We have rebooted VM hosts, iSCSI controller. We have enabled ESX firewall to allow swiscsi also. But no success. We dont want to re-install  the whole thing. There are 10 VMs running now.  Any expertise help is very much appreciated.
0
Comment
Question by:uniserveps
  • 8
  • 5
14 Comments
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 22957284
I have setup this scenerio with several customers, the 1510i usually works well and the HP DLs are solid servers.

1.  Are you using standalone ESX or Virtutal Center
2.  Are you seeing the datastores or not?  You stated that you can't create new VMs via SSH, have you logged in using Virtual Infrastructure Client?
3.  Do you have enough space on the datastores after VM creation for swap file, etc...?
4.  What type of error are you getting in the logs when you do try to create a VM?
0
 

Author Comment

by:uniserveps
ID: 22961542
HI Paul,
Here is my answers.
1. We are using stand alone ESX box (two) and this storage shared through a physical ethernet switch.
2. We are seeing datastore on both ESX boxes along with the local storage.
3. Yes. We have 400GB left (total 1.3TB ). There are 8 machines (Both ESX) are in and all working fine.
4.  The error is not specific. When I tried to create a new VM through Virtual Center Client it just gives "an internal error occured" . If I try to create a folder in that storage through SSH, it gives "can not allocate memory".

But all the guest machines are working fine.   When we try to browse the datastore through VC Client, we can see the folders list. We cannot navigate those folders.


0
Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

 

Author Comment

by:uniserveps
ID: 22961581
Hi Paul,
I mentioned as Virtual Center Client. Please change this as 'Virtual Infrastructure Client" . Sorry for the typo.
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 22964136
How large of a virtual machine are you trying to create? If it's over 256GB and you need to have a larger block size on the datastore if you set it up with the default block size.  Just trying to rule out the obvious first.

Can you brose the current VM folders or are you having problems only with new VM folders that you're trying to create?

If you have space try to create another datastore and see if you're having an issue with the particular datastore or something else.  Let's try to baseline each piece of of puzzle until we find the issue.  

Check VMware ESX Log File and make sure no error:

   - /var/log/vmkernel

   - /var/log/vmkwarning

   - /var/log/message

0
 

Author Comment

by:uniserveps
ID: 22964901
I tried to create VM with 80GB. So I think it should be well within the limitation of 256G.
I can browse the content of the datastore. There are few VMs running on that data store. But its weird that i cant browse inside the folders..ie  if we have machine name "acctsrv" and i can see this folder. but I cant go inside the folder to see the vmx/vmdk files. There was a problem sometime before and we created a new VM , but pointing to the VMDK inside that datastore. Since all of these machines in productions, we cant disturb them.  But now , we cant even create a new machine using that datastore.  I cant create a datastore since we used the optimal space.
Here are some more details :  
We have a separate physical vmware license server and these two VM hosts pointing this license server for license. We want to do clustering in future if we rectify this issue.
I looked all these logs and i dont see any error message.
I have one question : Can we do rescan for the iscsi storage? if we do , will we loose any data on the datastore ?

 
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 22965149
This is how I would proceed:

1.  Backup the VMs using VRanger (outside of VM) or Symantec Backup Exec System Recovery Image (inside of the vm), if anything goes wrong you're covered ( you can download trial copy of each in a pinch)
2.  Create a second datastore if possible and try to create another vm inside of it.  This will baseline datastore vs iscsi or other issue
3.  Rescan ISCSI
0
 

Author Comment

by:uniserveps
ID: 22965475
Thanks Paul.
* We have installed symantec backup exec and acronis true image. We will do this and lets see.
* I am not sure I can create another datastore since no space left out to create.
* After taking the back up, i will rescan iSCSI.
* We planned to do this during week end and update you
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 22965547
If you have a spare system you can setup openfiler (www.openfiler.com) in a pinch and setup for ISCSI. This can be done with any workstatation just for baseline, normally takes about 30 min. to install.  Setup a second datastore and try to create a VM, this will let us know whether it's an esx issue or datastore issue.

Symantec Backup Exec System Recovery usually does a good job for me because you can use vmware converter or the symantec built-in tool to convert to vmdk.
0
 

Author Comment

by:uniserveps
ID: 22969365
Unfortunately we dont have a spare one apart from one license server. Its already loaded with few other stuff. But I feel the backup of all data for the safer side and do the rescan. I dont see any error in the iSCSI controller when I connected through the management.  I am taking all the servers back up now and going to proceed with rescan. I will update you this night. Thanks.
0
 

Author Comment

by:uniserveps
ID: 22969526
When  I rescan i get the  warning error message log  like this  :

//SCSI: 5120: None of the paths to target vmhba0:1:1 are working .
 vmhost1 vmkernel: 3:01:14:58.393 cpu7:1037)WARNING: SCSI: 5625: status No connection, rstatus 0xc0de00 for vmhba0:1:1. residual R 998, CR 80, ER 3 ///


 
0
 
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 22970853
sounds like scsi reservation conflict try this thread

http://communities.vmware.com/thread/114224
0
 

Author Comment

by:uniserveps
ID: 22976311
Hi Paul,
I have gone through the entire log. I dont see anywhere "resource conflict" in my logs.
And also in that thread, it mentioned as two qlogics and already clustered environment.  While doing the rescan, I tried to do  the netstat and grep 3260 for anything going on through that port. I dont get any error.
But, all the machines in that storage are running.
0
 

Accepted Solution

by:
uniserveps earned 0 total points
ID: 22992051
Hi Paul,
This issue is resolved with the help of VM support. Issue is that the VM hosts created too many logs in that shared storage and locked the file system. we deleted all the vmware logs and rebooted the vmhosts.
thanks for your help
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Giving access to ESXi shell console is always an issue for IT departments to other Teams, or Projects. We need to find a way so that teams can use ESXTOP for their POCs, or tests without giving them the access to ESXi host shell console with a root …
It’s time for spooky stories and consuming way too much sugar, including the many treats we’ve whipped for you in the world of tech. Check it out!
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…
Suggested Courses

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question