We cant create new guest machine using iSCSI storage

We are having two vm hosts running on VMWARE ESX 3.0.2 on HP DL servers. We connected iSCSI storage using drive QLAN 4050c. Its connected through hardware initiator using HP MSA i1510. We installed few machines using this storage and everything works fine. But now , we noticed that neither we can create a new machine nor create new folder through SSH in that storage. We have rebooted VM hosts, iSCSI controller. We have enabled ESX firewall to allow swiscsi also. But no success. We dont want to re-install  the whole thing. There are 10 VMs running now.  Any expertise help is very much appreciated.
uniservepsAsked:
Who is Participating?
 
uniservepsConnect With a Mentor Author Commented:
Hi Paul,
This issue is resolved with the help of VM support. Issue is that the VM hosts created too many logs in that shared storage and locked the file system. we deleted all the vmware logs and rebooted the vmhosts.
thanks for your help
0
 
Paul SolovyovskySenior IT AdvisorCommented:
I have setup this scenerio with several customers, the 1510i usually works well and the HP DLs are solid servers.

1.  Are you using standalone ESX or Virtutal Center
2.  Are you seeing the datastores or not?  You stated that you can't create new VMs via SSH, have you logged in using Virtual Infrastructure Client?
3.  Do you have enough space on the datastores after VM creation for swap file, etc...?
4.  What type of error are you getting in the logs when you do try to create a VM?
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
uniservepsAuthor Commented:
HI Paul,
Here is my answers.
1. We are using stand alone ESX box (two) and this storage shared through a physical ethernet switch.
2. We are seeing datastore on both ESX boxes along with the local storage.
3. Yes. We have 400GB left (total 1.3TB ). There are 8 machines (Both ESX) are in and all working fine.
4.  The error is not specific. When I tried to create a new VM through Virtual Center Client it just gives "an internal error occured" . If I try to create a folder in that storage through SSH, it gives "can not allocate memory".

But all the guest machines are working fine.   When we try to browse the datastore through VC Client, we can see the folders list. We cannot navigate those folders.


0
 
uniservepsAuthor Commented:
Hi Paul,
I mentioned as Virtual Center Client. Please change this as 'Virtual Infrastructure Client" . Sorry for the typo.
0
 
Paul SolovyovskySenior IT AdvisorCommented:
How large of a virtual machine are you trying to create? If it's over 256GB and you need to have a larger block size on the datastore if you set it up with the default block size.  Just trying to rule out the obvious first.

Can you brose the current VM folders or are you having problems only with new VM folders that you're trying to create?

If you have space try to create another datastore and see if you're having an issue with the particular datastore or something else.  Let's try to baseline each piece of of puzzle until we find the issue.  

Check VMware ESX Log File and make sure no error:

   - /var/log/vmkernel

   - /var/log/vmkwarning

   - /var/log/message

0
 
uniservepsAuthor Commented:
I tried to create VM with 80GB. So I think it should be well within the limitation of 256G.
I can browse the content of the datastore. There are few VMs running on that data store. But its weird that i cant browse inside the folders..ie  if we have machine name "acctsrv" and i can see this folder. but I cant go inside the folder to see the vmx/vmdk files. There was a problem sometime before and we created a new VM , but pointing to the VMDK inside that datastore. Since all of these machines in productions, we cant disturb them.  But now , we cant even create a new machine using that datastore.  I cant create a datastore since we used the optimal space.
Here are some more details :  
We have a separate physical vmware license server and these two VM hosts pointing this license server for license. We want to do clustering in future if we rectify this issue.
I looked all these logs and i dont see any error message.
I have one question : Can we do rescan for the iscsi storage? if we do , will we loose any data on the datastore ?

 
0
 
Paul SolovyovskySenior IT AdvisorCommented:
This is how I would proceed:

1.  Backup the VMs using VRanger (outside of VM) or Symantec Backup Exec System Recovery Image (inside of the vm), if anything goes wrong you're covered ( you can download trial copy of each in a pinch)
2.  Create a second datastore if possible and try to create another vm inside of it.  This will baseline datastore vs iscsi or other issue
3.  Rescan ISCSI
0
 
uniservepsAuthor Commented:
Thanks Paul.
* We have installed symantec backup exec and acronis true image. We will do this and lets see.
* I am not sure I can create another datastore since no space left out to create.
* After taking the back up, i will rescan iSCSI.
* We planned to do this during week end and update you
0
 
Paul SolovyovskySenior IT AdvisorCommented:
If you have a spare system you can setup openfiler (www.openfiler.com) in a pinch and setup for ISCSI. This can be done with any workstatation just for baseline, normally takes about 30 min. to install.  Setup a second datastore and try to create a VM, this will let us know whether it's an esx issue or datastore issue.

Symantec Backup Exec System Recovery usually does a good job for me because you can use vmware converter or the symantec built-in tool to convert to vmdk.
0
 
uniservepsAuthor Commented:
Unfortunately we dont have a spare one apart from one license server. Its already loaded with few other stuff. But I feel the backup of all data for the safer side and do the rescan. I dont see any error in the iSCSI controller when I connected through the management.  I am taking all the servers back up now and going to proceed with rescan. I will update you this night. Thanks.
0
 
uniservepsAuthor Commented:
When  I rescan i get the  warning error message log  like this  :

//SCSI: 5120: None of the paths to target vmhba0:1:1 are working .
 vmhost1 vmkernel: 3:01:14:58.393 cpu7:1037)WARNING: SCSI: 5625: status No connection, rstatus 0xc0de00 for vmhba0:1:1. residual R 998, CR 80, ER 3 ///


 
0
 
Paul SolovyovskySenior IT AdvisorCommented:
sounds like scsi reservation conflict try this thread

http://communities.vmware.com/thread/114224
0
 
uniservepsAuthor Commented:
Hi Paul,
I have gone through the entire log. I dont see anywhere "resource conflict" in my logs.
And also in that thread, it mentioned as two qlogics and already clustered environment.  While doing the rescan, I tried to do  the netstat and grep 3260 for anything going on through that port. I dont get any error.
But, all the machines in that storage are running.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.