We cant create new guest machine using iSCSI storage

I have setup this scenerio with several customers, the 1510i usually works well and the HP DLs are solid servers.

1.  Are you using standalone ESX or Virtutal Center
2.  Are you seeing the datastores or not?  You stated that you can't create new VMs via SSH, have you logged in using Virtual Infrastructure Client?
3.  Do you have enough space on the datastores after VM creation for swap file, etc...?
4.  What type of error are you getting in the logs when you do try to create a VM?

Maybe you can check this post:
http://communities.vmware.com/message/1024105;jsessionid=68FC241114E8B9B2D262B560C39FBD6E

HI Paul,
Here is my answers.
1. We are using stand alone ESX box (two) and this storage shared through a physical ethernet switch.
2. We are seeing datastore on both ESX boxes along with the local storage.
3. Yes. We have 400GB left (total 1.3TB ). There are 8 machines (Both ESX) are in and all working fine.
4.  The error is not specific. When I tried to create a new VM through Virtual Center Client it just gives "an internal error occured" . If I try to create a folder in that storage through SSH, it gives "can not allocate memory".

But all the guest machines are working fine.   When we try to browse the datastore through VC Client, we can see the folders list. We cannot navigate those folders.

Hi Paul,
I mentioned as Virtual Center Client. Please change this as 'Virtual Infrastructure Client" . Sorry for the typo.

How large of a virtual machine are you trying to create? If it's over 256GB and you need to have a larger block size on the datastore if you set it up with the default block size.  Just trying to rule out the obvious first.

Can you brose the current VM folders or are you having problems only with new VM folders that you're trying to create?

If you have space try to create another datastore and see if you're having an issue with the particular datastore or something else.  Let's try to baseline each piece of of puzzle until we find the issue.  

Check VMware ESX Log File and make sure no error:

   - /var/log/vmkernel

   - /var/log/vmkwarning

   - /var/log/message

I tried to create VM with 80GB. So I think it should be well within the limitation of 256G.
I can browse the content of the datastore. There are few VMs running on that data store. But its weird that i cant browse inside the folders..ie  if we have machine name "acctsrv" and i can see this folder. but I cant go inside the folder to see the vmx/vmdk files. There was a problem sometime before and we created a new VM , but pointing to the VMDK inside that datastore. Since all of these machines in productions, we cant disturb them.  But now , we cant even create a new machine using that datastore.  I cant create a datastore since we used the optimal space.
Here are some more details :  
We have a separate physical vmware license server and these two VM hosts pointing this license server for license. We want to do clustering in future if we rectify this issue.
I looked all these logs and i dont see any error message.
I have one question : Can we do rescan for the iscsi storage? if we do , will we loose any data on the datastore ?

 

This is how I would proceed:

1.  Backup the VMs using VRanger (outside of VM) or Symantec Backup Exec System Recovery Image (inside of the vm), if anything goes wrong you're covered ( you can download trial copy of each in a pinch)
2.  Create a second datastore if possible and try to create another vm inside of it.  This will baseline datastore vs iscsi or other issue
3.  Rescan ISCSI

Thanks Paul.
* We have installed symantec backup exec and acronis true image. We will do this and lets see.
* I am not sure I can create another datastore since no space left out to create.
* After taking the back up, i will rescan iSCSI.
* We planned to do this during week end and update you

If you have a spare system you can setup openfiler (www.openfiler.com) in a pinch and setup for ISCSI. This can be done with any workstatation just for baseline, normally takes about 30 min. to install.  Setup a second datastore and try to create a VM, this will let us know whether it's an esx issue or datastore issue.

Symantec Backup Exec System Recovery usually does a good job for me because you can use vmware converter or the symantec built-in tool to convert to vmdk.

Unfortunately we dont have a spare one apart from one license server. Its already loaded with few other stuff. But I feel the backup of all data for the safer side and do the rescan. I dont see any error in the iSCSI controller when I connected through the management.  I am taking all the servers back up now and going to proceed with rescan. I will update you this night. Thanks.

When  I rescan i get the  warning error message log  like this  :

//SCSI: 5120: None of the paths to target vmhba0:1:1 are working .
 vmhost1 vmkernel: 3:01:14:58.393 cpu7:1037)WARNING: SCSI: 5625: status No connection, rstatus 0xc0de00 for vmhba0:1:1. residual R 998, CR 80, ER 3 ///


 

sounds like scsi reservation conflict try this thread

http://communities.vmware.com/thread/114224

Hi Paul,
I have gone through the entire log. I dont see anywhere "resource conflict" in my logs.
And also in that thread, it mentioned as two qlogics and already clustered environment.  While doing the rescan, I tried to do  the netstat and grep 3260 for anything going on through that port. I dont get any error.
But, all the machines in that storage are running.