VMWare ESX 4 hosts not seeing ISCSI storage - missing vmkiscsiconf file?

Yes, it's likely to be your issue if you've not got a /etc/vmkiscsi.conf file.

I would disable iSCSI Software Initiator.

Restart ESX server.

RE-enable and fill in the IP address and details for you MSA.

I remember there was something with iscsi and jumbo frames: http://mike.thegrahams.ws/?p=45

Perhaps it will lead you to the right direction.

This was all working when you enabled Jumbo Frames on you MSA and Switches?

Did you enable Jumbo Frames on the Servers?

check my solutio here for enabling jumbo frames on your servers, (or check MTU is 9000)

@hanccocka: do you ever sleep?

;)

What's sleep!!!

Quick reply guys, appreciated.  

It may sound dum but do I need the "ISCSI Software Adapter" enabled as well as the Software Initiator enabled that is listed as enabled on HBA's?

To answer your last post hancocka, this hasn't worked since reconfig.  I simply ran through the steps above getting back to where I could scan for storage.

The SAN is showing the HBA as discovered and the HBAs are showing the SAN as discovered.

Do you have a quick CLI command for disabling the Software Initiators?  They are greyed out in the GUI...

Okay, you need to enable Jumbo Frames on the server!

Because you need to enable jumbo frames everywhere.

Q. Are you using Software of Hardware iSCSI Initiators?

MTU for switches output

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         3           128               9000    vmnic5

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS             0        1           vmnic5

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch3         128         3           128               9000    vmnic7

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS2            0        1           vmnic7

Is this what you mean by enabling on the server?

okay, that looks okay.

Q. Are you using Software of Hardware iSCSI Initiators?

OK the Broadcoms don't work as hardware initiator though they were listed that they should.
So the work around has been to set them as software initiators as per VMWare KB above..

esxcfg-swiscsi -d

I have to say, it was a bit of a pig to set up originally...when it used to work

okay, so you are ujsing Hardware iSCSI Initiators, so the iSCSI Software Stuff does not apply to this.

did it used to work without jumbo frames?

OK done, rebooting hosts

No Software iSCSI

My issue was as below;

No storage devices visible with Broadcom iSCSI offload-enabled adapters
Symptoms
When using Broadcom network interfaces with Hardware iSCSI Offload capabilities such as the BCM5709 and the BCM57711, each of the network interfaces shows up as individual storage adapters in the Configuration tab of an ESX Host and each adapter also has an individual IQN name. However when you configure the adapter for iSCSI, you observe these symptoms:

    * You are also unable to add any discovery addresses
    * When observing the Dynamic or Static Discovery tabs in the iSCSI Initiator Properties window, the following message is displayed:

      The host bus adapter is not associated with a vmknic. To configure targets the adapter should be associated with a vmknic. Refer to the VMware documentation to associate the adapter with a vmknic.

    * Each one of the adaptors shows no associated storage devices

Resolution
Broadcom cards with iSCSI Offload capabilities are also Hardware iSCSI Initiators. To configure them for hardware iSCSI, perform one of these procedures:

Note: To log into the terminal for VMware ESXi hosts, see Using Tech Support Mode in ESXi 4.1 (1017910).

   1. Configure LUN Access, as any other hardware iSCSI HBA, within the BIOS of the Broadcom Card during system boot. It is recommended to ensure that your SAN is supported to use Broadcom as a hardware initiator before using this option.

      Or

   2. Add the Physical Network Interface as a binding to the Software iSCSI HBA Device by running the following commands. In this example, we are referencing the Software iSCSI HBA as vmhba32:

         1.
            List the currently-bound Network Interfaces to the Software iSCSI HBA by running the following command:

            # esxcli swiscsi vmnic list -d vmhba32

         2.
            List the current available network interfaces available on the ESX Host and identify the interface(s) you wish to bind:

            # esxcfg-vmknic -l

         3.
            Bind the network interface(s) identified in the previous step to the Software iSCSI HBA Device:

            # esxcli swiscsi nic add -n vmk0 -d vmhba32

         4.
            Verify the network interface(s) were successfully bound to the HBA:

            # esxcli swiscsi vmnic list -d vmhba32

         5.
            Enable the Software iSCSI Adapter in the Storage Adapters section of the Configuration tab.
         6.
            Enter the dynamic discovery addresses of the storage array.
         7. At the top-right, perform a Rescan of the Software iSCSI HBA.

      Note: You may repeat this process and create additional bindings. For additional information, see the below section.

You may have to ENABLED jumbo frames on the Hardware Initiators?

I don't think these are vmnic 5 or vmnic 7 are they?

yes, i'm getting there......

it used to work before jumbo frames BUT it didn't work the first time or the second or the third....you get the picture.  And then when I had reset all the CHAP setting on the SAN (which I tried this time) it did eventually work but I couldn't put my finger on why because it all looked good..

is there a blanket command for esxcfg-swiscsi to renable?

have you set the MTU of 9000 on the HBAs?

esxcfg-swiscsi -e

OK The Hosts are back.  The software initiator is showing as enabled and discovered hosts are showing.  Can't see where I enable the Jumbo frames on the HBAs....not in the GUI anyhow

esxcfg-vmknic -l still shows MTU 9000 if that's what you mean..

I can accept that.  Shall I roll back the config back to MTU 1500?  Crazy though if it's true....

No we've done lots of work in this area, and I was only citing a reference to this yesterday on EE.

We have found that the Software iSCSI Initiator with Jumbo Frames enabled, was faster in performance, than Hardware iSCSI Initiators without Jumbo Frames (only because most HBAs we've used don't support Jumbo Frames).

Latency!

I'll try and find the EE link....

Officially they do but maybe this is a setting in the card BIOS?  It's selectable from Windows apparently.

you have a choice

1. Rollback (reset back to MTU 1500) to Hardware Initators as before.

2. Enable Software init, enable Jumbo Frames, and ditch BCM5709/BCM57711 HBAs and use Software Initiator only.

I think at this stage to reduce risk Rollback, and Test 2 above some other time.

you could just quickly see here:-

esxcli swiscsi vmnic list -d vmhba34

esxcli swiscsi nic list -d vmhba34

to show bound nics

vmnic5
    vmnic name: vmnic5
    mac address: d4:85:64:57:65:f2
    mac address settable: NO
    maximum transfer rate: 1000
    current transfer rate: 1000
    maximum frame size: 9000

https://www.experts-exchange.com/questions/26730342/Best-practice-for-migrating-VM's-from-different-ESX-hosts-which-some-shares-the-same-LAN-and-some-not.html

this was the link, very last comment but one.

Going to roll back....still concerned about that missing missing vmkiscsiconf file though.  Is this generated upon configuration?

well the MTU is there 9000.

Okay, I've just found this

http://communities.vmware.com/message/1670737

this confirms that The Broadcom 5709 in 4.1 does not support Jumbo Frames AT THIS TIME.

29 Dec 2010.

As you are on ESX 4, this confirms my suspicions, this post also strengthens our findings:-

"I recommend using the SW initiator w/ Jumbo frames as performance will be better then HW anyway....according to many tests done by others on blogs the performance with HW offload is not enough to AT THIS TIME to use it."

Sorry you had a wasted night.

1. Rollback MTU to 1500.

2. Use Software Init Only at 9000.

yes, it's generated, after config usually.

Got it all back and running under the original settings.  I *may* try again on one host in the future.
I did notice some strange behaviour connecting.  Proceedure I followed was remove mappings, deleted chap on the HBA and SAN.  Forced rescan.  You can only add the storage after the CHAP has been reconfigured fresh.

Cheers Hancocka for your out of hours help, you deserve all the points and more!

Great help

Many Thanks