VMWare ESX 4 hosts not seeing ISCSI storage - missing vmkiscsiconf file?

Hi There,

Have a major issue seeing an existing iscsi san from a pair of ESX 4 hosts.
When I first set these up I had a fair bit of difficulty in getting it to connect but eventualy I did.
Now I have (though I should have done before, doh) enabled Jumbo frames using the advice i this article;

http://www.sysadmintutorials.com/tutorials/vmware-vsphere-4/esx-4/enabling-jumbo-frames-with-vmware-esx-4/

I have also enabled jumbo frames on the SAN an HP MSA2312i.
The switches are 3Com Baseline Switch 2928 unit which support jumbo frames.

OK, I have this additional issue with particular Broadcom nics in use in this installation that VMWARE are aware of;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025644

I have done as they have said and now the HBAs once again discover the iscsi SAN.

BUT try as I might I cannot rediscover the storage.  I have rebooted and no change.  I can vmkping the SAN addresses.

Some searching around has led me to a troubleshooting page;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083

My big concern here is that I don't have a /etc/vmkiscsi.conf file!  Could this be my issue?

This is a pretty bad situation as VMs are down and I need it to be up (obviously)

Anyone got any clue?
abssupportAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes, it's likely to be your issue if you've not got a /etc/vmkiscsi.conf file.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
I would disable iSCSI Software Initiator.

Restart ESX server.

RE-enable and fill in the IP address and details for you MSA.
Ernie BeekExpertCommented:
I remember there was something with iscsi and jumbo frames: http://mike.thegrahams.ws/?p=45

Perhaps it will lead you to the right direction.
Ensure Business Longevity with As-A-Service

Using the as-a-service approach for your business model allows you to grow your revenue stream with new practice areas, without forcing you to part ways with existing clients just because they don’t fit the mold of your new service offerings.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
This was all working when you enabled Jumbo Frames on you MSA and Switches?

Did you enable Jumbo Frames on the Servers?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
check my solutio here for enabling jumbo frames on your servers, (or check MTU is 9000)
Ernie BeekExpertCommented:
@hanccocka: do you ever sleep?

;)
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
What's sleep!!!
abssupportAuthor Commented:
Quick reply guys, appreciated.  

It may sound dum but do I need the "ISCSI Software Adapter" enabled as well as the Software Initiator enabled that is listed as enabled on HBA's?

To answer your last post hancocka, this hasn't worked since reconfig.  I simply ran through the steps above getting back to where I could scan for storage.

The SAN is showing the HBA as discovered and the HBAs are showing the SAN as discovered.

Do you have a quick CLI command for disabling the Software Initiators?  They are greyed out in the GUI...
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Okay, you need to enable Jumbo Frames on the server!

Because you need to enable jumbo frames everywhere.

Q. Are you using Software of Hardware iSCSI Initiators?
abssupportAuthor Commented:
MTU for switches output

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         3           128               9000    vmnic5

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS             0        1           vmnic5

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch3         128         3           128               9000    vmnic7

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS2            0        1           vmnic7

Is this what you mean by enabling on the server?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
okay, that looks okay.

Q. Are you using Software of Hardware iSCSI Initiators?

abssupportAuthor Commented:
OK the Broadcoms don't work as hardware initiator though they were listed that they should.
So the work around has been to set them as software initiators as per VMWare KB above..
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
esxcfg-swiscsi -d
abssupportAuthor Commented:
I have to say, it was a bit of a pig to set up originally...when it used to work
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
okay, so you are ujsing Hardware iSCSI Initiators, so the iSCSI Software Stuff does not apply to this.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
did it used to work without jumbo frames?
abssupportAuthor Commented:
OK done, rebooting hosts
abssupportAuthor Commented:
No Software iSCSI

My issue was as below;

No storage devices visible with Broadcom iSCSI offload-enabled adapters
Symptoms
When using Broadcom network interfaces with Hardware iSCSI Offload capabilities such as the BCM5709 and the BCM57711, each of the network interfaces shows up as individual storage adapters in the Configuration tab of an ESX Host and each adapter also has an individual IQN name. However when you configure the adapter for iSCSI, you observe these symptoms:

    * You are also unable to add any discovery addresses
    * When observing the Dynamic or Static Discovery tabs in the iSCSI Initiator Properties window, the following message is displayed:

      The host bus adapter is not associated with a vmknic. To configure targets the adapter should be associated with a vmknic. Refer to the VMware documentation to associate the adapter with a vmknic.

    * Each one of the adaptors shows no associated storage devices

Resolution
Broadcom cards with iSCSI Offload capabilities are also Hardware iSCSI Initiators. To configure them for hardware iSCSI, perform one of these procedures:

Note: To log into the terminal for VMware ESXi hosts, see Using Tech Support Mode in ESXi 4.1 (1017910).

   1. Configure LUN Access, as any other hardware iSCSI HBA, within the BIOS of the Broadcom Card during system boot. It is recommended to ensure that your SAN is supported to use Broadcom as a hardware initiator before using this option.

      Or

   2. Add the Physical Network Interface as a binding to the Software iSCSI HBA Device by running the following commands. In this example, we are referencing the Software iSCSI HBA as vmhba32:

         1.
            List the currently-bound Network Interfaces to the Software iSCSI HBA by running the following command:

            # esxcli swiscsi vmnic list -d vmhba32

         2.
            List the current available network interfaces available on the ESX Host and identify the interface(s) you wish to bind:

            # esxcfg-vmknic -l

         3.
            Bind the network interface(s) identified in the previous step to the Software iSCSI HBA Device:

            # esxcli swiscsi nic add -n vmk0 -d vmhba32

         4.
            Verify the network interface(s) were successfully bound to the HBA:

            # esxcli swiscsi vmnic list -d vmhba32

         5.
            Enable the Software iSCSI Adapter in the Storage Adapters section of the Configuration tab.
         6.
            Enter the dynamic discovery addresses of the storage array.
         7. At the top-right, perform a Rescan of the Software iSCSI HBA.

      Note: You may repeat this process and create additional bindings. For additional information, see the below section.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You may have to ENABLED jumbo frames on the Hardware Initiators?

I don't think these are vmnic 5 or vmnic 7 are they?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
yes, i'm getting there......
abssupportAuthor Commented:
it used to work before jumbo frames BUT it didn't work the first time or the second or the third....you get the picture.  And then when I had reset all the CHAP setting on the SAN (which I tried this time) it did eventually work but I couldn't put my finger on why because it all looked good..
abssupportAuthor Commented:
is there a blanket command for esxcfg-swiscsi to renable?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
have you set the MTU of 9000 on the HBAs?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
esxcfg-swiscsi -e
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Okay, I don't think you are going to like this much, I don't think the Dependent iSCSI Adapter BCM5709/BCM57711 supports Jumbo Frames.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
abssupportAuthor Commented:
OK The Hosts are back.  The software initiator is showing as enabled and discovered hosts are showing.  Can't see where I enable the Jumbo frames on the HBAs....not in the GUI anyhow

esxcfg-vmknic -l still shows MTU 9000 if that's what you mean..
abssupportAuthor Commented:
I can accept that.  Shall I roll back the config back to MTU 1500?  Crazy though if it's true....
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
No we've done lots of work in this area, and I was only citing a reference to this yesterday on EE.

We have found that the Software iSCSI Initiator with Jumbo Frames enabled, was faster in performance, than Hardware iSCSI Initiators without Jumbo Frames (only because most HBAs we've used don't support Jumbo Frames).

Latency!

I'll try and find the EE link....

abssupportAuthor Commented:
Officially they do but maybe this is a setting in the card BIOS?  It's selectable from Windows apparently.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
you have a choice

1. Rollback (reset back to MTU 1500) to Hardware Initators as before.

2. Enable Software init, enable Jumbo Frames, and ditch BCM5709/BCM57711 HBAs and use Software Initiator only.

I think at this stage to reduce risk Rollback, and Test 2 above some other time.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
you could just quickly see here:-

esxcli swiscsi vmnic list -d vmhba34
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
esxcli swiscsi nic list -d vmhba34

to show bound nics
abssupportAuthor Commented:
vmnic5
    vmnic name: vmnic5
    mac address: d4:85:64:57:65:f2
    mac address settable: NO
    maximum transfer rate: 1000
    current transfer rate: 1000
    maximum frame size: 9000
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
http://www.experts-exchange.com/Software/VMWare/Q_26730342.html

this was the link, very last comment but one.
abssupportAuthor Commented:
Going to roll back....still concerned about that missing missing vmkiscsiconf file though.  Is this generated upon configuration?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
well the MTU is there 9000.

Okay, I've just found this

http://communities.vmware.com/message/1670737

this confirms that The Broadcom 5709 in 4.1 does not support Jumbo Frames AT THIS TIME.

29 Dec 2010.

As you are on ESX 4, this confirms my suspicions, this post also strengthens our findings:-

"I recommend using the SW initiator w/ Jumbo frames as performance will be better then HW anyway....according to many tests done by others on blogs the performance with HW offload is not enough to AT THIS TIME to use it."

Sorry you had a wasted night.

1. Rollback MTU to 1500.

2. Use Software Init Only at 9000.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
yes, it's generated, after config usually.
abssupportAuthor Commented:
Got it all back and running under the original settings.  I *may* try again on one host in the future.
I did notice some strange behaviour connecting.  Proceedure I followed was remove mappings, deleted chap on the HBA and SAN.  Forced rescan.  You can only add the storage after the CHAP has been reconfigured fresh.

Cheers Hancocka for your out of hours help, you deserve all the points and more!
abssupportAuthor Commented:
Great help
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Many Thanks
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.