VMWare ESX 4 hosts not seeing ISCSI storage - missing vmkiscsiconf file?

abssupport
abssupport used Ask the Experts™
on
Hi There,

Have a major issue seeing an existing iscsi san from a pair of ESX 4 hosts.
When I first set these up I had a fair bit of difficulty in getting it to connect but eventualy I did.
Now I have (though I should have done before, doh) enabled Jumbo frames using the advice i this article;

http://www.sysadmintutorials.com/tutorials/vmware-vsphere-4/esx-4/enabling-jumbo-frames-with-vmware-esx-4/

I have also enabled jumbo frames on the SAN an HP MSA2312i.
The switches are 3Com Baseline Switch 2928 unit which support jumbo frames.

OK, I have this additional issue with particular Broadcom nics in use in this installation that VMWARE are aware of;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025644

I have done as they have said and now the HBAs once again discover the iscsi SAN.

BUT try as I might I cannot rediscover the storage.  I have rebooted and no change.  I can vmkping the SAN addresses.

Some searching around has led me to a troubleshooting page;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083

My big concern here is that I don't have a /etc/vmkiscsi.conf file!  Could this be my issue?

This is a pretty bad situation as VMs are down and I need it to be up (obviously)

Anyone got any clue?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Yes, it's likely to be your issue if you've not got a /etc/vmkiscsi.conf file.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
I would disable iSCSI Software Initiator.

Restart ESX server.

RE-enable and fill in the IP address and details for you MSA.
Ernie BeekSenior infrastructure engineer
Top Expert 2012

Commented:
I remember there was something with iscsi and jumbo frames: http://mike.thegrahams.ws/?p=45

Perhaps it will lead you to the right direction.
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
This was all working when you enabled Jumbo Frames on you MSA and Switches?

Did you enable Jumbo Frames on the Servers?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
check my solutio here for enabling jumbo frames on your servers, (or check MTU is 9000)
Ernie BeekSenior infrastructure engineer
Top Expert 2012

Commented:
@hanccocka: do you ever sleep?

;)
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
What's sleep!!!

Author

Commented:
Quick reply guys, appreciated.  

It may sound dum but do I need the "ISCSI Software Adapter" enabled as well as the Software Initiator enabled that is listed as enabled on HBA's?

To answer your last post hancocka, this hasn't worked since reconfig.  I simply ran through the steps above getting back to where I could scan for storage.

The SAN is showing the HBA as discovered and the HBAs are showing the SAN as discovered.

Do you have a quick CLI command for disabling the Software Initiators?  They are greyed out in the GUI...
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Okay, you need to enable Jumbo Frames on the server!

Because you need to enable jumbo frames everywhere.

Q. Are you using Software of Hardware iSCSI Initiators?

Author

Commented:
MTU for switches output

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         3           128               9000    vmnic5

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS             0        1           vmnic5

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch3         128         3           128               9000    vmnic7

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMwareNFS2            0        1           vmnic7

Is this what you mean by enabling on the server?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
okay, that looks okay.

Q. Are you using Software of Hardware iSCSI Initiators?

Author

Commented:
OK the Broadcoms don't work as hardware initiator though they were listed that they should.
So the work around has been to set them as software initiators as per VMWare KB above..
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
esxcfg-swiscsi -d

Author

Commented:
I have to say, it was a bit of a pig to set up originally...when it used to work
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
okay, so you are ujsing Hardware iSCSI Initiators, so the iSCSI Software Stuff does not apply to this.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
did it used to work without jumbo frames?

Author

Commented:
OK done, rebooting hosts

Author

Commented:
No Software iSCSI

My issue was as below;

No storage devices visible with Broadcom iSCSI offload-enabled adapters
Symptoms
When using Broadcom network interfaces with Hardware iSCSI Offload capabilities such as the BCM5709 and the BCM57711, each of the network interfaces shows up as individual storage adapters in the Configuration tab of an ESX Host and each adapter also has an individual IQN name. However when you configure the adapter for iSCSI, you observe these symptoms:

    * You are also unable to add any discovery addresses
    * When observing the Dynamic or Static Discovery tabs in the iSCSI Initiator Properties window, the following message is displayed:

      The host bus adapter is not associated with a vmknic. To configure targets the adapter should be associated with a vmknic. Refer to the VMware documentation to associate the adapter with a vmknic.

    * Each one of the adaptors shows no associated storage devices

Resolution
Broadcom cards with iSCSI Offload capabilities are also Hardware iSCSI Initiators. To configure them for hardware iSCSI, perform one of these procedures:

Note: To log into the terminal for VMware ESXi hosts, see Using Tech Support Mode in ESXi 4.1 (1017910).

   1. Configure LUN Access, as any other hardware iSCSI HBA, within the BIOS of the Broadcom Card during system boot. It is recommended to ensure that your SAN is supported to use Broadcom as a hardware initiator before using this option.

      Or

   2. Add the Physical Network Interface as a binding to the Software iSCSI HBA Device by running the following commands. In this example, we are referencing the Software iSCSI HBA as vmhba32:

         1.
            List the currently-bound Network Interfaces to the Software iSCSI HBA by running the following command:

            # esxcli swiscsi vmnic list -d vmhba32

         2.
            List the current available network interfaces available on the ESX Host and identify the interface(s) you wish to bind:

            # esxcfg-vmknic -l

         3.
            Bind the network interface(s) identified in the previous step to the Software iSCSI HBA Device:

            # esxcli swiscsi nic add -n vmk0 -d vmhba32

         4.
            Verify the network interface(s) were successfully bound to the HBA:

            # esxcli swiscsi vmnic list -d vmhba32

         5.
            Enable the Software iSCSI Adapter in the Storage Adapters section of the Configuration tab.
         6.
            Enter the dynamic discovery addresses of the storage array.
         7. At the top-right, perform a Rescan of the Software iSCSI HBA.

      Note: You may repeat this process and create additional bindings. For additional information, see the below section.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
You may have to ENABLED jumbo frames on the Hardware Initiators?

I don't think these are vmnic 5 or vmnic 7 are they?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
yes, i'm getting there......

Author

Commented:
it used to work before jumbo frames BUT it didn't work the first time or the second or the third....you get the picture.  And then when I had reset all the CHAP setting on the SAN (which I tried this time) it did eventually work but I couldn't put my finger on why because it all looked good..

Author

Commented:
is there a blanket command for esxcfg-swiscsi to renable?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
have you set the MTU of 9000 on the HBAs?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
esxcfg-swiscsi -e
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
Okay, I don't think you are going to like this much, I don't think the Dependent iSCSI Adapter BCM5709/BCM57711 supports Jumbo Frames.

Author

Commented:
OK The Hosts are back.  The software initiator is showing as enabled and discovered hosts are showing.  Can't see where I enable the Jumbo frames on the HBAs....not in the GUI anyhow

esxcfg-vmknic -l still shows MTU 9000 if that's what you mean..

Author

Commented:
I can accept that.  Shall I roll back the config back to MTU 1500?  Crazy though if it's true....
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
No we've done lots of work in this area, and I was only citing a reference to this yesterday on EE.

We have found that the Software iSCSI Initiator with Jumbo Frames enabled, was faster in performance, than Hardware iSCSI Initiators without Jumbo Frames (only because most HBAs we've used don't support Jumbo Frames).

Latency!

I'll try and find the EE link....

Author

Commented:
Officially they do but maybe this is a setting in the card BIOS?  It's selectable from Windows apparently.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
you have a choice

1. Rollback (reset back to MTU 1500) to Hardware Initators as before.

2. Enable Software init, enable Jumbo Frames, and ditch BCM5709/BCM57711 HBAs and use Software Initiator only.

I think at this stage to reduce risk Rollback, and Test 2 above some other time.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
you could just quickly see here:-

esxcli swiscsi vmnic list -d vmhba34
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
esxcli swiscsi nic list -d vmhba34

to show bound nics

Author

Commented:
vmnic5
    vmnic name: vmnic5
    mac address: d4:85:64:57:65:f2
    mac address settable: NO
    maximum transfer rate: 1000
    current transfer rate: 1000
    maximum frame size: 9000
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
http://www.experts-exchange.com/Software/VMWare/Q_26730342.html

this was the link, very last comment but one.

Author

Commented:
Going to roll back....still concerned about that missing missing vmkiscsiconf file though.  Is this generated upon configuration?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
well the MTU is there 9000.

Okay, I've just found this

http://communities.vmware.com/message/1670737

this confirms that The Broadcom 5709 in 4.1 does not support Jumbo Frames AT THIS TIME.

29 Dec 2010.

As you are on ESX 4, this confirms my suspicions, this post also strengthens our findings:-

"I recommend using the SW initiator w/ Jumbo frames as performance will be better then HW anyway....according to many tests done by others on blogs the performance with HW offload is not enough to AT THIS TIME to use it."

Sorry you had a wasted night.

1. Rollback MTU to 1500.

2. Use Software Init Only at 9000.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
yes, it's generated, after config usually.

Author

Commented:
Got it all back and running under the original settings.  I *may* try again on one host in the future.
I did notice some strange behaviour connecting.  Proceedure I followed was remove mappings, deleted chap on the HBA and SAN.  Forced rescan.  You can only add the storage after the CHAP has been reconfigured fresh.

Cheers Hancocka for your out of hours help, you deserve all the points and more!

Author

Commented:
Great help
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Many Thanks

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial