Best practices for switching portion of iSCSI setup

Posted on 2013-01-14
Medium Priority
Last Modified: 2013-05-05
I do not normally touch network side and am struggling getting my head around exact requirements/preferences for iSCSI traffic. So, I thought I'd break down to see if I get a consensus. I have a single HP 2810 switch for the iSCSI traffic and have 3 Vmware hosts and a small SAN with 6 ports plugged into the switch. My issue is the Vmware side seems to disconnect from SAN and I have been going around with Vmware, SAN vendor and switch vendor. Hosts have multiple iSCSI NICs and vNIC configured round robin.

I have updated to most current firmware on both and Vmware 5.1. SAN vendor says that iSCSI session is dropping after 5 min or so. Basically Vmware performs slowly all day.

Spanning tree- I have heard both ways whether it should be enabled or not. Last version I read said enabled but set for edge on HP switch. Can't find exact guide on how to do this.

Flow control- I have not had this set and want to try and several references suggest.

Jumbo frames- I do not have configured across as I did not want to introduce new issues until I resolved original.

Any other suggestions or settings on the iSCSI switch side, let me know? I am attaching my current config on switch 1 and I am about to add sw2 so I don't have the multiple VLANs (so I will be transplanting the storage VLAN configuration).
Question by:smeek
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
LVL 57

Expert Comment

ID: 38776627
Is the iSCSI connection from host or the guest operating systems?

Since there is a single VLAN on the iSCSI ports spanning-tree should be disabled on those ports.

Is that file the whole configuration for the switch?  I have not used the HP 2810, but that config looks fairly sparse to me.

Do you have NIC's that are dedicated for iSCSI on the VMWare host?

I'm not sure using round robin for iSCSI is a good choice.

Author Comment

ID: 38776647
iSCSi from host.
HP config much shorter than comparable Cisco config.
Using Hardware iSCSI (4 NICs per host and Vmware verified all is correct on the Vmware side of things for iSCSI config).
LVL 57

Expert Comment

ID: 38776847
Found one  post:


That seem to indicate the 2810 does not have enough buffer memory to keep up under heavy load.  Are you stressing the system?
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Author Comment

ID: 38776867
My understanding was that buffer was only important during contention or in blocking switches.

Utilization is not high. HP support and I pulled logs and they said issue is not the switch, but of course, I am looking for second opinion.
LVL 57

Expert Comment

ID: 38777979
What iSCSI SAN is it?

If it is disconnecting that often you might be able to do a packet capture and see what is going on.  I am assuming that the HP Switch supports port mirroring.

Is the traffic low enough that you could disable/disconnect some of the NIC's?  That would make it easier to do the packet capture from the switch.

Author Comment

ID: 38778614
I appreciate your help trying to troubleshoot but my post was aimed at finding what the best practices were (enable or disable functionality). Do you have hands on knowledge of what functions should be enabled and why on my switch?
LVL 57

Accepted Solution

giltjr earned 750 total points
ID: 38778700
Best practices from a switch point of view:

1) Enable Jumbo Frames and use the biggest common frame size.
2) Enable QOS.  Especially important when traffic is flowing over any port using tagged VLAN's.
3) iSCSI traffic should be on isolated ports when possible.  That is do not use tagged interfaces and mix iSCSI and non-iSCSI traffic on the same port if at all possible.  If you do, see #2.
4) Spanning-tree should be disabled on any port that you know for sure is connected to a host.  

IIRC VMWare's vswitches will prevent loops and so any connection to a VMWare host should be treated as being connected to a host, not to another switch.  So you should disable spanning-tree on the ports connected to the VMWare hosts.
LVL 10

Assisted Solution

ddiazp earned 750 total points
ID: 38780672
The things i'd check that nobody has mentioned:

1. Cables - are the cables in good working condition? Are these the proper cables? Do interfaces on the switch flap? Can you check interface statistics and look for packet drops, packet errors?

2. Speed and duplex - what's configured on vmware hosts, switch and san? i'd hardcode them all.

3. If you're enabling jumbo frames, you must do it everywhere (switch, san and vmware hosts). Check MTU on all those 3, they should all be the same. Switch may need to be higher to allow for extra frame headers

4. Perhaps there's a duplicate IP address in use by some host somewhere? If a host is using one of the IPs of the SAN, you'd have similar issues

5. What's the path status of all 3 hosts? Go to Configuration>Storage, select each datastore, and look at the Paths for each one. They should be either Active, Active (I/O) or Standby. If you see 'Dead' then verify your vmkernel interfaces, IPs in use, subnets, etc.

Author Closing Comment

ID: 38794470
I will try verify all and follow general suggestions. Thanks for your feedback.

Someone else told me I need Cat 6 cables. The cables are newer Cat 5 cables but I think they should be adequate as we just changed them all out.
LVL 57

Expert Comment

ID: 38794505
To do GigE you need Cat 5e at a minimum.  In environments where you will really push GigE to its limits it is suggested you use Cat6.

Cat 6 is even certified for 10 G for short runs (20 meters).

Featured Post

Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
This article is in regards to the Cisco QSFP-4SFP10G-CU1M cables, which are designed to uplink/downlink 40GB ports to 10GB SFP ports. I recently experienced this and found very little configuration documentation on how these are supposed to be confi…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Suggested Courses

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question