Best practices for switching portion of iSCSI setup

Posted on 2013-01-14
Last Modified: 2013-05-05
I do not normally touch network side and am struggling getting my head around exact requirements/preferences for iSCSI traffic. So, I thought I'd break down to see if I get a consensus. I have a single HP 2810 switch for the iSCSI traffic and have 3 Vmware hosts and a small SAN with 6 ports plugged into the switch. My issue is the Vmware side seems to disconnect from SAN and I have been going around with Vmware, SAN vendor and switch vendor. Hosts have multiple iSCSI NICs and vNIC configured round robin.

I have updated to most current firmware on both and Vmware 5.1. SAN vendor says that iSCSI session is dropping after 5 min or so. Basically Vmware performs slowly all day.

Spanning tree- I have heard both ways whether it should be enabled or not. Last version I read said enabled but set for edge on HP switch. Can't find exact guide on how to do this.

Flow control- I have not had this set and want to try and several references suggest.

Jumbo frames- I do not have configured across as I did not want to introduce new issues until I resolved original.

Any other suggestions or settings on the iSCSI switch side, let me know? I am attaching my current config on switch 1 and I am about to add sw2 so I don't have the multiple VLANs (so I will be transplanting the storage VLAN configuration).
Question by:smeek
  • 5
  • 4
LVL 57

Expert Comment

ID: 38776627
Is the iSCSI connection from host or the guest operating systems?

Since there is a single VLAN on the iSCSI ports spanning-tree should be disabled on those ports.

Is that file the whole configuration for the switch?  I have not used the HP 2810, but that config looks fairly sparse to me.

Do you have NIC's that are dedicated for iSCSI on the VMWare host?

I'm not sure using round robin for iSCSI is a good choice.

Author Comment

ID: 38776647
iSCSi from host.
HP config much shorter than comparable Cisco config.
Using Hardware iSCSI (4 NICs per host and Vmware verified all is correct on the Vmware side of things for iSCSI config).
LVL 57

Expert Comment

ID: 38776847
Found one  post:

That seem to indicate the 2810 does not have enough buffer memory to keep up under heavy load.  Are you stressing the system?

Author Comment

ID: 38776867
My understanding was that buffer was only important during contention or in blocking switches.

Utilization is not high. HP support and I pulled logs and they said issue is not the switch, but of course, I am looking for second opinion.
LVL 57

Expert Comment

ID: 38777979
What iSCSI SAN is it?

If it is disconnecting that often you might be able to do a packet capture and see what is going on.  I am assuming that the HP Switch supports port mirroring.

Is the traffic low enough that you could disable/disconnect some of the NIC's?  That would make it easier to do the packet capture from the switch.
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline


Author Comment

ID: 38778614
I appreciate your help trying to troubleshoot but my post was aimed at finding what the best practices were (enable or disable functionality). Do you have hands on knowledge of what functions should be enabled and why on my switch?
LVL 57

Accepted Solution

giltjr earned 250 total points
ID: 38778700
Best practices from a switch point of view:

1) Enable Jumbo Frames and use the biggest common frame size.
2) Enable QOS.  Especially important when traffic is flowing over any port using tagged VLAN's.
3) iSCSI traffic should be on isolated ports when possible.  That is do not use tagged interfaces and mix iSCSI and non-iSCSI traffic on the same port if at all possible.  If you do, see #2.
4) Spanning-tree should be disabled on any port that you know for sure is connected to a host.  

IIRC VMWare's vswitches will prevent loops and so any connection to a VMWare host should be treated as being connected to a host, not to another switch.  So you should disable spanning-tree on the ports connected to the VMWare hosts.
LVL 10

Assisted Solution

ddiazp earned 250 total points
ID: 38780672
The things i'd check that nobody has mentioned:

1. Cables - are the cables in good working condition? Are these the proper cables? Do interfaces on the switch flap? Can you check interface statistics and look for packet drops, packet errors?

2. Speed and duplex - what's configured on vmware hosts, switch and san? i'd hardcode them all.

3. If you're enabling jumbo frames, you must do it everywhere (switch, san and vmware hosts). Check MTU on all those 3, they should all be the same. Switch may need to be higher to allow for extra frame headers

4. Perhaps there's a duplicate IP address in use by some host somewhere? If a host is using one of the IPs of the SAN, you'd have similar issues

5. What's the path status of all 3 hosts? Go to Configuration>Storage, select each datastore, and look at the Paths for each one. They should be either Active, Active (I/O) or Standby. If you see 'Dead' then verify your vmkernel interfaces, IPs in use, subnets, etc.

Author Closing Comment

ID: 38794470
I will try verify all and follow general suggestions. Thanks for your feedback.

Someone else told me I need Cat 6 cables. The cables are newer Cat 5 cables but I think they should be adequate as we just changed them all out.
LVL 57

Expert Comment

ID: 38794505
To do GigE you need Cat 5e at a minimum.  In environments where you will really push GigE to its limits it is suggested you use Cat6.

Cat 6 is even certified for 10 G for short runs (20 meters).

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now