iscsi topology for best network throughput

Hi - I hope this is a fun question.  I have been having a hard time finding answers in white papers and on the web.

OK, Here is my situation and goals -
I want to be able to make sure the network is not the bottleneck in connections between multiple hosts and sans

- I have 3 VM esxi Hosts running a mix of servers and desktops.
- I have 3 datastores, all connected via iscsi  They all have 7200 rpm enterprise sas drives with ssd cache. In know not optimal, but we work with what we have available
- I am using 1 gigabit network connections using the software hba included with vmware and intel nics
- I want to connect each host to each datastore.
- It seems like 2 connections from each host to a data store for redundancy and round robin connections is the best answer, but I could be wrong.

- In your experience, what is the best iscsi topology to accomplish this? I want make the hard drives the bottleneck in the connections

assumed -
     I am using two switches
     I have enough network ports - 6 in each host, 6 in each datastore

All ideas and insights are appreciated. I look forward to your comments and questions.

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

I'd use 2 nics per server for iSCSI connectivity with jumbo frames 9000MTU remember to set switch port MTU size as well :-)

2 nics for vMotion

2 nics for VM traffic

identical config on the switches and dedicated vLANs for my nics, if you want to use LACP and ether-channel on the switches that can also be cool, please consult the documentation on limitations to VMware features depending on version.

So you should have initiators on the storage controllers spread evenly across 2 switches to 2 initiators per server.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
here are my EE Articles, and this is Best Practice on how you setup iSCSI

HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client

HOW TO: Add an iSCSI Software Adaptor and Create an iSCSI Multipath Network in VMware vSphere Hypervisor ESXi 5.0

So you've already configured 2 nics for iSCSI Storage Network, which leaves 4 nics, if you want resilienace, Team 2 nics for

Virtual Machine Network

vMotion Network

or you could use VLANs, and trunk 4 nics.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dpedersen13Author Commented:
Hi dipopo and hanccoka -
Thanks for your comments and advice.  The tutorial is fantastic, which I'd found it a couple of weeks ago.

I'll share a bit more as to where I am -
I'm not ready for vmotion and i've got a couple more nics on the hosts for VM network traffic.  I'm looking at the 6 nics per to handle iscsi only

I have the first SAN running wire speed to one host.  using iperf and iometer, I can saturate a single 1gb ethernet connection and use about 60% of 2 round robin.  To get there I have -
Updated all Nic drivers (intel only) on the esxi server (5.0.0) and the san windows 2012.  On the esxi I had to update the ibg drivers by installing the vib through esxcli.  I've got jumbo frames across the board and made sure the cabling is solid.

Your tutorial is spot on on the iscsi adapter configuration for the first san and the first host.  after bonding, I tried fixed and round robin, they both worked great.  So now I have redundancy for that connection and maybe some extra bandwidth.

OK, so now what I do with the next two hosts going to the same san?  I've found plenty materials on one host, one san, but next to nothing about a multiple host / san environment.
Do I just use the same two initiators and hope I don't saturate the 1 gb / round robin?  Do I configure 2 more initiator / nics on a different subnet in the same configuration to the second san?  Is it silly to worry about this / overkill?

Thanks again for the assistance, you are confirming I'm on the right path.
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would suggest you start with 2 nics for iSCSI, and see if you saturate 2 nics, 6 nics for iSCSI is a alot!

next two hosts do the same...
Do I just use the same two initiators and hope I don't saturate the 1 gb / round robin?  Do I configure 2 more initiator / nics on a different subnet in the same configuration to the second san?

I would not worry too much about saturating the storage connection especially since these are 1GB connections.

For multiple SAN environment you can have a many-to-many connection on the iSCSI targets.
dpedersen13Author Commented:
OK, so what I'm hearing is that 2 initiators per host connected to 6 targets shouldn't be a problem.

Same true on the Sans?  It would be the reverse, 2 targets for 6 initiators.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Should not be an issue.

Check performance and network utilisation as you rollout
dpedersen13Author Commented:
Thanks for all the help. I'll rollout with 2 nics on each device and see how the bandwidth / latency goes. Both answers compliment each other, thanks for the help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.