Too many ISCSI Multipaths Slow Performance?


Design: Our SAN has 4 ISCSI NICS. Each ESX Host has 4 NICS dedicated to ISCSI connected to 1 vSwitch (screenshot attached). There is no routable network between the SAN and ESX Hosts, just one flat switch configured for Jumbo Frames and optimized for ISCSI traffic.

We are getting terrible latency and sometimes APD on our ESX Hosts when the SAN experiences high IOPS (from an overnight SAN to SAN replication). When our ESX hosts witness the latency, they curl up in a ball and die (APD).

I wonder whether our VMware ISCSI configuration may have something to do with it. (We are investing in more disks on the SAN to get better IOPS/lower latency).

By using a PSP of Round Robin, we end up with 16 possible paths to the SAN for each SAN volume (4 SAN NICS x 4 VMKNICS = 16 paths).

My Question:-

Should I be spreading the VMKNICS across perhaps 4 vswitches to better handle load balancing? Someone suggested this but I cant find any evidence to support it. Or should we reduce the default Round Robin IOPS from 1000 to a lower number, perhaps 3? (I cant find the recommended setting for a Compellent San but a Dell Equalogic is 3 IOPS). Or can you spot anything else?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Have you created your Multipath as follows:-

HOW TO: Add an iSCSI Software Adaptor and Create an iSCSI Multipath Network in VMware vSphere Hypervisor ESXi 5.0

also enable jumbo fames

HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client

What is the SAN, are you using their multi-pathing kit ?

Do you need 4 NICs for iSCSI, have you tried two ?
klwnAuthor Commented:
Thank you gentlemen.

You will see that my question is not so much about how to setup Multipathing and Jumbo frames. More however about optimally configuring iscsi multipathing across 4 vmknics to 4 SAN targets.

I need someone with knowledge around VMware PSP and adjusting Round Robing Default IOPS.

I want 4 vmknics from each host to give me throughput - 2 will not suffice.

PS. I have followed VMware best practice guides for "Running VMware on ISCSI" and "Multipathing Configuration for Software ISCSI using Port Binding"
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Have you checked with Dell if there is MULTIPATH MODULE for the SAN you are using?

I know there is for an Equallogic.

Are you sure 2 vmknics will not suffice, have you checked your throughput, and this is not just an assumption?

Have you reduced to 2 nics as a test, as clearly you are having issues with 4!

How many VMs, How many LUNs, How many datastores ?
klwnAuthor Commented:

Compellent Storage Centre Documentation

"If the Physical Network is comprised of a single subnet for ISCSI, then use a single  vSwitch with two ISCSI virtual ports with the traditional 1:1 mapping. Note that more ISCSI virtual ports can be use if the controller has more front end ports available".

This would suggest that our configuration of 4 VMKNICS on one vSwitch is supported.

Separate vSwitches are only required if the ISCSI comprises of multiple subnets.

Now we need to optimise the RR PSP Policy as I requested help on earlier
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I did not state it was not supported, I asked, if you used two vmknics, does the performance get better or worse.

Is there no Compellant Multi-Patch module?

All paths Active (IO)
klwnAuthor Commented:
Here is a look at one of our devices using

esxcli storage nmp device list

Open in new window

   Device Display Name: COMPELNT iSCSI Disk (naa.6000d31000123f00000000000000010                     f)
   Storage Array Type: VMW_SATP_DEFAULT_AA
   Storage Array Type Device Config: SATP VMW_SATP_DEFAULT_AA does not support d                     evice configuration.
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useA                     NO=0; lastPathIndex=9: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba37:C0:T3:L2, vmhba37:C2:T2:L2, vmhba37:C1:T2:L2, vmhba37:                     C3:T1:L2, vmhba37:C0:T2:L2, vmhba37:C2:T1:L2, vmhba37:C1:T1:L2, vmhba37:C3:T0:L2                     , vmhba37:C0:T1:L2, vmhba37:C2:T0:L2, vmhba37:C1:T0:L2, vmhba37:C0:T0:L2, vmhba3                     7:C3:T3:L2, vmhba37:C2:T3:L2, vmhba37:C1:T3:L2, vmhba37:C3:T2:L2
   Is Local SAS Device: false
   Is Boot USB Device: false

I have heard it suggested to adjust the number of bytes in the policy so a new path is chosen every time a jumbo frame is processed instead of the number of IOPS. i.e. bytes = 8800 (9000 minus overhead)

Any thoughts?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
which version of ESXi are you using ?
klwnAuthor Commented:
Sorry Andrew, just read your reply after I posted my last comment. The compellent is an Active-Active storage system where all paths are available all of the time (unless a path fails). I am waiting for a reply from their support but I do not know of any specific modules.

As the question ages, and I find out more information, I am looking how to optimally configure multipathing across 4 vmknics for a Dell Compellent.

4 NICS are required per host, believe me! 38 datastores, 123LUNS, 143vms, 3 DR hosts

ESXi 5.5 U1
klwnAuthor Commented:
making interesting reading, but not compellent specific:-
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
That's not a lot of VMs, and a small installation.

I've seen more VMs and clusters using only  two iSCSI VMKnics with jumbo frames, with no issues, and not saturating the 1GBe uplinks.

Round Robin should be the best path selection policy, if there is no Policy available from Dell, like there is for the Equallogic.

if you select Manage Paths in the vSphere Client for one of your LUNs, I assume it does state Active (IO) for all paths, and not just one path ?
klwnAuthor Commented:
Yes, all paths active as expected.

It is not so much the number of vm's that is the question here when looking at  whether 2 or 4 nics is required, more the IO being generated by the vms.

I haven't asked the question "are 2 ISCSI 1GB Nics enough", that's impossible to answer unless I provide IO statistics, regardless of the size of the environment.

I am looking for some advice on Compellent SAN, multipath. Has anyone had any luck getting an optimum setting by adjusting the IOPS or perhaps bytes limit on the PSP.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
To increase IOPS we change the RAID, add disks or SSDs, but recently we've been caching before the iSCSI datastore and SAN.
Jim_NimSenior EngineerCommented:
Changing the VMW_PSP_RR IOPS value to 3 would definitely be worth trying - this would help balance I/O more evenly across all the available paths.

You may also want to ensure that you have a "one-to-one" VMkernel port mapping, where each VMkernel port's failover order has all but 1 NIC set to "unused", and each VMkernel port correlates to a single physical NIC.

Have you contacted Compellent support to ask about this? Or checked their site for any documentation outlining the recommended configuration specifics?
klwnAuthor Commented:
Hi Jim,

Thanks for the comment. I'm interested where you get the figure "3" from? There is a lot of recommendations about settings IOPS to 1 or even 0 but another figure is normally recommended by the SAN vendor - are you referring to an EMC?

We do only have 1 active adapter for each vmknic, and that adapter is different for each of the 4 vmknics on our vswitch (vsphere 5 highlights a configuration issue if you do not correctly configure the correct active/unused adapters in multipathing)

I did just get a reply from Compellent:-

Our documentation (Dell Compellent Best Practices with VMware vSphere 5.X, technically states that we recommend the default round robin pathing policy, this would mean the 1000 IOPS presetup before swapping path.  With what previous customer experiences as well as online resources I have been able to find is that it is possible to see an increase in performance by adjusting the value to 1, but really only when we are looking at running a single VM as compared to multiple so I don’t think this would be necessary to adjust on your system.

Regarding the number of paths (16 per volume corresponding with the 4 server NICs and 4 Storage NICS), I wouldn’t see that path count being 16 as a problem unless you are concerned that with the MPIO policy we are not truly utilizing all 16 paths.  In that scenario we could certainly give it a try lowering the IOPS the MPIO policy uses before swapping paths, but we do not expect that it would give any benefit to performance.

I guess the only real way to test the IOPS =1 (or 3 as you suggest) would be to do an IO load test on the stack. I know for sure multipathing is working correctly from the values I see using ESXTOP (packets are evenly distributed across VMKNICS).
klwnAuthor Commented:
ESXi hosts using iSCSI/FC/FC/FCoE storage experiences latency issues with no signs of latency on the SAN side. Great little article...
klwnAuthor Commented:
Going to try the following to set all my compellent volumes and the default SATP rule to default to 1IOPS instead of 1000

esxcli storage nmp satp set -P VMW_PSP_RR -s VMW_SATP_DEFAULT_AA

esxcli storage nmp satp rule add -s VMW_SATP_DEFAULT_AA  -V "COMPELNT" -M "Compellent Vol" -P "VMW_PSP_RR" -O "iops=1"

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.6000d3`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

Open in new window

Where "naa.6000d3" identifies  our Compellent Volumes attached to our ISCSI Initiator

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jim_NimSenior EngineerCommented:
I guess the only real way to test the IOPS =1 (or 3 as you suggest) would be to do an IO load test on the stack
This is exactly what I intended to suggest by saying "would definitely be worth trying". My suggestion of 3 was based on experience with Equallogic iSCSI storage... but the best thing to do is test it out first. Run some benchmarks with an I/O footprint that best represents the production environment you're expecting, and go with the configuration that gives that testing the best performance. (You may also want to run other tests on the extremes of large sequential / small random I/O just to see how the changes affect these as well).
I'm surprised that Compellent doesn't have an official recommendation on this though. If they don't, it may very well have little to no impact on the performance you see with this storage.
klwnAuthor Commented:
Ok, Ill get Veeam up and running again using the new RR PSP IOP of 1 and monitor the latency on my vmknics. I'm guessing queue depths should certainly decrease if i'm switching paths more often. Nothing like testing with real data and scenarios.

In conclusion, Compellent recommend using the default of 1000 IOPS but also have no objection to playing with the IOP numbers on the policy.

Answered really by compellent but spreading the points and love for yours help.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
did it solve this:-

terrible latency and sometimes APD on our ESX Hosts when the SAN experiences high IOPS ???
klwnAuthor Commented:
No real documented advice given on IOP settings for a Dell Compellent. Points spread because I valued the input of the few guys who answered.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.