ESX 4.0 VSphere MPIO to Openfiler iSCSI LACP cannot get more than 110MB/sec

LACP does not provide link aggregation on the ESX unless you're using the a NEXUS 1000 Cisco virtual Cisco switch.  The only way to do this is via Cisco EtherChannel.  To setup MPIO correctly you'll also need to setup multipple VKernel ports as listed below


mpio p32-34

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf

I have read all those documents and threads extensively, and posted on a few.      My tests are certainly enough data;  i've got multiple VM's running disk benchmarks.  

the LACP is on the OF side, not on the VMware side.   I do in fact have Multiple VMK's, with different IP addresses.   I can see both  VMK's hitting the box at 50% of the 1GB link, rather than LACP spilling over an utilizing 2x1GB links on the OF / iSCSI side.

This is more of an "What am I missing in LACP conceptually" question, and than "OK so how do I fix this?"

We're using ESXi 4 and I went talked with Dell and VMware about this topic a few weeks ago.  I get different stories but we currently get max throughput equivalent to a single nic and I haven't tried these yet but they top my list for believable solutions.  Maybe you'll try them and save me the trouble?

1. The Round Robin is synchronous unless you have the Enterprise ESXi, so you'd get exactly the results I see.  Don't think I'll test this answer as we aren't upgrading at this point.

2. http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html  This one is great though it seems like it would be more widely known if true.

3. You have to dedicate 2 or more nics to a single VM that you want higher throughput on, and then use the Windows iScsi initiator in that VM set to round robin.  It will do async on the nics.  Takes more nics and harder to set up but this is helpful if you have say a single database server that needs higher throughput.

I am running Enterprise ESX 4.0, not ESXi.  

Read #2 already.  Still says it SHOULD work in 4.0, theoretically.

#3 does work, but some of my concern is the StorageMotion time to migrate, as well as hosting an Exchange server on the VM without tying it to the MS box.     Going from an U320 array to a 110meg iSCSI isn't going to cut it.  

Someone HAS to have done this before over MPIO iSCSI.  
   

I felt like I was the first person to worry about getting higher throughput on Gb iScsi.  The most believable support person assured me #1 and #3 work, although #3 does has several drawbacks.  You've done the flow control (enable) and if your san supports jumbo frames you've enabled that on the switch, the storage adapter, and the individual ports on the adapter?  And you set up multiple ports on your iScsi switch, which multiple vmnics on each port but only one is active and the rest are unused (on each port)?

Were you (or have you) going to try #2?

OK So I have more information now.  This is definately on the VMWARE side, because the OF box is now running 10GigE.  

I still can't push more than 110meg out of any 1 VM.    I've rechecked the VMK's go to individual adapters, the MPIO Round Robin  settings....etc.  

What the heck am I missing?

I'm upping the point value.