Link to home
Start Free TrialLog in
Avatar of WayToBlue
WayToBlue

asked on

Low throughput in one direction on gigabit ethernet

Hello, I am setting up a small network and trying to verify throughput levels between machines, but I am seeing some odd behavior.  Specifically, I am seeing slow throughput in one direction between two machines, but not in the other.  I am using each of the machines built in NICs (same mobo for both, Intel D975XBX, also same CPU and ram), both of which are gigabit adapters.  Latency between the machines is < 1ms.  They are both attached to a Netgear GS724T switch by CAT5E cables, but I have rerun these tests with just a CAT6 crossover cable between them with no change in results, so the cables and the switch should be irrelevant.

The first machine is Windows XP SP2 with no firewall software of any kind - 192.168.1.32 (Bender)
The second machine is Fedora Core 5 with no firewall rules and defaults to ACCEPT - 192.168.1.33 (InfoSphere)

My results (using iperf) are as follows:

Bender receive TCP (8k buffer on Bender, 16k buffer on Infosphere)
[1844] local 192.168.1.32 port 5001 connected with 192.168.1.33 port 42011
[ ID] Interval       Transfer     Bandwidth
[1844]  0.0-60.0 sec  6.53 GBytes   935 Mbits/sec

Bender receive UDP (8k buffer on Bender, 124k buffer on Infosphere)
[1932] local 192.168.1.32 port 5001 connected with 192.168.1.33 port 32797
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[1932]  0.0-60.0 sec  6.57 GBytes   940 Mbits/sec  0.013 ms 29005/4825353 (0.6%)
[1932]  0.0-60.0 sec  1 datagrams received out-of-order

Infosphere receive TCP (8k buffer on Bender, 85.3k buffer on Infosphere)
[1916]  0.0-60.0 sec  2.04 GBytes   293 Mbits/sec

Infosphere receive UDP (8k buffer on Bender, 124k buffer on Infosphere)
[1916]  0.0-60.0 sec   338 MBytes  47.3 Mbits/sec  1.162 ms    0/241397 (0%)
[1916] Sent 241397 datagrams

I have repeated these tests multiple times with the same results, when data is sent from the InfoSphere to Bender I get very good (nearly the full 1000 Mbit) speeds, but from Bender to the InfoSphere is much slower, and even much slower still for UDP.

From Bender:
GlobalMaxTcpWindowSize: 400000
Tcp Receive Window: 400000
Window Scaling: Yes
Time Stamping: Yes
Selective Acks: Yes
MTU: 1500

From InfoSphere:
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.tcp_mem = 393216       524288  786432
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_fack = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.rmem_default = 126976
net.core.wmem_default = 126976
net.core.rmem_max = 108544
net.core.wmem_max = 108544
ASKER CERTIFIED SOLUTION
Avatar of SysExpert
SysExpert
Flag of Israel image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of WayToBlue
WayToBlue

ASKER

1) The NIC drivers are definitely not identical since they are for different OSs.

2) There is no malware on the system (and there is no odd network traffic during idle)

3) Possible, I certainly hope not, but possible

4) I was actually downloading knoppix while I wrote this question for that exact reason :)  I'll try it tonight and post back.
WaytoBlue:
What I see from here is: The transfer from Linux to Windows is Ok and the transfer from Windows to Linux is slow.

To me, is the driver. If the system reckoned the card when installed, you'll have to install it again, because, it could be an older driver. Install the latest drivers from the manufacturers of your G-card and check the card properties. If is not the drivers then, it has to be the configuration of the G-card.

After checking it, open Windows Task Manager (open it with CTRL-Alt-Del) and check the Performance (CPU usage) and Networking tabs.

Also, I assume that if there is a big difference in the Processor or the FSB of the MB on the Windows computer, comparing to the Linux one, it can't do the same throughput. (I'm guessing in the last one).
SysExpert:

OK, some updates.  I tried testing with knoppix running on Bender (the windows machine), throughput was great in both directions, ~ 950 Mbps UDP and ~930 Mbps TCP, so it's definitely OS related.

I booted back into windows and updated to the latest Intel drivers, my speeds got a little better, but still far too low (430 Mbps TCP, 220 Mbps UDP).  I tried playing with some of the NIC settings (receive/transmit descriptors, checksum offloading, flow control, etc...), no change.  I disabled every "item" attached to the "Local Area Connection" (i.e. Client for Microsoft Windows, File and Print Services, Network Monitor, VMWare bridging), everything got disabled except "TCP/IP."  No change.

I then rebooted into safe mode with all startup options off, all non-MS services disabled, and even proceeded to turn off every non-essential service.  No notable change.

So at this point I'm out of ideas, I ran HijackThis 2 Beta on my normal bootup, here's what I got:

Logfile of Trend Micro HijackThis v2.0.0 (BETA)
Scan saved at 1:35:38 AM, on 3/30/2007
Platform: Windows XP SP2 (WinNT 5.01.2600)
Boot mode: Normal

Running processes:
C:\WINDOWS\System32\smss.exe
C:\WINDOWS\system32\winlogon.exe
C:\WINDOWS\system32\services.exe
C:\WINDOWS\system32\lsass.exe
C:\WINDOWS\system32\svchost.exe
C:\WINDOWS\System32\svchost.exe
C:\WINDOWS\system32\spoolsv.exe
C:\Program Files\Common Files\Acronis\Schedule2\schedul2.exe
C:\Program Files\Intel\IDU\awServ.exe
C:\Program Files\Cisco Systems\VPN Client\cvpnd.exe
C:\Program Files\Intel\Intel Matrix Storage Manager\iaantmon.exe
C:\Program Files\McAfee\Managed VirusScan\Agent\myAgtSvc.exe
C:\WINDOWS\system32\nvsvc32.exe
C:\Program Files\VMware\VMware Workstation\vmware-authd.exe
C:\Program Files\Common Files\VMware\VMware Virtual Image Editing\vmount2.exe
C:\WINDOWS\system32\vmnat.exe
C:\WINDOWS\system32\vmnetdhcp.exe
C:\PROGRA~1\McAfee\MANAGE~1\VScan\McShield.exe
C:\WINDOWS\system32\wscntfy.exe
C:\WINDOWS\Explorer.EXE
C:\Program Files\Elaborate Bytes\VirtualCloneDrive\VCDDaemon.exe
C:\Program Files\Acronis\TrueImage\TrueImageMonitor.exe
C:\Program Files\Java\jre1.5.0_11\bin\jusched.exe
C:\Program Files\CyberLink\PowerDVD\PDVDServ.exe
C:\Program Files\McAfee\Managed VirusScan\Agent\myagttry.exe
C:\WINDOWS\System32\DeltTray.exe
C:\Program Files\Intel\IDU\iptray.exe
C:\Program Files\Adobe\Acrobat 8.0\Acrobat\Acrotray.exe
C:\Program Files\Eraser\eraser.exe
C:\Program Files\SlySoft\AnyDVD\AnyDVD.exe
C:\Program Files\Common Files\Macrovision Shared\FLEXnet Publisher\FNPLicensingService.exe
C:\Program Files\Mozilla Firefox\firefox.exe
C:\Documents and Settings\Alan\My Documents\My Downloads\HiJackThis_v2.exe

O2 - BHO: Adobe PDF Reader Link Helper - {06849E9F-C8D7-4D59-B87D-784B7D6BE0B3} - C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\AcroIEHelper.dll
O2 - BHO: SSVHelper Class - {761497BB-D6F0-462C-B6EB-D4DAF1D92D43} - C:\Program Files\Java\jre1.5.0_11\bin\ssv.dll
O2 - BHO: Adobe PDF Conversion Toolbar Helper - {AE7CD045-E861-484f-8273-0445EE161910} - C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll
O3 - Toolbar: Adobe PDF - {47833539-D0C5-4125-9FA8-0819E2EAAC93} - C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll
O4 - HKLM\..\Run: [VirtualCloneDrive] "C:\Program Files\Elaborate Bytes\VirtualCloneDrive\VCDDaemon.exe" /s
O4 - HKLM\..\Run: [TrueImageMonitor.exe] C:\Program Files\Acronis\TrueImage\TrueImageMonitor.exe
O4 - HKLM\..\Run: [SunJavaUpdateSched] "C:\Program Files\Java\jre1.5.0_11\bin\jusched.exe"
O4 - HKLM\..\Run: [RemoteControl] "C:\Program Files\CyberLink\PowerDVD\PDVDServ.exe"
O4 - HKLM\..\Run: [nwiz] nwiz.exe /install
O4 - HKLM\..\Run: [NvMediaCenter] RunDLL32.exe NvMCTray.dll,NvTaskbarInit
O4 - HKLM\..\Run: [NvCplDaemon] RUNDLL32.EXE C:\WINDOWS\system32\NvCpl.dll,NvStartup
O4 - HKLM\..\Run: [NeroFilterCheck] C:\WINDOWS\system32\NeroCheck.exe
O4 - HKLM\..\Run: [MVS Splash] C:\Program Files\McAfee\Managed VirusScan\Agent\Splash.exe
O4 - HKLM\..\Run: [McAfee Managed Services Tray] "C:\Program Files\McAfee\Managed VirusScan\Agent\myagttry.exe"
O4 - HKLM\..\Run: [M-Audio Delta Taskbar Icon] C:\WINDOWS\System32\DeltTray.exe
O4 - HKLM\..\Run: [ipTray.exe] "C:\Program Files\Intel\IDU\iptray.exe"
O4 - HKLM\..\Run: [DeltTray] DeltTray.exe
O4 - HKLM\..\Run: [Acrobat Assistant 8.0] "C:\Program Files\Adobe\Acrobat 8.0\Acrobat\Acrotray.exe"
O4 - HKCU\..\Run: [Vidalia] "C:\Program Files\Vidalia\vidalia.exe"
O4 - HKCU\..\Run: [Exodus] C:\Program Files\Exodus\Exodus.exe
O4 - HKCU\..\Run: [Eraser] C:\Program Files\Eraser\eraser.exe -hide
O4 - HKCU\..\Run: [AnyDVD] C:\Program Files\SlySoft\AnyDVD\AnyDVD.exe
O4 - Global Startup: Adobe Acrobat Speed Launcher.lnk = ?
O4 - Global Startup: Adobe Acrobat Synchronizer.lnk = C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AdobeCollabSync.exe
O4 - Global Startup: Adobe Gamma.lnk = C:\Program Files\Common Files\Adobe\Calibration\Adobe Gamma Loader.exe
O4 - Global Startup: Adobe Reader Speed Launch.lnk = C:\Program Files\Adobe\Acrobat 7.0\Reader\reader_sl.exe
O4 - Global Startup: Privoxy.lnk = C:\Program Files\Privoxy\privoxy.exe
O4 - Global Startup: VPN Client.lnk = ?
O8 - Extra context menu item: Append to existing PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIEAppend.html
O8 - Extra context menu item: Convert link target to Adobe PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIECapture.html
O8 - Extra context menu item: Convert link target to existing PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIEAppend.html
O8 - Extra context menu item: Convert selected links to Adobe PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIECaptureSelLinks.html
O8 - Extra context menu item: Convert selected links to existing PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIEAppendSelLinks.html
O8 - Extra context menu item: Convert selection to Adobe PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIECapture.html
O8 - Extra context menu item: Convert selection to existing PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIEAppend.html
O8 - Extra context menu item: Convert to Adobe PDF - res://C:\Program Files\Adobe\Acrobat 8.0\Acrobat\AcroIEFavClient.dll/AcroIECapture.html
O8 - Extra context menu item: E&xport to Microsoft Excel - res://C:\PROGRA~1\MICROS~2\OFFICE11\EXCEL.EXE/3000
O9 - Extra button: (no name) - {08B0E5C0-4FCB-11CF-AAA5-00401C608501} - C:\Program Files\Java\jre1.5.0_11\bin\ssv.dll
O9 - Extra 'Tools' menuitem: Sun Java Console - {08B0E5C0-4FCB-11CF-AAA5-00401C608501} - C:\Program Files\Java\jre1.5.0_11\bin\ssv.dll
O9 - Extra button: Research - {92780B25-18CC-41C8-B9BE-3C9C571A8263} - C:\PROGRA~1\MICROS~2\OFFICE11\REFIEBAR.DLL
O9 - Extra button: Messenger - {FB5F1910-F110-11d2-BB9E-00C04F795683} - C:\Program Files\Messenger\msmsgs.exe
O9 - Extra 'Tools' menuitem: Windows Messenger - {FB5F1910-F110-11d2-BB9E-00C04F795683} - C:\Program Files\Messenger\msmsgs.exe
O16 - DPF: {40C83AF8-FEA7-4A6A-A470-431EE84A0886} (SecureObjectFactory Class) - http://enu.vs.mcafeeasap.com/VS2/bin/myCioAgt.20051117222623.cab
O22 - SharedTaskScheduler: Browseui preloader - {438755C2-A8BA-11D1-B96B-00A0C90312E1} - C:\WINDOWS\system32\browseui.dll
O22 - SharedTaskScheduler: Component Categories cache daemon - {8C7461EF-2B13-11d2-BE35-3078302C2030} - C:\WINDOWS\system32\browseui.dll
O23 - Service: Acronis Scheduler2 Service (AcrSch2Svc) - Acronis - C:\Program Files\Common Files\Acronis\Schedule2\schedul2.exe
O23 - Service: Adobe LM Service - Adobe Systems - C:\Program Files\Common Files\Adobe Systems Shared\Service\Adobelmsvc.exe
O23 - Service: Admin Works Agent X8 (AWService) - OSA Technologies Inc., An Avocent Company - C:\Program Files\Intel\IDU\awServ.exe
O23 - Service: Cisco Systems, Inc. VPN Service (CVPND) - Cisco Systems, Inc. - C:\Program Files\Cisco Systems\VPN Client\cvpnd.exe
O23 - Service: FLEXnet Licensing Service - Macrovision Europe Ltd. - C:\Program Files\Common Files\Macrovision Shared\FLEXnet Publisher\FNPLicensingService.exe
O23 - Service: Intel(R) Matrix Storage Event Monitor (IAANTMon) - Intel Corporation - C:\Program Files\Intel\Intel Matrix Storage Manager\iaantmon.exe
O23 - Service: InstallDriver Table Manager (IDriverT) - Macrovision Corporation - C:\Program Files\Common Files\InstallShield\Driver\11\Intel 32\IDriverT.exe
O23 - Service: McShield - McAfee, Inc. - C:\PROGRA~1\McAfee\MANAGE~1\VScan\McShield.exe
O23 - Service: McAfee Virus and Spyware Protection Service (myAgtSvc) - McAfee, Inc. - C:\Program Files\McAfee\Managed VirusScan\Agent\myAgtSvc.exe
O23 - Service: NVIDIA Display Driver Service (NVSvc) - NVIDIA Corporation - C:\WINDOWS\system32\nvsvc32.exe
O23 - Service: Remote Packet Capture Protocol v.0 (experimental) (rpcapd) - CACE Technologies - C:\Program Files\WinPcap\rpcapd.exe
O23 - Service: VMware Authorization Service (VMAuthdService) - VMware, Inc. - C:\Program Files\VMware\VMware Workstation\vmware-authd.exe
O23 - Service: VMware DHCP Service (VMnetDHCP) - VMware, Inc. - C:\WINDOWS\system32\vmnetdhcp.exe
O23 - Service: VMware Virtual Mount Manager Extended (vmount2) - VMware, Inc. - C:\Program Files\Common Files\VMware\VMware Virtual Image Editing\vmount2.exe
O23 - Service: VMware NAT Service - VMware, Inc. - C:\WINDOWS\system32\vmnat.exe

--
End of file - 8580 bytes

I'm not seeing anything that shouldn't be there.

Any other ideas?


streaminglinux:

The CPU usage never goes above 20-30% or so when receiving at full speed, and is notably less than that when sending (albeit at these reduced rates).  The MB, CPU, and RAM of both machines is identical (as is FSB).
One more thing, I updated the BIOS to the latest for Bender to try to resolve this yesterday, so that's current as well.
OK.

Next things to check.

1) Safe mode with networking

2) goto dslreports.com and look at the utilities for speed tests including MTU and other stuff.

for a 1 Gbps connection, some internal TCPIP settings may need to be tweaked to get the best speed.

Otherwise it could be that windows drivers are just poorly written.



Link Speed/Duplex Mode is an option to modify in the Advanced Options tab, this is in the NIC's Properties, after you click on Configuration. You have to put this in 1000Mbps/Full Duplex instead of Auto Negotiation.

So right click G-Eth icon, ->Status ->Properties ->Configuration ->Advanced Options ->Link Speed/Duplex Mode, and change it to 1000 Mbps/Full Duplex
I want to know what happens,
Regards
SysExpert:

1) I tried safe mode with networking previously:

"I then rebooted into safe mode with all startup options off, all non-MS services disabled, and even proceeded to turn off every non-essential service.  No notable change."

2) I am using DrTCP from there, but none of that seems to be helping really (most of their stuff is meant to optimize a broadband connection, not GigE).  I am just using a normal 1500 MTU because linux says that my network card doesn't support jumbo frames (even though my windows driver will allow it, probably the windows driver is emulating support for it that the card itself doesn't have).  Considering how good the speeds are in the linux->windows direction with normal framing I'm not that upset about it.

streaminglinux:
Both cards are already in 1000 Mbps / Full Duplex (the cards both report that and the switch lights agree).


A bit of an update however, by playing with iperf variables I was able to achieve the following TCP results:

C:\iperf>iperf -c 192.168.1.33 -w 64k -l 256k -i 5 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.33, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[1916] local 192.168.1.32 port 1306 connected with 192.168.1.33 port 5001
[ ID] Interval       Transfer     Bandwidth
[1916]  0.0- 5.0 sec   560 MBytes   940 Mbits/sec
[1916]  5.0-10.0 sec   563 MBytes   944 Mbits/sec
[1916] 10.0-15.0 sec   562 MBytes   943 Mbits/sec
[1916] 15.0-20.0 sec   563 MBytes   944 Mbits/sec
[1916] 20.0-25.0 sec   563 MBytes   944 Mbits/sec
[1916] 25.0-30.0 sec   562 MBytes   943 Mbits/sec
[1916] 30.0-35.0 sec   562 MBytes   943 Mbits/sec
[1916] 35.0-40.0 sec   563 MBytes   944 Mbits/sec
[1916] 40.0-45.0 sec   563 MBytes   944 Mbits/sec
[1916] 45.0-50.0 sec   563 MBytes   944 Mbits/sec
[1916] 50.0-55.0 sec   563 MBytes   944 Mbits/sec
[1916] 55.0-60.0 sec   562 MBytes   943 Mbits/sec
[1916]  0.0-60.0 sec  6.59 GBytes   943 Mbits/sec

Full speed TCP send.  However UDP still seems maxed out just over 500:

C:\iperf>iperf -c 192.168.1.33 -u -b 1G -l 32k -i 1 -t 10
------------------------------------------------------------
Client connecting to 192.168.1.33, UDP port 5001
Sending 32768 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 192.168.1.32 port 1309 connected with 192.168.1.33 port 5001
[ ID] Interval       Transfer     Bandwidth
[1916]  0.0- 1.0 sec  60.7 MBytes   509 Mbits/sec
[1916]  1.0- 2.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  2.0- 3.0 sec  60.7 MBytes   509 Mbits/sec
[1916]  3.0- 4.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  4.0- 5.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  5.0- 6.0 sec  60.7 MBytes   509 Mbits/sec
[1916]  6.0- 7.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  7.0- 8.0 sec  60.7 MBytes   509 Mbits/sec
[1916]  8.0- 9.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  9.0-10.0 sec  60.6 MBytes   509 Mbits/sec
[1916]  0.0-10.0 sec   606 MBytes   508 Mbits/sec
[1916] Server Report:
[1916]  0.0-10.0 sec   606 MBytes   509 Mbits/sec  0.931 ms    1/19405 (0.0052%)
[1916] Sent 19405 datagrams

Changing the UDP window size did nothing, and if I increase -l to 64k+ for UDP I get:

C:\iperf>iperf -c 192.168.1.33 -u -b 1G -l 64k -i 1 -t 10
------------------------------------------------------------
Client connecting to 192.168.1.33, UDP port 5001
Sending 65536 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 192.168.1.32 port 1314 connected with 192.168.1.33 port 5001
write failed: Message too long
[ ID] Interval       Transfer     Bandwidth
[1916]  0.0- 0.0 sec  0.00 Bytes    -1 bits/sec
[1916] WARNING: did not receive ack of last datagram after 10 tries.
[1916] Sent 1 datagrams

So now I'm stuck again, which is frustrating because I'm so close now.
Ok, final update, everything is working.  I have been using iperf 2.0.2 on the linux side and iperf 1.7 on the windows side (newest binary they show), but since I was fresh out of other ideas I found a 3rd party compiled iperf 2.0.2 binary for windows (from the kperf package).  Using that, my UDP speeds are up where they should be:

C:\Program Files\iperf-2.0.2\bin>iperf.exe -c 192.168.1.33 -u -b 1G -w 1m -l 32k -i 5 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.33, UDP port 5001
Sending 32768 byte datagrams
UDP buffer size: 1.00 MByte
------------------------------------------------------------
[  3] local 192.168.1.32 port 1375 connected with 192.168.1.33 port 5001
[  3]  0.0- 5.0 sec    556 MBytes    932 Mbits/sec
[  3]  5.0-10.0 sec    561 MBytes    941 Mbits/sec
[  3] 10.0-15.0 sec    558 MBytes    936 Mbits/sec
[  3] 15.0-20.0 sec    560 MBytes    939 Mbits/sec
[  3] 20.0-25.0 sec    556 MBytes    932 Mbits/sec
[  3] 25.0-30.0 sec    555 MBytes    932 Mbits/sec
[  3] 30.0-35.0 sec    558 MBytes    937 Mbits/sec
[  3] 35.0-40.0 sec    558 MBytes    936 Mbits/sec
[  3] 40.0-45.0 sec    559 MBytes    938 Mbits/sec
[  3] 45.0-50.0 sec    557 MBytes    935 Mbits/sec
[  3] 50.0-55.0 sec    562 MBytes    943 Mbits/sec
[  3] 55.0-60.0 sec    558 MBytes    937 Mbits/sec
[  3]  0.0-60.0 sec  6.54 GBytes    937 Mbits/sec
[  3] Sent 214358 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  6.54 GBytes    936 Mbits/sec  0.466 ms    3/214357 (0.0014%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

So it seems my UDP issue was version based, not sure if using iperf 1.7 on the linux side would have worked or not, but at least it's working now.

I think the most useful step given to me in this was using knoppix to test it that way, and even though I was already doing that I didn't mention it so it was still a good suggestion, so credit to SysExpert.
Glad to hear this.

I woukd do one more test using a big file, just to make sure that everything is OK.

I ended up going with the following settings:

1M buffer for receive and send windows
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]
DWORD: DefaultReceiveWindow = 0x100000
DWORD: DefaultSendWindow = 0x100000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
DWORD: GlobalMaxTcpWindowSize = 0x100000
DWORD: TcpWindowSize = 0x100000
DWORD: SackOpts = 1
DWORD: Tcp1323Opts = 3

Here's the results of an FTP of a 700M file, big enough to get a reasonable feel for the network speeds:

ftp> get test.bin
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for test.bin (728945805 bytes).
226 File send OK.
ftp: 728945805 bytes received in 16.34Seconds 44602.94Kbytes/sec.
ftp> put test.bin
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 File receive OK.
ftp: 728945805 bytes sent in 6.47Seconds 112700.34Kbytes/sec.

The GET (from Infosphere to Bender) was 44602.94KBps => 357 Mbps
The PUT (from Bender to Infosphere) was 112700.34KBps => 902 Mbps

The PUT was great, especially considering that ftp only counts the data transferred, not the headers, which means it was sending even slightly faster than this.  The GET was slow, however I suspected this is because the local hard drive wasn't able to keep up.  Going with that assumption I created a 900MB ram disk (H:) and reran the test:

ftp> lcd h:\
Local directory now H:\.
ftp> get test.bin
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for test.bin (728945805 bytes).
226 File send OK.
ftp: 728945805 bytes received in 6.42Seconds 113507.60Kbytes/sec.
ftp> put test.bin
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 File receive OK.
ftp: 728945805 bytes sent in 6.27Seconds 116333.52Kbytes/sec.

This confirms that the hard drive was the limiting factor, as both GET and PUT are now > 900 Mbps.
Great testing !

Looks like everything is resolved.