Solved

How do I get maximum bandwith on 4 1GBe ports with windows 2012 r2 nic teaming and HP Procurve switch

Posted on 2015-01-01
14
857 Views
Last Modified: 2015-01-07
I have a HP Procurve 5406zl connected to two Windows 2012 R2 Servers (IBM 3650M4) with four 1 GBe ports each. One is a database server, the other is a Backup Server that should write data to a tape.

I'm trying to set up trunking with the Switch such that Maximum bandwidth (~4GB) can be used to send data from the db Server to the backup Server/tape. Unfortunatly no matter how I set up teaming and trunking, I do get only a Maximum bandwith of 1 GB for the Connection between both Servers.

I tried all kinds of combinations on the Windows side (Switch Independent teaming, Switch dependent teaming LACP and static) and on the Procurve side (LACP dynamic trunk, LACP static trunk and static trunk).

From what I read from the Microsoft documentation, the Windows side should be able to give me 4 GB/s for the outgoing Connections, however for the incoming Connection the Switch is responsible for the proper load balancing. I'm using a ProCurve 5406zl,  sw Version K.15.10.0009, ROM K.15.30. I understand that there are newer Versions avalaible but does this help with my Problem ?

I tested I/O Performance with file copy operations and IOmeter Network tests between the two machines.
0
Comment
Question by:PeterSausD
14 Comments
 
LVL 16

Expert Comment

by:Emmanuel Adebayo
ID: 40526881
Hi,

The four NICs need to be set up on Standby/Standby mode,? Are you getting 4GB from the server output?

Regards
0
 

Author Comment

by:PeterSausD
ID: 40526913
I don't understand what you mean by "Standby/Standyby mode". The Switch is configured LACP passive and the Windows 2012 R2 teaming is always LACP active if I understand it correctly.

I get 2 GBit throughput on the Server if I connect with two different Clients.
0
 
LVL 56

Expert Comment

by:Cliff Galiher
ID: 40527066
You'll need switch dependent teams on both machines (not just the backup server) using LACP. Otherwise the switch will not handle traffic properly. If you are still having issues, please post your switch config.
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40527086
I'm trying to set up trunking with the Switch such that Maximum bandwidth (~4GB) can be used to send data from the db Server to the backup Server/tape.
NIC teaming isn't going to help (that much).  While NIC Teaming (or Link Aggregation, Port Groups, etc.) distributes traffic over multiple links, the distribution is done on a per-flow basis.  Which means that all the traffic from a single flow will still cross only one link. Since the traffic from the DB server to the backup server will be a single flow, you will not see any improvement in the maximum data rate because it will still cross only one of the 1g links.  

However, if there is other traffic going to/from the DB server or backup server, it may be sent over the other links making more of the bandwidth available for your backup.
0
 

Author Comment

by:PeterSausD
ID: 40527405
NIC teaming isn't going to help (that much).  While NIC Teaming (or Link Aggregation, Port Groups, etc.) distributes traffic over multiple links, the distribution is done on a per-flow basis.  Which means that all the traffic from a single flow will still cross only one link. Since the traffic from the DB server to the backup server will be a single flow, you will not see any improvement in the maximum data rate because it will still cross only one of the 1g links.  

The Microsoft paper "Windows Server 2012 R2 NIC Teaming (LBFO) Deployment and Management" clearly states:

3.2      Configurations for NIC Teaming
...
•      Switch-dependent teaming. This configuration that requires the switch to participate in the teaming. Switch dependent teaming requires all the members of the team to be connected to the same physical switch.  There are two modes of operation for switch-dependent teaming:  
o      Generic or static teaming (IEEE 802.3ad draft v1). This mode requires configuration on both the switch and the host to identify which links form the team. Since this is a statically configured solution there is no additional protocol to assist the switch and the host to identify incorrectly plugged cables or other errors that could cause the team to fail to perform. This mode is typically supported by server-class switches.
o      Link Aggregation Control Protocol teaming (IEEE 802.1ax, LACP). This mode is also commonly referred to as IEEE 802.3ad as it was developed in the IEEE 802.3ad committee before being published as IEEE 802.1ax.  IEEE 802.1ax works by using the Link Aggregation Control Protocol (LACP) to dynamically identify links that are connected between the host and a given switch. This enables the automatic creation of a team and, in theory but rarely in practice, the expansion and reduction of a team simply by the transmission or receipt of LACP packets from the peer entity. Typical server-class switches support IEEE 802.1ax but most require the network operator to administratively enable LACP on the port.     Windows NIC Teaming always operates in LACP’s Active mode with a short timer.  No option is presently available to modify the timer or change the LACP mode.
Both of these modes allow both inbound and outbound traffic to approach the practical limits of the aggregated bandwidth because the pool of team members is seen as a single pipe.
Because the switch determines the inbound load distribution it is important to research what options may be available for inbound load distribution management.  Many switches only use destination IP address to team member mapping which may result in a less granular distribution than is needed to get good inbound load distribution.  Since this guide can’t cover all the settings on all switches it remains an exercise for the reader to understand the capabilities of the adjacent network switches.

0
 

Author Comment

by:PeterSausD
ID: 40527409
You'll need switch dependent teams on both machines (not just the backup server) using LACP. Otherwise the switch will not handle traffic properly. If you are still having issues, please post your switch config.

I configured both machines on the same Switch with the same Settings.

Here is the current Switch config, wbackup is the backup Server, DW1 the DB server:

; J8697A Configuration Editor; Created on release #K.15.10.0009
; Ver #03:03.1f.ef:f0
hostname "HP-5406zl"
module 1 type j8705a
module 2 type j8705a
module 3 type j9637a
module 4 type j9550a
fault-finder bad-driver sensitivity high
fault-finder bad-transceiver sensitivity high
fault-finder bad-cable sensitivity high
fault-finder too-long-cable sensitivity high
fault-finder over-bandwidth sensitivity high
fault-finder broadcast-storm sensitivity high
fault-finder loss-of-link sensitivity high
fault-finder duplex-mismatch-hdx sensitivity high
fault-finder duplex-mismatch-fdx sensitivity high
fault-finder link-flap sensitivity high
power-over-ethernet pre-std-detect
timesync sntp
sntp unicast
sntp server priority 1 192.168.101.2
ip default-gateway 192.168.101.8
ip timep manual 192.168.101.2
ip route 192.168.140.0 255.255.255.0 192.168.101.10
ip routing
interface C1
   flow-control
   lacp passive
   name "wbackup"
   exit
interface C2
   flow-control
   lacp passive
   name "wbackup"
   exit
interface C3
   flow-control
   lacp passive
   name "wbackup"
   exit
interface C4
   flow-control
   lacp passive
   name "wbackup"
   exit
interface D21
   flow-control
   lacp passive
   name "DW1"
   exit
interface D22
   flow-control
   lacp passive
   name "DW1"
   exit
interface D23
   flow-control
   lacp passive
   name "DW1"
   exit
interface D24
   flow-control
   lacp passive
   name "DW1"
   exit
spanning-tree
spanning-tree priority 0
no autorun
no dhcp config-file-update
no dhcp image-file-update    

Open in new window


"Show lacp" Displays this:

                                   LACP

          LACP      Trunk     Port                LACP      Admin   Oper
   Port   Enabled   Group     Status    Partner   Status    Key     Key
   ----   -------   -------   -------   -------   -------   ------  ------
   C1     Passive   Dyn1      Up        Yes       Success   0        0
   C2     Passive   Dyn1      Up        Yes       Success   0        0
   C3     Passive   Dyn1      Up        Yes       Success   0        0
   C4     Passive   Dyn1      Up        Yes       Success   0        0
   D21    Passive   Dyn2      Up        Yes       Success   0        0
   D22    Passive   Dyn2      Up        Yes       Success   0        0
   D23    Passive   Dyn2      Up        Yes       Success   0        0
   D24    Passive   Dyn2      Up        Yes       Success   0        0

Open in new window

0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40527783
I appreciate your confusion.  I run into this all the time.

I think the issue arises because with aggregating links, it's assumed that there will be many flows.  In other words, lots of devices on one side sends/receiving traffic to one or more devices on the other side.

In that scenario, aggregation works very well.  The traffic is distributed across all the links (hopefully equally, but that's not assured). So if you have a number of devices sending a total of 3gbp/s of traffic across four 1gbp/s aggregated links, you may see 3gbp/s of total traffic going over those links.  

But traffic of a single flow will only cross one of the links... period. Which means that for a flow of traffic, you will never see more bandwidth than a single link.

To address the Microsoft paper:

Both of these modes allow both inbound and outbound traffic to approach the practical limits of the aggregated bandwidth because the pool of team members is seen as a single pipe.
This is true... assuming multiple flows.  
Because the switch determines the inbound load distribution it is important to research what options may be available for inbound load distribution management.
I'm happy to see this. Too many documents use the term balance when discussing load distribution which leads to the conclusion that there will be an equal distribution of traffic over the physical links. There isn't.

Many switches only use destination IP address to team member mapping which may result in a less granular distribution than is needed to get good inbound load distribution.
And here it the crux of the matter.  Lets assume an IP load distribution algorithm.  A hash is performed against the source, destination or both IP addresses.  This will result in a link ID.  The link ID is the link that all traffic which matches this flow will be sent over.  If you have three other links, it's not going to help this flow. But for other flows, hopefully they will be sent over one of those links.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:PeterSausD
ID: 40528283
I appreciate your confusion.  I run into this all the time.

Maybe I'm not confused at all but just reading the MS doc differently than you and maybe you have didn't explore all cases.

Fact is I connected the two machines also for a test with two crossover cables and created a "cross_team" nic teaming on both machines, LACP/dynamic or LACP/Addr-hash.

If I start IOmeter with one or two Network workers, the bandwidth keeps below 1 GB but as soon as I start three or more workers, the bandwidth goes beyond 1 GB nearly 1.9 GB.

If I start four cmd.exe Shells and copy large files in each of them to the other machine I also get about 1.8 to 1.9 GB/s throughput.

This would be good for my database backups as the could be streamed from several Agent Jobs to the backup machine at the same time.

However the Goal would be to set up the Procurve Switch to make a similar smart load Distribution that obviously Windows 2012 R2 is doing.
wbackup---192.168.101.0---Remotedesktopv
0
 
LVL 50

Accepted Solution

by:
Don Johnston earned 500 total points
ID: 40528430
Maybe I'm not confused at all but just reading the MS doc differently than you and maybe you have didn't explore all cases.

I apologize if you feel I'm implying that you are confused. I guess what I meant to express is that the documentation does a poor job of explaining how link aggregation operates.

If I start IOmeter with one or two Network workers, the bandwidth keeps below 1 GB but as soon as I start three or more workers, the bandwidth goes beyond 1 GB nearly 1.9 GB.

If you are using a L4 based load distribution algorithm, then the "flow" would be defined by the layer 4 (TCP or UDP) port numbers which are used by the applications.  So if you have multiple processes (what I assume is a "network worker") and each process is sending/receiving traffic, then the traffic from each flow could be sent over different links.

If I start four cmd.exe Shells and copy large files in each of them to the other machine I also get about 1.8 to 1.9 GB/s throughput.
Correct.  If you using L4 load distribution then the traffic generated by each cmd.exe shell would be a different flow and will be sent over a specific physical link.

This would be good for my database backups as the could be streamed from several Agent Jobs to the backup machine at the same time.
If the backups are being done from different processes then this could work. But only If all parties (servers and intermediate switches) supported a L4 load distribution algorithm.  I can't speak to what your servers can do, but the HP Procurve 5406zl only use L3 distribution.   So since these processes will be going to/from the same address pair, the traffic will only use one link. If we were talking about Cisco switches, once you get to the 4500 series, they can do L4 load distribution.
0
 

Assisted Solution

by:PeterSausD
PeterSausD earned 0 total points
ID: 40528465
but the HP Procurve 5406zl only use L3 distribution

Good tip on the L4 distirbution: I was just in the process of updating the ProCurve software and the release notes state the L4 trunking is now supported. Also the following Wikipedia article mentions L4 trunking and Windows 2012 R2.
http://en.wikipedia.org/wiki/Link_aggregation

I will update the Switch this night and Report back.

This is from the "HP Switch Software Management and Configuration Guide K/KA/KB.15.16:

Enabling L4-based trunk load balancing
Enter the following command with the L4-based option to enable load balancing on Layer 4
information when it is present.
Syntax
trunk-load-balance [ L3-based | L4-based ]
When the L4-based option is configured, enables load balancing based on Layer 4 information
if it is present. If it is not present, Layer 3 information is used if present; if Layer 3 information is
not present, Layer 2 information is used. The configuration is executed in global configuration
context and applies to the entire switch.
Load balance on Layer 3 information if present, L3-based or Layer 2 information.
Load balance on Layer 4 port information if present, or Layer 3 if present, or Layer 2.
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40528469
Good to know.  

I was looking at older documentation.
0
 

Author Comment

by:PeterSausD
ID: 40528576
It works: after updating to K.15.16 and enabling L4 load balancing on the ProCurve, I now get throughput over 1 GBIt (currently only testing with two Nics).

I enable L4 LB with the command:

trunk-load-balance L4-based
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40528595
Nicely done.
0
 

Author Closing Comment

by:PeterSausD
ID: 40535221
FOund the L4 during updating the Switch but in parallel Don hinted on it also.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

This article will review the basic installation and configuration for Windows Software Update Services (WSUS) in a Windows 2012 R2 environment.  WSUS is a Microsoft tool that allows administrators to manage and control updates to be approved and ins…
OfficeMate Freezes on login or does not load after login credentials are input.
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles from a Windows Server 2008 domain controller to a Windows Server 2012 domain controlle…
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now