Solved

LACP Bandwidth Question

Posted on 2011-03-01
4
4,181 Views
Last Modified: 2012-05-11
I want to clarify one point having to do with Link Agregation (LACP) on a Cisco Switches.  Specifically I am working with a Cisco 6509 switch and 2960 switch.

I am testing throughput over an LACP Channel.  If I understand the Cisco implementation of LACP correctly, if I send data between two hosts on either side of the channel, the max bandwidth will only equal the bandwidth of an individual port.  I believe the reason for this is to prevent out of order packets.

So If I have four 100Mb ports set up with LACP for a LACP Channel of 400Mbps, the maximum throughput between two hosts send a stream of data will be equal to the bandwidth of one of the individual ports comprising the LACP channel (theoretically speaking, actual data throughput will be less).  This value would be 100 Mbps.  If I send data between two devices with Gigabit connections, though they can send up to 1 Gbps, I will only be able to achive a max throughput of 100 Mbps to to proprietary hashing algorithm used by Cisco.

In order to realize the bandwidth of LACP, you would have to have multiple data streams across the switches.  Multiple hosts sending data.  This just a test, but I am clarifying this point to other members of my staff that I report to.
0
Comment
Question by:Javier196
  • 2
4 Comments
 
LVL 17

Expert Comment

by:Kvistofta
Comment Utility
First of all: The reason you mention in the beginning has nothing to do with ordering of packets. LACP works by calculating a hash out of the sender and receiver mac-addresses and select a port member out of that. So any two hosts will always have their traffic sent in the same member of the channel, no matter of load.

If you have one host on the left side and 4 hosts on the right side, and a channel consisting of 4 channel members ("cables"), you could either have the 4 sessions distributed with one session per channel member, or two or more of the sessions in the same member. If you are really unlucky, all 4 sessions goes in the same channel member and the remaining 3 members are totally unused.

That being said, the best way to fully utilize an channel with equal distributed load is to have as many hosts on each side as possible. The fewer sessions, the more likely it is that these sessions are not equally spread among the members.

Best regards
Kvistofta
0
 
LVL 23

Accepted Solution

by:
Mysidia earned 500 total points
Comment Utility
"I am testing throughput over an LACP Channel.  If I understand the Cisco implementation of LACP correctly, if I send data between two hosts on either side of the channel, the max bandwidth will only equal the bandwidth of an individual port.  I believe the reason for this is to prevent out of order packets."

In some cases that will be true.  This depends on how you are balancing load on the Etherchannel.  the maximum bandwidth of any one ethernet link in the channel group is the speed of that ethernet link, however. LACP uses  load balancing.  Balancing of the load is what accomplishes bandwidth aggregation in an Etherchannel.

Every time a packet (frame) arrives at the switch that is to be sent across the Etherchannel;  one of the member links in that aggregation needs to be selected to transmit the frame over.

The way the decision is made (which link to send a packet over) is the  load balancing algorithm.

There are different load balancing algorithms available in  Cisco's  EtherChannel  implementation that have different consequences,
depending on which algorithm your platform has selected, see:

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml#matrix

Basically there is a  _deterministic_    pre-defined link that every packet will be transmitted over,  that just depends on the
number of links in the Etherchannel,  Source and Destination address, and sometimes Layer 4  port numbers.
There is literally an algorithm you could use to calculate which of the links every packet will be sent over.


So the aggregate links are shared, as in  traffic is divided over it,  BUT as far as each packet is concerned,  there is just one
link for it,  and there is no algorithm that will cause the packet to be sent over a link with less load.
If the link for that packet to be transmitted over is congested, the packet will be dropped, just as with any link.


If your platform is a 6500,  and your configured load balancing algorithm includes  Layer 2,3,4 information (MAC Addresses, IP addresses,
Port Numbers),  then frames that two different hosts exchange between each other that have different port number pairs  may travel across different links.


For example, host A may transmit to host B   on port 123  and host A may transmit to host B on port 124.

The packets from host A to host B  on different port numbers may select different links from the Etherchannel,
then  100mbps  would be "available"  to each of those connections,  on an Etherchannel built from 4 100mbps links.


On the other hand,  depending on the calculated value for the hash,  packets sent  might hash to the same link.


"So If I have four 100Mb ports set up with LACP for a LACP Channel of 400Mbps, the maximum throughput between two hosts send a stream of data will be equal to the bandwidth of one of the individual ports comprising the LACP channel (theoretically speaking, actual data throughput will be less).  This value would be 100 Mbps."

In most cases that will be true.   If Layer 4 information is not utilized in the selected load balancing,  OR if all data transfer between the two hosts has the same Layer 2 source and destination MAC addresses, Layer 3 source and dest MAC addresses, and Layer 4 source and dest port numbers,
then yes;  the packets should all transmit across the same link,  effectively limiting the max throughput to 100Mbps.

"If I send data between two devices with Gigabit connections, though they can send up to 1 Gbps, I will only be able to achive a max throughput of 100 Mbps to to proprietary hashing algorithm used by Cisco."

Packets will be dropped/discarded on output of the Etherchannel, if  you send a  1 Gbps  packet stream  with contents such that all those frames will go over the same link,  according to the  load balancing hash code.


"In order to realize the bandwidth of LACP, you would have to have multiple data streams across the switches.
Multiple hosts sending data.  This just a test, but I am clarifying this point to other members of my staff that I report to."


If layer 4 information is utilized, it is sufficient  to transmit to the other host over different port numbers,
for the packets to be sent down a different link.

This would not happen automatically, even if the port numbers were different,  you would have to plan it
(calculate which link traffic will go over, and choose source/dest host and source/dest ports to select a different
link for each 'connection'),  or use hundreds of data connections to achieve that through averaging.


If you just utilize a bunch of different random hosts with random port numbers, there
is no guarantee  each host/port number will hash to a different link in any particular
proportion;  you can have 10 hosts involved, and all data winds up going over the same link
of the Etherchannel aggregation.

There is a non-zero probability that would happen.


Etherchannel/LACP aggregation is often used primarily for redundancy, and secondarily for bandwidth aggregation.
However, a bandwidth aggregation improvement is only really going to be an  _average_ improvement when you have a massive
number of hosts/ports communicating to aggregate.

On average, the sum of all the data being transferred should evenly balance over the links, with
reduced congestion as compared to utilizing just one link for the purpose.

This is something you can calculate utilizing statistics.   With a small number of hosts, your actual observations are likely to deviate from the average;  for example,  the situation with 10 different hosts communicating, and all 10 connections happening to send packets over the same link.


These characteristics of Etherchannel can make troubleshooting harder, also, and when you are considering a small number of hosts /
data transfers,  there will likely always  be some statistical outliers,  where traffic is not evenly balanced.


For small numbers of hosts that need  >X mbps of bandwidth between any one pair of hosts,  you should select
an Etherchannel with   >X mbps  member links.

















0
 
LVL 17

Expert Comment

by:Kvistofta
Comment Utility
..which was exactly what I already wrote, but with fewer words. ;)

/Kvistofta
0
 
LVL 1

Author Closing Comment

by:Javier196
Comment Utility
Very thorough.  Good Detail
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
EXSi 6.x hosts on vCenter 5.5 7 78
Watchguard XTM 2 50
Setup ADSL modem with Router 7 41
Cisco layer 3 ring topology 1 51
Hello to you all, I hear of many people congratulate AWS (Amazon Web Services) on how easy it is to spin up and create new EC2 (Elastic Compute Cloud) instances, but then fail and struggle to connect to them using simple tools such as SSH (Secure…
Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now