?
Solved

Network issues with Linux Bonding and Cisco 2960-S

Posted on 2013-11-12
9
Medium Priority
?
905 Views
Last Modified: 2016-11-23
Hi,

I have an environment that I am having some networking issues with.  Major packet loss and multiple mac addresses on arp replies.  My current thought is that it is related to configuration settings of the bonds and the configuration of the cisco switches.  Possibly an issue with vlans and/or etherchannel.  All of which I am unfamiliar with.

I'm sure you will want more information, but to start, please see the configs here:

Server (Debian GNU/Linux 6.0 \n \l):

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto bond0
iface bond0 inet static
        address 10.1.4.15
        netmask 255.255.0.0
        network 10.1.0.0
        broadcast 10.1.255.255
        gateway 10.1.1.1
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth4 eth5

auto bond1
iface bond1 inet manual
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth2 eth3


auto br0
iface br0 inet manual
        #bridge_ports bond1
        #bridge_stp off
        #bridge_fd 0
        #bridge_maxwait 0
        pre-up ip link set bond1 down
        pre-up brctl addbr br0
        pre-up brctl addif br0 bond1
        pre-up ip link set bond1 up
        pre-up ip link set br0 up
        post-down ip link set br0 down
        post-down brctl delbr br0


Cisco 2960-S Stack (four Cisco 2960S with fiber interconnection)

Current config:

***#sh conf
Using 6326 out of 524288 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ***
!
boot-start-marker
boot-end-marker
!
enable secret 5 ***
enable password ***
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
switch 3 provision ws-c2960s-48ts-l
!
!
!
!
crypto pki trustpoint TP-self-signed-1561874560
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1561874560
 revocation-check none
 rsakeypair TP-self-signed-1561874560
!
!
crypto pki certificate chain TP-self-signed-1561874560
 certificate self-signed 01 nvram:IOS-Self-Sig#3030.cer
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/0/25
!
interface GigabitEthernet1/0/26
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
!
interface GigabitEthernet1/0/29
!
interface GigabitEthernet1/0/30
!
interface GigabitEthernet1/0/31
!
interface GigabitEthernet1/0/32
!
interface GigabitEthernet1/0/33
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
!
interface GigabitEthernet1/0/36
!
interface GigabitEthernet1/0/37
!
interface GigabitEthernet1/0/38
!
interface GigabitEthernet1/0/39
!
interface GigabitEthernet1/0/40
!
interface GigabitEthernet1/0/41
!
interface GigabitEthernet1/0/42
!
interface GigabitEthernet1/0/43
!
interface GigabitEthernet1/0/44
!
interface GigabitEthernet1/0/45
!
interface GigabitEthernet1/0/46
!
interface GigabitEthernet1/0/47
!
interface GigabitEthernet1/0/48
!
interface GigabitEthernet1/0/49
!
interface GigabitEthernet1/0/50
!
interface GigabitEthernet1/0/51
!
interface GigabitEthernet1/0/52
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
!
interface GigabitEthernet2/0/15
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface GigabitEthernet2/0/19
!
interface GigabitEthernet2/0/20
!
interface GigabitEthernet2/0/21
!
interface GigabitEthernet2/0/22
!
interface GigabitEthernet2/0/23
!
interface GigabitEthernet2/0/24
!
interface GigabitEthernet2/0/25
!
interface GigabitEthernet2/0/26
!
interface GigabitEthernet2/0/27
!
interface GigabitEthernet2/0/28
!
interface GigabitEthernet2/0/29
!
interface GigabitEthernet2/0/30
!
interface GigabitEthernet2/0/31
!
interface GigabitEthernet2/0/32
!
interface GigabitEthernet2/0/33
!
interface GigabitEthernet2/0/34
!
interface GigabitEthernet2/0/35
!
interface GigabitEthernet2/0/36
!
interface GigabitEthernet2/0/37
!
interface GigabitEthernet2/0/38
!
interface GigabitEthernet2/0/39
!
interface GigabitEthernet2/0/40
!
interface GigabitEthernet2/0/41
!
interface GigabitEthernet2/0/42
!
interface GigabitEthernet2/0/43
!
interface GigabitEthernet2/0/44
!
interface GigabitEthernet2/0/45
!
interface GigabitEthernet2/0/46
!
interface GigabitEthernet2/0/47
!
interface GigabitEthernet2/0/48
!
interface GigabitEthernet2/0/49
!
interface GigabitEthernet2/0/50
!
interface GigabitEthernet2/0/51
!
interface GigabitEthernet2/0/52
!
interface GigabitEthernet3/0/1
!
interface GigabitEthernet3/0/2
!
interface GigabitEthernet3/0/3
!
interface GigabitEthernet3/0/4
!
interface GigabitEthernet3/0/5
!
interface GigabitEthernet3/0/6
!
interface GigabitEthernet3/0/7
!
interface GigabitEthernet3/0/8
!
interface GigabitEthernet3/0/9
!
interface GigabitEthernet3/0/10
!
interface GigabitEthernet3/0/11
!
interface GigabitEthernet3/0/12
!
interface GigabitEthernet3/0/13
!
interface GigabitEthernet3/0/14
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
!
interface GigabitEthernet3/0/18
!
interface GigabitEthernet3/0/19
!
interface GigabitEthernet3/0/20
!
interface GigabitEthernet3/0/21
!
interface GigabitEthernet3/0/22
!
interface GigabitEthernet3/0/23
!
interface GigabitEthernet3/0/24
!
interface GigabitEthernet3/0/25
!
interface GigabitEthernet3/0/26
!
interface GigabitEthernet3/0/27
!
interface GigabitEthernet3/0/28
!
interface GigabitEthernet3/0/29
!
interface GigabitEthernet3/0/30
!
interface GigabitEthernet3/0/31
!
interface GigabitEthernet3/0/32
!
interface GigabitEthernet3/0/33
!
interface GigabitEthernet3/0/34
!
interface GigabitEthernet3/0/35
!
interface GigabitEthernet3/0/36
!
interface GigabitEthernet3/0/37
!
interface GigabitEthernet3/0/38
!
interface GigabitEthernet3/0/39
!
interface GigabitEthernet3/0/40
!
interface GigabitEthernet3/0/41
!
interface GigabitEthernet3/0/42
!
interface GigabitEthernet3/0/43
!
interface GigabitEthernet3/0/44
!
interface GigabitEthernet3/0/45
!
interface GigabitEthernet3/0/46
!
interface GigabitEthernet3/0/47
!
interface GigabitEthernet3/0/48
!
interface GigabitEthernet3/0/49
!
interface GigabitEthernet3/0/50
!
interface GigabitEthernet3/0/51
!
interface GigabitEthernet3/0/52
!
interface Vlan1
 no ip address
 shutdown
!
ip http server
ip http secure-server
!
line con 0
line vty 0 4
 password ***
 login
line vty 5 15
 password ***
 login
!
end

Please let me know what other information you need.

The bond config was working in active/backup mode on some old dell switches at the old companies' location.


Thank you!
0
Comment
Question by:ar3
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
9 Comments
 

Author Comment

by:ar3
ID: 39643213
Example output from a server pinging the dns name of the server:

17:47:21.553136 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 00:10:**:**:**:80 (oui Unknown), length 46
17:47:21.553158 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 84:2b:**:**:**:ab (oui Unknown), length 46
0
 

Author Comment

by:ar3
ID: 39643215
The Cisco switch doesn't show mac flapping or drops.
0
 
LVL 46

Accepted Solution

by:
Craig Beck earned 2000 total points
ID: 39644405
You're using LACP on the servers, so you need to use LACP on the switches too.

On the stack master, enter the following configuration...

conf t
int po1
 switchport mode access
 no shutdown
!
int gigabitethernet 1/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 2/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/2
 switchport mode access
 channel-group 1 mode active
!
end

Open in new window


That will configure an EtherChannel using LACP on ports 1/0/1, 2/0/1, 3/0/1 and 3/0/2.  Substitute the ports numbers with the actual port numbers.
0
Need protection from advanced malware attacks?

Look no further than WatchGuard's Total Security Suite, providing defense in depth against today's most headlining attacks like Petya 2.0 and WannaCry. Keep your organization out of the news with protection from known and unknown threats.

 
LVL 40

Expert Comment

by:noci
ID: 39644455
In my experience LACP might fail under heavy load. I have seen it break on a heavily used iSCSI channel.
So if you known the links before then IMHO the prefered setting is a static one.
(Missing interface will be spotted & removed anyway).

The automatic features are only handy for temporary equipment OR placing new systems while not managing your environment for those systems.
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 39644478
@noci, I would use multipath for iSCSi rather than LACP...
0
 
LVL 46

Expert Comment

by:Craig Beck
ID: 39644504
I would use multipath too... but the OP states LACP is being used, so that's what I've gone with :-)
0
 
LVL 40

Expert Comment

by:noci
ID: 39644522
That would help if both units had 2 interfaces. otherwise it will flood one of them...
redundancy was the first requirement,  bandwidth the 2nd, Using 2 storage units in mirror as backend... on a server with multiple cards.

(The discussion was about bonding / LACP not my iSCSI setup..  ;-)
0
 

Author Comment

by:ar3
ID: 39644738
Would a separate etherchannel group need to be configured for each bonded set of interfaces?
0
 
LVL 46

Expert Comment

by:Craig Beck
ID: 39644863
It depends how it is configured.  If it's all 4 in the same bond, then no.  If it's two bonded pairs added to a bridge then yes you will probably need two separate Etherchannels.  However it might not even work properly unless the OS knows to keep the passive bond disabled until it's required.
0

Featured Post

Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Do you have a computer or other electronic gear that is attached to a rat nest of cables, or alternatively have your cables all bundled nice at neat?  If so then read this post to sidstep common pitfalls. When I was a student at DeVry University,…
This article is a how to to configure a UCS Ethernet-uplink portchannel via the console. It is easy to do and can be done quite quickly. In certain versions of the UCS manager the portchannel has issues coming up and this is a workaround. I am…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses
Course of the Month10 days, 5 hours left to enroll

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question