?
Solved

Network issues with Linux Bonding and Cisco 2960-S

Posted on 2013-11-12
9
Medium Priority
?
986 Views
Last Modified: 2016-11-23
Hi,

I have an environment that I am having some networking issues with.  Major packet loss and multiple mac addresses on arp replies.  My current thought is that it is related to configuration settings of the bonds and the configuration of the cisco switches.  Possibly an issue with vlans and/or etherchannel.  All of which I am unfamiliar with.

I'm sure you will want more information, but to start, please see the configs here:

Server (Debian GNU/Linux 6.0 \n \l):

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto bond0
iface bond0 inet static
        address 10.1.4.15
        netmask 255.255.0.0
        network 10.1.0.0
        broadcast 10.1.255.255
        gateway 10.1.1.1
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth4 eth5

auto bond1
iface bond1 inet manual
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth2 eth3


auto br0
iface br0 inet manual
        #bridge_ports bond1
        #bridge_stp off
        #bridge_fd 0
        #bridge_maxwait 0
        pre-up ip link set bond1 down
        pre-up brctl addbr br0
        pre-up brctl addif br0 bond1
        pre-up ip link set bond1 up
        pre-up ip link set br0 up
        post-down ip link set br0 down
        post-down brctl delbr br0


Cisco 2960-S Stack (four Cisco 2960S with fiber interconnection)

Current config:

***#sh conf
Using 6326 out of 524288 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ***
!
boot-start-marker
boot-end-marker
!
enable secret 5 ***
enable password ***
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
switch 3 provision ws-c2960s-48ts-l
!
!
!
!
crypto pki trustpoint TP-self-signed-1561874560
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1561874560
 revocation-check none
 rsakeypair TP-self-signed-1561874560
!
!
crypto pki certificate chain TP-self-signed-1561874560
 certificate self-signed 01 nvram:IOS-Self-Sig#3030.cer
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/0/25
!
interface GigabitEthernet1/0/26
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
!
interface GigabitEthernet1/0/29
!
interface GigabitEthernet1/0/30
!
interface GigabitEthernet1/0/31
!
interface GigabitEthernet1/0/32
!
interface GigabitEthernet1/0/33
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
!
interface GigabitEthernet1/0/36
!
interface GigabitEthernet1/0/37
!
interface GigabitEthernet1/0/38
!
interface GigabitEthernet1/0/39
!
interface GigabitEthernet1/0/40
!
interface GigabitEthernet1/0/41
!
interface GigabitEthernet1/0/42
!
interface GigabitEthernet1/0/43
!
interface GigabitEthernet1/0/44
!
interface GigabitEthernet1/0/45
!
interface GigabitEthernet1/0/46
!
interface GigabitEthernet1/0/47
!
interface GigabitEthernet1/0/48
!
interface GigabitEthernet1/0/49
!
interface GigabitEthernet1/0/50
!
interface GigabitEthernet1/0/51
!
interface GigabitEthernet1/0/52
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
!
interface GigabitEthernet2/0/15
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface GigabitEthernet2/0/19
!
interface GigabitEthernet2/0/20
!
interface GigabitEthernet2/0/21
!
interface GigabitEthernet2/0/22
!
interface GigabitEthernet2/0/23
!
interface GigabitEthernet2/0/24
!
interface GigabitEthernet2/0/25
!
interface GigabitEthernet2/0/26
!
interface GigabitEthernet2/0/27
!
interface GigabitEthernet2/0/28
!
interface GigabitEthernet2/0/29
!
interface GigabitEthernet2/0/30
!
interface GigabitEthernet2/0/31
!
interface GigabitEthernet2/0/32
!
interface GigabitEthernet2/0/33
!
interface GigabitEthernet2/0/34
!
interface GigabitEthernet2/0/35
!
interface GigabitEthernet2/0/36
!
interface GigabitEthernet2/0/37
!
interface GigabitEthernet2/0/38
!
interface GigabitEthernet2/0/39
!
interface GigabitEthernet2/0/40
!
interface GigabitEthernet2/0/41
!
interface GigabitEthernet2/0/42
!
interface GigabitEthernet2/0/43
!
interface GigabitEthernet2/0/44
!
interface GigabitEthernet2/0/45
!
interface GigabitEthernet2/0/46
!
interface GigabitEthernet2/0/47
!
interface GigabitEthernet2/0/48
!
interface GigabitEthernet2/0/49
!
interface GigabitEthernet2/0/50
!
interface GigabitEthernet2/0/51
!
interface GigabitEthernet2/0/52
!
interface GigabitEthernet3/0/1
!
interface GigabitEthernet3/0/2
!
interface GigabitEthernet3/0/3
!
interface GigabitEthernet3/0/4
!
interface GigabitEthernet3/0/5
!
interface GigabitEthernet3/0/6
!
interface GigabitEthernet3/0/7
!
interface GigabitEthernet3/0/8
!
interface GigabitEthernet3/0/9
!
interface GigabitEthernet3/0/10
!
interface GigabitEthernet3/0/11
!
interface GigabitEthernet3/0/12
!
interface GigabitEthernet3/0/13
!
interface GigabitEthernet3/0/14
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
!
interface GigabitEthernet3/0/18
!
interface GigabitEthernet3/0/19
!
interface GigabitEthernet3/0/20
!
interface GigabitEthernet3/0/21
!
interface GigabitEthernet3/0/22
!
interface GigabitEthernet3/0/23
!
interface GigabitEthernet3/0/24
!
interface GigabitEthernet3/0/25
!
interface GigabitEthernet3/0/26
!
interface GigabitEthernet3/0/27
!
interface GigabitEthernet3/0/28
!
interface GigabitEthernet3/0/29
!
interface GigabitEthernet3/0/30
!
interface GigabitEthernet3/0/31
!
interface GigabitEthernet3/0/32
!
interface GigabitEthernet3/0/33
!
interface GigabitEthernet3/0/34
!
interface GigabitEthernet3/0/35
!
interface GigabitEthernet3/0/36
!
interface GigabitEthernet3/0/37
!
interface GigabitEthernet3/0/38
!
interface GigabitEthernet3/0/39
!
interface GigabitEthernet3/0/40
!
interface GigabitEthernet3/0/41
!
interface GigabitEthernet3/0/42
!
interface GigabitEthernet3/0/43
!
interface GigabitEthernet3/0/44
!
interface GigabitEthernet3/0/45
!
interface GigabitEthernet3/0/46
!
interface GigabitEthernet3/0/47
!
interface GigabitEthernet3/0/48
!
interface GigabitEthernet3/0/49
!
interface GigabitEthernet3/0/50
!
interface GigabitEthernet3/0/51
!
interface GigabitEthernet3/0/52
!
interface Vlan1
 no ip address
 shutdown
!
ip http server
ip http secure-server
!
line con 0
line vty 0 4
 password ***
 login
line vty 5 15
 password ***
 login
!
end

Please let me know what other information you need.

The bond config was working in active/backup mode on some old dell switches at the old companies' location.


Thank you!
0
Comment
Question by:ar3
  • 3
  • 3
  • 2
  • +1
9 Comments
 

Author Comment

by:ar3
ID: 39643213
Example output from a server pinging the dns name of the server:

17:47:21.553136 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 00:10:**:**:**:80 (oui Unknown), length 46
17:47:21.553158 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 84:2b:**:**:**:ab (oui Unknown), length 46
0
 

Author Comment

by:ar3
ID: 39643215
The Cisco switch doesn't show mac flapping or drops.
0
 
LVL 47

Accepted Solution

by:
Craig Beck earned 2000 total points
ID: 39644405
You're using LACP on the servers, so you need to use LACP on the switches too.

On the stack master, enter the following configuration...

conf t
int po1
 switchport mode access
 no shutdown
!
int gigabitethernet 1/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 2/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/2
 switchport mode access
 channel-group 1 mode active
!
end

Open in new window


That will configure an EtherChannel using LACP on ports 1/0/1, 2/0/1, 3/0/1 and 3/0/2.  Substitute the ports numbers with the actual port numbers.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 41

Expert Comment

by:noci
ID: 39644455
In my experience LACP might fail under heavy load. I have seen it break on a heavily used iSCSI channel.
So if you known the links before then IMHO the prefered setting is a static one.
(Missing interface will be spotted & removed anyway).

The automatic features are only handy for temporary equipment OR placing new systems while not managing your environment for those systems.
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 39644478
@noci, I would use multipath for iSCSi rather than LACP...
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 39644504
I would use multipath too... but the OP states LACP is being used, so that's what I've gone with :-)
0
 
LVL 41

Expert Comment

by:noci
ID: 39644522
That would help if both units had 2 interfaces. otherwise it will flood one of them...
redundancy was the first requirement,  bandwidth the 2nd, Using 2 storage units in mirror as backend... on a server with multiple cards.

(The discussion was about bonding / LACP not my iSCSI setup..  ;-)
0
 

Author Comment

by:ar3
ID: 39644738
Would a separate etherchannel group need to be configured for each bonded set of interfaces?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 39644863
It depends how it is configured.  If it's all 4 in the same bond, then no.  If it's two bonded pairs added to a bridge then yes you will probably need two separate Etherchannels.  However it might not even work properly unless the OS knows to keep the passive bond disabled until it's required.
0

Featured Post

NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

 One of the main issues with network wires is that you never have enough.  You run plenty and plan for the worst case but you still end up needing more.  What many people do not realize is with 10BaseT and 100BaseT (but not 1000BaseT) networks you …
In this article, the configuration steps in Zabbix to monitor devices via SNMP will be discussed with some real examples on Cisco Router/Switch, Catalyst Switch, NAS Synology device.
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…
Suggested Courses
Course of the Month14 days, 6 hours left to enroll

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question