asked on

Network issues with Linux Bonding and Cisco 2960-S

Hi,

I have an environment that I am having some networking issues with. Major packet loss and multiple mac addresses on arp replies. My current thought is that it is related to configuration settings of the bonds and the configuration of the cisco switches. Possibly an issue with vlans and/or etherchannel. All of which I am unfamiliar with.

I'm sure you will want more information, but to start, please see the configs here:

Server (Debian GNU/Linux 6.0 \n \l):

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto bond0
iface bond0 inet static
address 10.1.4.15
netmask 255.255.0.0
network 10.1.0.0
broadcast 10.1.255.255
gateway 10.1.1.1
bond_mode 802.3ad
bond_miimon 100
bond_downdelay 200
bond_updelay 200
slaves eth4 eth5

auto bond1
iface bond1 inet manual
bond_mode 802.3ad
bond_miimon 100
bond_downdelay 200
bond_updelay 200
slaves eth2 eth3

auto br0
iface br0 inet manual
#bridge_ports bond1
#bridge_stp off
#bridge_fd 0
#bridge_maxwait 0
pre-up ip link set bond1 down
pre-up brctl addbr br0
pre-up brctl addif br0 bond1
pre-up ip link set bond1 up
pre-up ip link set br0 up
post-down ip link set br0 down
post-down brctl delbr br0

Cisco 2960-S Stack (four Cisco 2960S with fiber interconnection)

Current config:

***#sh conf
Using 6326 out of 524288 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ***
!
boot-start-marker
boot-end-marker
!
enable secret 5 ***
enable password ***
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
switch 3 provision ws-c2960s-48ts-l
!
!
!
!
crypto pki trustpoint TP-self-signed-1561874560
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1561874560
revocation-check none
rsakeypair TP-self-signed-1561874560
!
!
crypto pki certificate chain TP-self-signed-1561874560
certificate self-signed 01 nvram:IOS-Self-Sig#3030.cer
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0
no ip address
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/0/25
!
interface GigabitEthernet1/0/26
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
!
interface GigabitEthernet1/0/29
!
interface GigabitEthernet1/0/30
!
interface GigabitEthernet1/0/31
!
interface GigabitEthernet1/0/32
!
interface GigabitEthernet1/0/33
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
!
interface GigabitEthernet1/0/36
!
interface GigabitEthernet1/0/37
!
interface GigabitEthernet1/0/38
!
interface GigabitEthernet1/0/39
!
interface GigabitEthernet1/0/40
!
interface GigabitEthernet1/0/41
!
interface GigabitEthernet1/0/42
!
interface GigabitEthernet1/0/43
!
interface GigabitEthernet1/0/44
!
interface GigabitEthernet1/0/45
!
interface GigabitEthernet1/0/46
!
interface GigabitEthernet1/0/47
!
interface GigabitEthernet1/0/48
!
interface GigabitEthernet1/0/49
!
interface GigabitEthernet1/0/50
!
interface GigabitEthernet1/0/51
!
interface GigabitEthernet1/0/52
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
!
interface GigabitEthernet2/0/15
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface GigabitEthernet2/0/19
!
interface GigabitEthernet2/0/20
!
interface GigabitEthernet2/0/21
!
interface GigabitEthernet2/0/22
!
interface GigabitEthernet2/0/23
!
interface GigabitEthernet2/0/24
!
interface GigabitEthernet2/0/25
!
interface GigabitEthernet2/0/26
!
interface GigabitEthernet2/0/27
!
interface GigabitEthernet2/0/28
!
interface GigabitEthernet2/0/29
!
interface GigabitEthernet2/0/30
!
interface GigabitEthernet2/0/31
!
interface GigabitEthernet2/0/32
!
interface GigabitEthernet2/0/33
!
interface GigabitEthernet2/0/34
!
interface GigabitEthernet2/0/35
!
interface GigabitEthernet2/0/36
!
interface GigabitEthernet2/0/37
!
interface GigabitEthernet2/0/38
!
interface GigabitEthernet2/0/39
!
interface GigabitEthernet2/0/40
!
interface GigabitEthernet2/0/41
!
interface GigabitEthernet2/0/42
!
interface GigabitEthernet2/0/43
!
interface GigabitEthernet2/0/44
!
interface GigabitEthernet2/0/45
!
interface GigabitEthernet2/0/46
!
interface GigabitEthernet2/0/47
!
interface GigabitEthernet2/0/48
!
interface GigabitEthernet2/0/49
!
interface GigabitEthernet2/0/50
!
interface GigabitEthernet2/0/51
!
interface GigabitEthernet2/0/52
!
interface GigabitEthernet3/0/1
!
interface GigabitEthernet3/0/2
!
interface GigabitEthernet3/0/3
!
interface GigabitEthernet3/0/4
!
interface GigabitEthernet3/0/5
!
interface GigabitEthernet3/0/6
!
interface GigabitEthernet3/0/7
!
interface GigabitEthernet3/0/8
!
interface GigabitEthernet3/0/9
!
interface GigabitEthernet3/0/10
!
interface GigabitEthernet3/0/11
!
interface GigabitEthernet3/0/12
!
interface GigabitEthernet3/0/13
!
interface GigabitEthernet3/0/14
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
!
interface GigabitEthernet3/0/18
!
interface GigabitEthernet3/0/19
!
interface GigabitEthernet3/0/20
!
interface GigabitEthernet3/0/21
!
interface GigabitEthernet3/0/22
!
interface GigabitEthernet3/0/23
!
interface GigabitEthernet3/0/24
!
interface GigabitEthernet3/0/25
!
interface GigabitEthernet3/0/26
!
interface GigabitEthernet3/0/27
!
interface GigabitEthernet3/0/28
!
interface GigabitEthernet3/0/29
!
interface GigabitEthernet3/0/30
!
interface GigabitEthernet3/0/31
!
interface GigabitEthernet3/0/32
!
interface GigabitEthernet3/0/33
!
interface GigabitEthernet3/0/34
!
interface GigabitEthernet3/0/35
!
interface GigabitEthernet3/0/36
!
interface GigabitEthernet3/0/37
!
interface GigabitEthernet3/0/38
!
interface GigabitEthernet3/0/39
!
interface GigabitEthernet3/0/40
!
interface GigabitEthernet3/0/41
!
interface GigabitEthernet3/0/42
!
interface GigabitEthernet3/0/43
!
interface GigabitEthernet3/0/44
!
interface GigabitEthernet3/0/45
!
interface GigabitEthernet3/0/46
!
interface GigabitEthernet3/0/47
!
interface GigabitEthernet3/0/48
!
interface GigabitEthernet3/0/49
!
interface GigabitEthernet3/0/50
!
interface GigabitEthernet3/0/51
!
interface GigabitEthernet3/0/52
!
interface Vlan1
no ip address
shutdown
!
ip http server
ip http secure-server
!
line con 0
line vty 0 4
password ***
login
line vty 5 15
password ***
login
!
end

Please let me know what other information you need.

The bond config was working in active/backup mode on some old dell switches at the old companies' location.

Thank you!

ar3

ASKER

Example output from a server pinging the dns name of the server:

17:47:21.553136 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 00:10:**:**:**:80 (oui Unknown), length 46
17:47:21.553158 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 84:2b:**:**:**:ab (oui Unknown), length 46

ar3

ASKER

The Cisco switch doesn't show mac flapping or drops.

ASKER CERTIFIED SOLUTION

Craig Beck

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

noci

In my experience LACP might fail under heavy load. I have seen it break on a heavily used iSCSI channel.
So if you known the links before then IMHO the prefered setting is a static one.
(Missing interface will be spotted & removed anyway).

The automatic features are only handy for temporary equipment OR placing new systems while not managing your environment for those systems.

ArneLovius

@noci, I would use multipath for iSCSi rather than LACP...

Craig Beck

I would use multipath too... but the OP states LACP is being used, so that's what I've gone with :-)

noci

That would help if both units had 2 interfaces. otherwise it will flood one of them...
redundancy was the first requirement, bandwidth the 2nd, Using 2 storage units in mirror as backend... on a server with multiple cards.

(The discussion was about bonding / LACP not my iSCSI setup.. ;-)

ar3

ASKER

Would a separate etherchannel group need to be configured for each bonded set of interfaces?

Craig Beck

It depends how it is configured. If it's all 4 in the same bond, then no. If it's two bonded pairs added to a bridge then yes you will probably need two separate Etherchannels. However it might not even work properly unless the OS knows to keep the passive bond disabled until it's required.