Solved

Network issues with Linux Bonding and Cisco 2960-S

Posted on 2013-11-12
9
863 Views
Last Modified: 2016-11-23
Hi,

I have an environment that I am having some networking issues with.  Major packet loss and multiple mac addresses on arp replies.  My current thought is that it is related to configuration settings of the bonds and the configuration of the cisco switches.  Possibly an issue with vlans and/or etherchannel.  All of which I am unfamiliar with.

I'm sure you will want more information, but to start, please see the configs here:

Server (Debian GNU/Linux 6.0 \n \l):

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto bond0
iface bond0 inet static
        address 10.1.4.15
        netmask 255.255.0.0
        network 10.1.0.0
        broadcast 10.1.255.255
        gateway 10.1.1.1
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth4 eth5

auto bond1
iface bond1 inet manual
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth2 eth3


auto br0
iface br0 inet manual
        #bridge_ports bond1
        #bridge_stp off
        #bridge_fd 0
        #bridge_maxwait 0
        pre-up ip link set bond1 down
        pre-up brctl addbr br0
        pre-up brctl addif br0 bond1
        pre-up ip link set bond1 up
        pre-up ip link set br0 up
        post-down ip link set br0 down
        post-down brctl delbr br0


Cisco 2960-S Stack (four Cisco 2960S with fiber interconnection)

Current config:

***#sh conf
Using 6326 out of 524288 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ***
!
boot-start-marker
boot-end-marker
!
enable secret 5 ***
enable password ***
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
switch 3 provision ws-c2960s-48ts-l
!
!
!
!
crypto pki trustpoint TP-self-signed-1561874560
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1561874560
 revocation-check none
 rsakeypair TP-self-signed-1561874560
!
!
crypto pki certificate chain TP-self-signed-1561874560
 certificate self-signed 01 nvram:IOS-Self-Sig#3030.cer
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/0/25
!
interface GigabitEthernet1/0/26
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
!
interface GigabitEthernet1/0/29
!
interface GigabitEthernet1/0/30
!
interface GigabitEthernet1/0/31
!
interface GigabitEthernet1/0/32
!
interface GigabitEthernet1/0/33
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
!
interface GigabitEthernet1/0/36
!
interface GigabitEthernet1/0/37
!
interface GigabitEthernet1/0/38
!
interface GigabitEthernet1/0/39
!
interface GigabitEthernet1/0/40
!
interface GigabitEthernet1/0/41
!
interface GigabitEthernet1/0/42
!
interface GigabitEthernet1/0/43
!
interface GigabitEthernet1/0/44
!
interface GigabitEthernet1/0/45
!
interface GigabitEthernet1/0/46
!
interface GigabitEthernet1/0/47
!
interface GigabitEthernet1/0/48
!
interface GigabitEthernet1/0/49
!
interface GigabitEthernet1/0/50
!
interface GigabitEthernet1/0/51
!
interface GigabitEthernet1/0/52
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
!
interface GigabitEthernet2/0/15
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface GigabitEthernet2/0/19
!
interface GigabitEthernet2/0/20
!
interface GigabitEthernet2/0/21
!
interface GigabitEthernet2/0/22
!
interface GigabitEthernet2/0/23
!
interface GigabitEthernet2/0/24
!
interface GigabitEthernet2/0/25
!
interface GigabitEthernet2/0/26
!
interface GigabitEthernet2/0/27
!
interface GigabitEthernet2/0/28
!
interface GigabitEthernet2/0/29
!
interface GigabitEthernet2/0/30
!
interface GigabitEthernet2/0/31
!
interface GigabitEthernet2/0/32
!
interface GigabitEthernet2/0/33
!
interface GigabitEthernet2/0/34
!
interface GigabitEthernet2/0/35
!
interface GigabitEthernet2/0/36
!
interface GigabitEthernet2/0/37
!
interface GigabitEthernet2/0/38
!
interface GigabitEthernet2/0/39
!
interface GigabitEthernet2/0/40
!
interface GigabitEthernet2/0/41
!
interface GigabitEthernet2/0/42
!
interface GigabitEthernet2/0/43
!
interface GigabitEthernet2/0/44
!
interface GigabitEthernet2/0/45
!
interface GigabitEthernet2/0/46
!
interface GigabitEthernet2/0/47
!
interface GigabitEthernet2/0/48
!
interface GigabitEthernet2/0/49
!
interface GigabitEthernet2/0/50
!
interface GigabitEthernet2/0/51
!
interface GigabitEthernet2/0/52
!
interface GigabitEthernet3/0/1
!
interface GigabitEthernet3/0/2
!
interface GigabitEthernet3/0/3
!
interface GigabitEthernet3/0/4
!
interface GigabitEthernet3/0/5
!
interface GigabitEthernet3/0/6
!
interface GigabitEthernet3/0/7
!
interface GigabitEthernet3/0/8
!
interface GigabitEthernet3/0/9
!
interface GigabitEthernet3/0/10
!
interface GigabitEthernet3/0/11
!
interface GigabitEthernet3/0/12
!
interface GigabitEthernet3/0/13
!
interface GigabitEthernet3/0/14
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
!
interface GigabitEthernet3/0/18
!
interface GigabitEthernet3/0/19
!
interface GigabitEthernet3/0/20
!
interface GigabitEthernet3/0/21
!
interface GigabitEthernet3/0/22
!
interface GigabitEthernet3/0/23
!
interface GigabitEthernet3/0/24
!
interface GigabitEthernet3/0/25
!
interface GigabitEthernet3/0/26
!
interface GigabitEthernet3/0/27
!
interface GigabitEthernet3/0/28
!
interface GigabitEthernet3/0/29
!
interface GigabitEthernet3/0/30
!
interface GigabitEthernet3/0/31
!
interface GigabitEthernet3/0/32
!
interface GigabitEthernet3/0/33
!
interface GigabitEthernet3/0/34
!
interface GigabitEthernet3/0/35
!
interface GigabitEthernet3/0/36
!
interface GigabitEthernet3/0/37
!
interface GigabitEthernet3/0/38
!
interface GigabitEthernet3/0/39
!
interface GigabitEthernet3/0/40
!
interface GigabitEthernet3/0/41
!
interface GigabitEthernet3/0/42
!
interface GigabitEthernet3/0/43
!
interface GigabitEthernet3/0/44
!
interface GigabitEthernet3/0/45
!
interface GigabitEthernet3/0/46
!
interface GigabitEthernet3/0/47
!
interface GigabitEthernet3/0/48
!
interface GigabitEthernet3/0/49
!
interface GigabitEthernet3/0/50
!
interface GigabitEthernet3/0/51
!
interface GigabitEthernet3/0/52
!
interface Vlan1
 no ip address
 shutdown
!
ip http server
ip http secure-server
!
line con 0
line vty 0 4
 password ***
 login
line vty 5 15
 password ***
 login
!
end

Please let me know what other information you need.

The bond config was working in active/backup mode on some old dell switches at the old companies' location.


Thank you!
0
Comment
Question by:ar3
  • 3
  • 3
  • 2
  • +1
9 Comments
 

Author Comment

by:ar3
ID: 39643213
Example output from a server pinging the dns name of the server:

17:47:21.553136 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 00:10:**:**:**:80 (oui Unknown), length 46
17:47:21.553158 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 84:2b:**:**:**:ab (oui Unknown), length 46
0
 

Author Comment

by:ar3
ID: 39643215
The Cisco switch doesn't show mac flapping or drops.
0
 
LVL 45

Accepted Solution

by:
Craig Beck earned 500 total points
ID: 39644405
You're using LACP on the servers, so you need to use LACP on the switches too.

On the stack master, enter the following configuration...

conf t
int po1
 switchport mode access
 no shutdown
!
int gigabitethernet 1/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 2/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/2
 switchport mode access
 channel-group 1 mode active
!
end

Open in new window


That will configure an EtherChannel using LACP on ports 1/0/1, 2/0/1, 3/0/1 and 3/0/2.  Substitute the ports numbers with the actual port numbers.
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 40

Expert Comment

by:noci
ID: 39644455
In my experience LACP might fail under heavy load. I have seen it break on a heavily used iSCSI channel.
So if you known the links before then IMHO the prefered setting is a static one.
(Missing interface will be spotted & removed anyway).

The automatic features are only handy for temporary equipment OR placing new systems while not managing your environment for those systems.
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 39644478
@noci, I would use multipath for iSCSi rather than LACP...
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39644504
I would use multipath too... but the OP states LACP is being used, so that's what I've gone with :-)
0
 
LVL 40

Expert Comment

by:noci
ID: 39644522
That would help if both units had 2 interfaces. otherwise it will flood one of them...
redundancy was the first requirement,  bandwidth the 2nd, Using 2 storage units in mirror as backend... on a server with multiple cards.

(The discussion was about bonding / LACP not my iSCSI setup..  ;-)
0
 

Author Comment

by:ar3
ID: 39644738
Would a separate etherchannel group need to be configured for each bonded set of interfaces?
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39644863
It depends how it is configured.  If it's all 4 in the same bond, then no.  If it's two bonded pairs added to a bridge then yes you will probably need two separate Etherchannels.  However it might not even work properly unless the OS knows to keep the passive bond disabled until it's required.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I see many questions here on Experts Exchange regarding switch port configurations and trunks. This article is meant for beginners in the subject to help to get basic knowledge about Virtual Local Area Network (VLAN (http://en.wikipedia.org/wiki/Vir…
Before I go to far, let's explain HA (High Availability) and why you should consider it.  High availability is the mechanism used to provide redundancy to any service at the same site and appears as a single service to the users of that service.  As…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question