Solved

Network issues with Linux Bonding and Cisco 2960-S

Posted on 2013-11-12
9
865 Views
Last Modified: 2016-11-23
Hi,

I have an environment that I am having some networking issues with.  Major packet loss and multiple mac addresses on arp replies.  My current thought is that it is related to configuration settings of the bonds and the configuration of the cisco switches.  Possibly an issue with vlans and/or etherchannel.  All of which I am unfamiliar with.

I'm sure you will want more information, but to start, please see the configs here:

Server (Debian GNU/Linux 6.0 \n \l):

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto bond0
iface bond0 inet static
        address 10.1.4.15
        netmask 255.255.0.0
        network 10.1.0.0
        broadcast 10.1.255.255
        gateway 10.1.1.1
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth4 eth5

auto bond1
iface bond1 inet manual
        bond_mode 802.3ad
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200
        slaves eth2 eth3


auto br0
iface br0 inet manual
        #bridge_ports bond1
        #bridge_stp off
        #bridge_fd 0
        #bridge_maxwait 0
        pre-up ip link set bond1 down
        pre-up brctl addbr br0
        pre-up brctl addif br0 bond1
        pre-up ip link set bond1 up
        pre-up ip link set br0 up
        post-down ip link set br0 down
        post-down brctl delbr br0


Cisco 2960-S Stack (four Cisco 2960S with fiber interconnection)

Current config:

***#sh conf
Using 6326 out of 524288 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ***
!
boot-start-marker
boot-end-marker
!
enable secret 5 ***
enable password ***
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
switch 3 provision ws-c2960s-48ts-l
!
!
!
!
crypto pki trustpoint TP-self-signed-1561874560
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1561874560
 revocation-check none
 rsakeypair TP-self-signed-1561874560
!
!
crypto pki certificate chain TP-self-signed-1561874560
 certificate self-signed 01 nvram:IOS-Self-Sig#3030.cer
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/0/25
!
interface GigabitEthernet1/0/26
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
!
interface GigabitEthernet1/0/29
!
interface GigabitEthernet1/0/30
!
interface GigabitEthernet1/0/31
!
interface GigabitEthernet1/0/32
!
interface GigabitEthernet1/0/33
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
!
interface GigabitEthernet1/0/36
!
interface GigabitEthernet1/0/37
!
interface GigabitEthernet1/0/38
!
interface GigabitEthernet1/0/39
!
interface GigabitEthernet1/0/40
!
interface GigabitEthernet1/0/41
!
interface GigabitEthernet1/0/42
!
interface GigabitEthernet1/0/43
!
interface GigabitEthernet1/0/44
!
interface GigabitEthernet1/0/45
!
interface GigabitEthernet1/0/46
!
interface GigabitEthernet1/0/47
!
interface GigabitEthernet1/0/48
!
interface GigabitEthernet1/0/49
!
interface GigabitEthernet1/0/50
!
interface GigabitEthernet1/0/51
!
interface GigabitEthernet1/0/52
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
!
interface GigabitEthernet2/0/15
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface GigabitEthernet2/0/19
!
interface GigabitEthernet2/0/20
!
interface GigabitEthernet2/0/21
!
interface GigabitEthernet2/0/22
!
interface GigabitEthernet2/0/23
!
interface GigabitEthernet2/0/24
!
interface GigabitEthernet2/0/25
!
interface GigabitEthernet2/0/26
!
interface GigabitEthernet2/0/27
!
interface GigabitEthernet2/0/28
!
interface GigabitEthernet2/0/29
!
interface GigabitEthernet2/0/30
!
interface GigabitEthernet2/0/31
!
interface GigabitEthernet2/0/32
!
interface GigabitEthernet2/0/33
!
interface GigabitEthernet2/0/34
!
interface GigabitEthernet2/0/35
!
interface GigabitEthernet2/0/36
!
interface GigabitEthernet2/0/37
!
interface GigabitEthernet2/0/38
!
interface GigabitEthernet2/0/39
!
interface GigabitEthernet2/0/40
!
interface GigabitEthernet2/0/41
!
interface GigabitEthernet2/0/42
!
interface GigabitEthernet2/0/43
!
interface GigabitEthernet2/0/44
!
interface GigabitEthernet2/0/45
!
interface GigabitEthernet2/0/46
!
interface GigabitEthernet2/0/47
!
interface GigabitEthernet2/0/48
!
interface GigabitEthernet2/0/49
!
interface GigabitEthernet2/0/50
!
interface GigabitEthernet2/0/51
!
interface GigabitEthernet2/0/52
!
interface GigabitEthernet3/0/1
!
interface GigabitEthernet3/0/2
!
interface GigabitEthernet3/0/3
!
interface GigabitEthernet3/0/4
!
interface GigabitEthernet3/0/5
!
interface GigabitEthernet3/0/6
!
interface GigabitEthernet3/0/7
!
interface GigabitEthernet3/0/8
!
interface GigabitEthernet3/0/9
!
interface GigabitEthernet3/0/10
!
interface GigabitEthernet3/0/11
!
interface GigabitEthernet3/0/12
!
interface GigabitEthernet3/0/13
!
interface GigabitEthernet3/0/14
!
interface GigabitEthernet3/0/15
!
interface GigabitEthernet3/0/16
!
interface GigabitEthernet3/0/17
!
interface GigabitEthernet3/0/18
!
interface GigabitEthernet3/0/19
!
interface GigabitEthernet3/0/20
!
interface GigabitEthernet3/0/21
!
interface GigabitEthernet3/0/22
!
interface GigabitEthernet3/0/23
!
interface GigabitEthernet3/0/24
!
interface GigabitEthernet3/0/25
!
interface GigabitEthernet3/0/26
!
interface GigabitEthernet3/0/27
!
interface GigabitEthernet3/0/28
!
interface GigabitEthernet3/0/29
!
interface GigabitEthernet3/0/30
!
interface GigabitEthernet3/0/31
!
interface GigabitEthernet3/0/32
!
interface GigabitEthernet3/0/33
!
interface GigabitEthernet3/0/34
!
interface GigabitEthernet3/0/35
!
interface GigabitEthernet3/0/36
!
interface GigabitEthernet3/0/37
!
interface GigabitEthernet3/0/38
!
interface GigabitEthernet3/0/39
!
interface GigabitEthernet3/0/40
!
interface GigabitEthernet3/0/41
!
interface GigabitEthernet3/0/42
!
interface GigabitEthernet3/0/43
!
interface GigabitEthernet3/0/44
!
interface GigabitEthernet3/0/45
!
interface GigabitEthernet3/0/46
!
interface GigabitEthernet3/0/47
!
interface GigabitEthernet3/0/48
!
interface GigabitEthernet3/0/49
!
interface GigabitEthernet3/0/50
!
interface GigabitEthernet3/0/51
!
interface GigabitEthernet3/0/52
!
interface Vlan1
 no ip address
 shutdown
!
ip http server
ip http secure-server
!
line con 0
line vty 0 4
 password ***
 login
line vty 5 15
 password ***
 login
!
end

Please let me know what other information you need.

The bond config was working in active/backup mode on some old dell switches at the old companies' location.


Thank you!
0
Comment
Question by:ar3
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
9 Comments
 

Author Comment

by:ar3
ID: 39643213
Example output from a server pinging the dns name of the server:

17:47:21.553136 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 00:10:**:**:**:80 (oui Unknown), length 46
17:47:21.553158 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.15 is-at 84:2b:**:**:**:ab (oui Unknown), length 46
0
 

Author Comment

by:ar3
ID: 39643215
The Cisco switch doesn't show mac flapping or drops.
0
 
LVL 46

Accepted Solution

by:
Craig Beck earned 500 total points
ID: 39644405
You're using LACP on the servers, so you need to use LACP on the switches too.

On the stack master, enter the following configuration...

conf t
int po1
 switchport mode access
 no shutdown
!
int gigabitethernet 1/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 2/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/1
 switchport mode access
 channel-group 1 mode active
!
int gigabitethernet 3/0/2
 switchport mode access
 channel-group 1 mode active
!
end

Open in new window


That will configure an EtherChannel using LACP on ports 1/0/1, 2/0/1, 3/0/1 and 3/0/2.  Substitute the ports numbers with the actual port numbers.
0
Flexible connectivity for any environment

The KE6900 series can extend and deploy computers with high definition displays across multiple stations in a variety of applications that suit any environment. Expand computer use to stations across multiple rooms with dynamic access.

 
LVL 40

Expert Comment

by:noci
ID: 39644455
In my experience LACP might fail under heavy load. I have seen it break on a heavily used iSCSI channel.
So if you known the links before then IMHO the prefered setting is a static one.
(Missing interface will be spotted & removed anyway).

The automatic features are only handy for temporary equipment OR placing new systems while not managing your environment for those systems.
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 39644478
@noci, I would use multipath for iSCSi rather than LACP...
0
 
LVL 46

Expert Comment

by:Craig Beck
ID: 39644504
I would use multipath too... but the OP states LACP is being used, so that's what I've gone with :-)
0
 
LVL 40

Expert Comment

by:noci
ID: 39644522
That would help if both units had 2 interfaces. otherwise it will flood one of them...
redundancy was the first requirement,  bandwidth the 2nd, Using 2 storage units in mirror as backend... on a server with multiple cards.

(The discussion was about bonding / LACP not my iSCSI setup..  ;-)
0
 

Author Comment

by:ar3
ID: 39644738
Would a separate etherchannel group need to be configured for each bonded set of interfaces?
0
 
LVL 46

Expert Comment

by:Craig Beck
ID: 39644863
It depends how it is configured.  If it's all 4 in the same bond, then no.  If it's two bonded pairs added to a bridge then yes you will probably need two separate Etherchannels.  However it might not even work properly unless the OS knows to keep the passive bond disabled until it's required.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I see many questions here on Experts Exchange regarding switch port configurations and trunks. This article is meant for beginners in the subject to help to get basic knowledge about Virtual Local Area Network (VLAN (http://en.wikipedia.org/wiki/Vir…
When posting a question about a Cisco ASA, Cisco Router or Cisco Switch, it can aid diagnosis if a suitably sanitised copy of the config is provided. It is much better to leave as much of the configuration as original as possible, as it could be tha…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question