Solved

Bonding two NIC Ubuntu 12.04.1 LTS

Posted on 2012-12-31
25
1,501 Views
Last Modified: 2013-01-17
OS is currently Ubuntu 12.04.1 LTS with a network tap (not aggregator).

I have a new nTap device and I am trying to create a bonded interface so I canput the two streams of traffic back together so my snort IDS can evaluate the traffic.

I have tried many, many different settings in several different files but I cannot seem to get both NICS to become slaves to the bond0 interface.

Might anyone be interesting in working through the possible configurations that would allow me to accomplish my goal?

Thanks
0
Comment
Question by:spencerturbine
  • 11
  • 10
  • 2
  • +1
25 Comments
 
LVL 76

Expert Comment

by:arnold
ID: 38733608
Unfortunately you did not include what you tried or what has not worked.

Do you have a managed switch where you can combined the two switch ports into which you areplugging the two cards from the system?

The mode of the bonded interface needs to be set to match.
https://help.ubuntu.com/community/UbuntuBonding
0
 

Author Comment

by:spencerturbine
ID: 38733629
I don't have a switch connected to these two ports of the Ubuntu server. I have an network tap (splitter, not aggregator).

http://www.cdw.com/shop/products/Network-Instruments-nTAP-10-100-1000-Copper-tap-splitter/822880.aspx

I have followed the guide provided at the link you provided. However this is the output of the cat /proc/net/bonding/bond0 command:

root@ids:/etc/modprobe.d# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 8
        Number of ports: 1
        Actor Key: 17
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:02:a5:4e:da:39
Aggregator ID: 8
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:02:a5:4e:da:38
Aggregator ID: 9
Slave queue ID: 0

Open in new window

0
 
LVL 76

Expert Comment

by:arnold
ID: 38733682
To do what you need, you need a switch that will aggregate the connections to make it appear as one.
What is it connecting to on the other side? Do you have a disaggregator or do you split the feed back out before connecting it into a switch?


What is the issue? You are using 802.3ad which requires a bonding of the ports on the switch side.
0
 

Author Comment

by:spencerturbine
ID: 38733702
I have the following connections Internet --> Cisco Perim Router --> Cisco ASA --> nTAP

The nTap feed one leg to our Cisco stack of switches.  Now I have two other ports to use to connect to my IDS each having one side of the conversation. (Hence why its a splitter and not an aggregator).

Now I must combine those two streams back together so my IDS can do its job.

I have connected the two ports on the nTap to two ports on a single NIC I have on the IDS server. I have a third single port NIC serving as my management connection.

eth1 sees outbound traffic
eth2 sees inbound traffic

eth0 is my managment traffic.

I need the eth1 and eth2 bonded together so as to the two seperate streams merged into one so the IDS can understand the stateful traffic.

The major point here is that there is NO switch between the nTAP and the Server.
0
 

Author Comment

by:spencerturbine
ID: 38733707
From the nTAP site:

Q: The NIC in my analysis device has a single “receive” port. Will I get visibility into a full-duplex connection with an nTAP?

A: Yes, but you will only be able to view one side of the full-duplex link at a time. This is not recommended. An nTAP sends copies of the TX and RX of a full-duplex signal out through separate “send” ports to the monitoring device. If your monitoring device is equipped with a dual-receive capture card, you will be able to view both streams of data. However, if your monitoring device is equipped with only one receive port, it will only be able to view one stream of data. To view the entire full-duplex stream for analysis, the monitoring device should have two receive ports and the ability to aggregate TX and RX into a single stream.

I have equiped the server with a dual port capture card and I am trying to bond them together so at the applicaiton layer, the IDS sees both sides of the conversation.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733719
To do what you want you need to aggregate.
You are working on the premise that a single 1GB is not enough to handle the inbound traffic and pass it on?

You can use ASA as the aggregator.
http://www.cisco.com/en/US/docs/security/asa/asa84/configuration/guide/interface_start.html


Not sure what the purpose of yuur setup is.

Internet <=> cisco perimeter<=> asa <=> Ubuntu IDS router <=> LAN
0
 

Author Comment

by:spencerturbine
ID: 38733743
Diagram
0
 

Author Comment

by:spencerturbine
ID: 38733745
I am not bonding to handle increased traffic. I am bonding to get both the RX and the TX back into one stream.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733748
The whole point of IDS/IPS is detect and prevent the incorrect data from passing.

Your setup seems to be an observer is watching the door and records when a thief enters the premises.

What is it you want from the setup?
The split in TX/RX is the tap configuration and not something you can control.
Are you planing on adding an IPS using iptables/snort?


Your setup is a one feed there is no reason anything will be transmitted out from your IDS setup.
0
 

Author Comment

by:spencerturbine
ID: 38733756
The IDS's job is just to monitor. It does not transmit traffic. It will never run in inline mode so it is strictly a monitoring device.

What I want from the set up is for the IDS to monitor traffic going to the firewall and alerting me based on configuration rules and preprocessor configuration.

The IDS cannot do its job if the traffic is split over the two links (one with RX and one with TX). The two streams must be recombined and thats what I want to use bonding to accomplish.

This set up should work according to NetOptics the problem is just me, I am missing something in the configuration.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733759
Looking at the display of your hardware, one port is incoming traffic while the other is outgoing traffic.

I.e. RX stream is a stream from the router into the tap,
TX stream is the data from the ASA to the router.
I.e. eth1 will detect intrusive packets coming while eth2 will see traffic originating from the LAN. I.e. detects intrusive packets originating from your LAN in the event you have hacker on your network attacking external resources or have a local compromised system.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733772
Perform the following exercise using tcpdump connect in eth1 as you have without bonding in permisous mode and at the same time debug the inbound traffic from the router on the ASA and you should see the two being identical.
Repeating the same in reverse using the debug on the outgoing traffic from the ASA To the router should match the eth2 TX traffic.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:spencerturbine
ID: 38733778
Well yes... that's what I would expect. The nTAP splits the traffic. Bonding two NICS should recombine the split traffic.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733779
A managed switch with port monitoring configuration would combine the stream of data that you want.
Placing the monitor on the port from the ASA feeding the LAN will reduce the monitoring to traffic that passes your ASA configured rules versus looking at data of no significance.

I.e. the difference whether you place an observer outside the door or behind the door.

The outside the door will note individuals who pass by the door who have no intention of or do are denied the option to enter while the one inside the door, only records those who have gained access.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733795
http://www.networkinstruments.com/assets/pdf/white-papers/nTAP_FullDuplex_wp.pdf

Provides an illustration of the various monitoring options.
0
 

Author Comment

by:spencerturbine
ID: 38733800
I appreciate your comments Arnold, but we are sort of off track. We are more or less talking about architecture now instead of focusing on how to bond two NICS.

I previously used switchport monitoring using a Dell switch. The reason for the change to an nTAP was to understand how to set up a network tap in situations where it is required.

I have seen quite a few talks, and read a number of white papers over the years that talked about using a network tap instead of a hub/ switch with port mirroring or span port.

So you see I have serveral goals in mind here, learn more about networking in Ubuntu/Linux and increasing my understanding of network taps.
0
 

Author Comment

by:spencerturbine
ID: 38733815
Dual-receive means that
the network card on the analysis device has two receive channels rather than the transmit and receive channels associated with a
standard full-duplex link.

I find this statement to be somewhat of a problem. Try can google a nic with dual receive channels. I couldn't find one. This prompted a call to NetOptics in which I was told that meant two full duplex ports and in that case I would need to use bonding to bring them back to a single stream so the IDS could function properly.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38733849
I read through this trying to make sense.  I understand what you are attempting to do, but I don't think it work work.  What I think is the problem is your statement:

     "Bonding two NICS should recombine the split traffic."

I don't think it will.  The problem, as I see it, is that whatever is capturing the traffic will see both steams as input.

I'm not sure if you ran tcpdump or not or what the results were.

If you run tcpdump on the Ubuntu box using "-i any" does wireshark see the data from both NIC's?  Does it see everything being related as you would expect?
0
 
LVL 76

Expert Comment

by:arnold
ID: 38733896
What happens when you do tcpdump -i bond0?
0
 
LVL 61

Expert Comment

by:gheist
ID: 38734298
Bonding works without switch assistance too.
ntap requires bonding mode 2 and miimon ( are you sure your wires are plugged into right ports?)

mode 4 that you have configured needs "port-channel" on CCO side.
0
 

Author Comment

by:spencerturbine
ID: 38735394
I am certain I have the cables plugged into the correct ports.

tcpdump on bond0 currenttly on shows ingress traffic only. I do not see the egress traffic.

tcpdump eth1 shows only ingress traffic.

tcpdump eth2 shows only egress traffic.
0
 
LVL 76

Expert Comment

by:arnold
ID: 38735427
Change the mode of your bond0 interface to mode-4/802.3ab.
To mode 5 balance-tlb and see if that combines the two data streams.

The tap you have is often to analyze one side, you would need to configure your IDS to review each feed/interface.
0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 38735454
I suggest you contact the company makes the tap and see if they have any help for you or try other bonding modes.   What I notice is the following from the command "cat /proc/net/bonding/bond0"

Active Aggregator info section, it has "Number of ports: 1".    This should be equal to the number of ports that were actually bonded, in your case 2.  

It has the Aggregator ID of 8, and only eth2 shows matching this ID,  eth1 shows Aggregator ID of 9.  If they were bonding together properly, the ID's would match.

Active Aggregator info shows the partner mac address of all zeros.  This means that as far as the bonding code is concerned the device it is connected two does not support link aggregation, that is 802.3ad.

You can read this post:

     http://lkml.indiana.edu/hypermail/linux/kernel/0712.3/0115.html

And the follow up:

     http://lkml.indiana.edu/hypermail/linux/kernel/0712.3/0359.html

Remember 802.3ad is true link aggregation, I believe it needs to be negotiated between the devices  connected together.   It appears the tap is passive and there for needs some other form of bonding that does not require negotiation.

You may want to try bond-mode 0, 2, 3, or 5.  Although they talk about how it works for transmitting, these modes are all "active-active" modes that do not seem to need any special negotiation or switch configuration, where 802.3ad (mode 4) does.
0
 
LVL 61

Expert Comment

by:gheist
ID: 38735819
Mode 4 needs port-channel configured on cisco. It will not work without it (and it in turn needs some IOS hi-end options). Documentation says it needs  assistance form switch but does not detail enough.

You might  try mode 0 or mode 2. Modes 5 and 6 are for generic switches like unmanaged ones. 1 is for low-function network cards like 100Mbps or realtek. 3 is in the event you ever replace switch with hub (and as such it does not apply where gigabit is involved)
0
 

Author Closing Comment

by:spencerturbine
ID: 38790373
I think you all contributed but giltjr seemed to be the only one that suggested the mode that actually ended up working for me.
0

Featured Post

Free camera licenses with purchase of My Cloud NAS

Milestone Arcus software is compatible with thousands of industry-leading cameras for added flexibility. Upon installation on your My Cloud NAS, you will receive two (2) camera licenses already enabled in the software. And for a limited time, get additional camera licenses FREE.

Join & Write a Comment

Let’s list some of the technologies that enable smooth teleworking. 
PRTG Network Monitor lets you monitor your bandwidth usage, so you know who is using up your bandwidth, and what they're using it for.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now