Solved

tcpdump affecting iptables in linux

Posted on 2011-09-04
12
1,343 Views
Last Modified: 2012-05-12
I am trying to nat packets originating from within my network going to my external ip to a specific machine back within the same network

The reason for this, is software is setup to connect to the external address outside the network and I wish to speed it up by connecting locally within the network without changing the IP (and internal DNS isn't an option)

The strange thing is, if I monitor the connection with tcpdump, it all works. As soon as I turn off tcpdump, it stops working, the packets seem to never get SNAT'd
eg: tcpdump -i br0 host 10.0.0.2 and tcp port 81

Thinking tcpdump turns on promiscuous mode, i have tried turning on promisc mode on the internal network card, however that didn't help.

I have also monitored everything I could in /proc/sys/net/ to see if tcpdump modifies any of the files, however, I can't see any changes.

I have also cleared out every other rule in iptables to specifically test this one.

br0 is a bridge of my ethernet cards, with my external ip renamed to <extip>
iptables config:

Chain PREROUTING (policy ACCEPT 6256 packets, 817K bytes)
 pkts bytes target     prot opt in     out     source               destination        
 2020  121K DNAT       tcp  --  br0    *       0.0.0.0/0            <extip>      tcp dpt:6060 to:10.0.0.2:81

Chain INPUT (policy ACCEPT 5054 packets, 633K bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 1296 packets, 95746 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain POSTROUTING (policy ACCEPT 1337 packets, 101K bytes)
 pkts bytes target     prot opt in     out     source               destination        
  252 15552 SNAT       tcp  --  *      *       0.0.0.0/0            10.0.0.2            tcp dpt:81 to:10.0.0.1
0
Comment
Question by:DFPITC
  • 4
  • 3
  • 3
  • +1
12 Comments
 
LVL 16

Expert Comment

by:Blaz
ID: 36482400
Some questions:
- could you provide us with information (or a graph) of the network interfaces on this linux machine
- is this machine the default gateway for the internal network
- did you restart iptables after turning off tcpdump
- what are your iptables FORWARD chain rules
0
 

Author Comment

by:DFPITC
ID: 36482428
it has a ppp0 and br0(eth0,eth1,eth2)
yes, it is the default gateway
not sure what you mean by "restart iptables", iptables doesn't need restarting, it is in the kernel.
there are no forward rules (I assume you mean on the filter table) and the default policy is ACCEPT
0
 
LVL 16

Assisted Solution

by:Blaz
Blaz earned 100 total points
ID: 36482460
> not sure what you mean by "restart iptables"

I mean flush the iptables and reinsert the rules. In some distributions you can do this with "service iptables restart". On the other hand you could restart the entire system.

> there are no forward rules

Is this your firewall and you have no rules?

You could add "-j LOG" rules to see where the packet stops (or if it does not reach PREROUTING chain). But for the start you could zero out the counters ("iptables -Z") and see if they are incremented when you try to make a connection.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 36483359
I find it fascinating that by running tcpdump you can change the behaviour of the network. That is not supposed to happen.
One thing I have noticed with tcpdump is that it will display incoming packets that iptables drops.
On which system were you running tcpdump? (10.0.0.2 / 10.0.0.1 / other (please specify))
Which system has the rules you posted? (10.0.0.2 / 10.0.0.1 / other (please specify))
Your 2 rules appear to me to be a little asymmetric - I would have thought the POSTROUTING rule would map to 10.0.0.1:6060
0
 
LVL 39

Expert Comment

by:noci
ID: 36495195
@Duncan_roe
This is double nat...

External packets get first DNAT'ed to the internal server, and later they get SNATéd to the internal IP address of the router...
(Then you don't need to add routes to the nat gateway. If all protocols can handle this is a BIG if.... that depends on the protocol at hand.
And based on data the br0 interface has 10.0.0.1 as interface.  10.0.0.2 is the intended server.
This setup is generaly usable when the system doing NAT is NOT the gateway for this kind of traffic. But is should be avoided otherwise if possible.

so packets from x.x.x.x:???? -> ext-ip:6060 become:     10.0.0.1:???? -> 10.0.0.2:81
Iptables will take care of the return track.  The POSTROUTING you refer to is for firewall that need to nat all traffic from a local net to the internet at large.

Also promiscuous mode has to do with receipt of packets (a network adapter will filter on MAC address[few can be defined], unless Promiscuous mode is effective, then all packets get transferred to the driver).

@DFPITC
Did you also test with getting a bunch of transfers acros and start tcpdump after that?
0
 

Author Comment

by:DFPITC
ID: 36508219
@Blaz:
I have restarted the system with no luck, i've also cleared all iptables and routes and started from scratch.
I do have firewall rules, however, to simplify the problem I have cleared them all to test this.
for this test, I have no forward rules, however, the default policy is accept

@duncan_roe:
I agree it is strange that tcpdump can change the network's behavior. This is why I came here :)
I am running tcpdump on the same machine (the gateway: 10.0.0.1)
The two rules need to exist together for it to work, eg: DNAT forwards the source machine packets to the destination machine, however, if there was no SNAT rule, because the source machine is on the same network as the destination machine, the destination machine would reply to the source machine directly, not the gateway. The source machine expects packets back from the gateway, so it would ignore the packets directly from the destination machine. the SNAT rule changes the packet source to be the gateway so the destination machine replies to it, and the gateway replies back to the source machine.
Hope this makes sense.

@noci:
These rules are to forward internal packets only,

@all:
Just to clarify again what's going on:
There is software that runs on a mobile or laptop, it connects to www.mydomain.com:6060
www.mydomain.com is the gateway machine using 10.0.0.1 internally

External Access: (note I have disabled these rules for this test and have not included this in the example)
If the mobile or laptop is on the internet (outside the network) it connects to www.mydomain.com:6060
A rule is set to DNAT this to the internal server 10.0.0.2:81
This works properly as the source address for the packets will be set to an internet address and the default gateway for 10.0.0.2 is 10.0.0.1, so the packets get back to the gateway and back to the mobile / laptop.

Internal Access: (this is where the problem happens)
If the mobile or laptop is inside the network it still connects to www.mydomain.com:6060 (I can't change this)
Rather that it going out to the internet and back again (very slow) I want to simply tell the gateway to redirect all requests that are coming from inside the network (on interface br0) to the internal server on port 81 (10.0.0.2:81)
As stated before, in order to do this, i need two rules, a DNAT rule and an SNAT rule.

The problem is, this works fine, only while tcpdump is running

0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 39

Assisted Solution

by:noci
noci earned 100 total points
ID: 36508786
The SNAT is not needed if the default route on 10.0.0.2 points to 10.0.0.1
The SNAT only serves as a poormans routing fix.
You try to use U-turn routing, which is not quite a common practice.
It might help if the SNAT rule only matches source 10.0.0.0/24 packets instead of ALL packets.
iptstate might be of help here: http://www.phildev.net/iptstate/ in analysing the iptables architecture.

I see 2 solutions here:
Use the routing normaly Add an.interface to the gateway, hang the internal network on that (so the 10.0.0.2 machine is NOT on the local network anymore)

Use a proxy (f.e. squid ) that relays the request.

If you want non-intrusive tcpdumps, create a copy/monitor port through a managed switch or use a HUB (NOT a SWITCH) between the port & swtich
and monitor the traffic way.
0
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 100 total points
ID: 36509702
The question remains, why ever should tcpdump intrude?
As for the SNAT rule, do things work without it and with tcpdump running? I didn't think conntrack took care of SNAT. I would still be interested to know if also SNATting the port as I suggested before works any better.
0
 

Accepted Solution

by:
DFPITC earned 0 total points
ID: 36511655
ok.. I finally found the answer...
ifconfig br0 promisc

because promiscuous mode was off for the bridge, the network card on the gateway never received packets not destined for itself, because it was on the same network with a different IP, the gateway ignored the packets.. damn that was hard to diagnose.

also:

@noci: I can't explain it any more than I did in my previous post.. normally, you wouldn't need to DNAT/SNAT within your own network, however, in this case I do.

@duncan_roe: As I suspected, the only thing that tcpdump is turn promiscuous mode on, which ended up being the case.
0
 
LVL 39

Expert Comment

by:noci
ID: 36513600
The problem is why do you need promiscuous mode on the bridge.

How many interfaces are on the bridge.
If it is one it definitely should be working...
I assume more than one:
then are those interfaces all connected to ONE switch (or ONE switch stack) and are those interface configured as a channel (name depending on supplier).

If the switch is an unmanaged switch you are in trouble because the interface will flip traffic and data might end up on the wrong interface. In that case promiscuous mode repairs the direct problem.
A managed switch without configuring the channels will show flapping in the logs.
But it doesn't solve the trouble in your switch which might show itself as a rather slow interface...

If only one interface in the bridge is use i presume the need for promiscuous mode probably vanishes.













0
 

Author Closing Comment

by:DFPITC
ID: 36534837
Ok, even though I solved this myself, I'm awarding some points cas I want to thank you for the effort :)
0
 
LVL 16

Expert Comment

by:Blaz
ID: 36515334
Hi

> because promiscuous mode was off for the bridge, the network card on the gateway
> never received packets not destined for itself, because it was on the same network
> with a different IP, the gateway ignored the packets.. damn that was hard to diagnose.

Just for my curiosity - could you elaborate a bit on this. Where were the packets destined if not the gateway machine?
For us to undesrstand the setup it is very helpfull to see a network diagram of the machine - all interfaces, all IPs, where the internet is connected, where the not working machine is connected, where the source (test) machine is connected etc.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now