Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1575
  • Last Modified:

No IP traffic possible: ip_conntrack: table full.

Since yesterday, my Linux server writes mad things to its logfile (/var/log/messages). Not even a ping to the local IP addresses of the proxy is possible when the problem occurs, not even from its own local console.
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.

Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)

What can I do?

This is an extract from the log file:

May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.

My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?

My SNORT logs are completely(!) empty.

Please help!
0
zzconsumer
Asked:
zzconsumer
  • 4
  • 3
1 Solution
 
MFCRichCommented:
I'm no Squid expert but is it possible that Squid is establishing lots of NAT'ed connections and keeping them open much longer than required? Perhaps there are some Squid config directives that control this.
0
 
zzconsumerAuthor Commented:
I'm not sure if that's a possibility. I didn't find anything according to that in the squid documentation.
0
 
The--CaptainCommented:
When this is happening, what does

cat /proc/net/ip_conntrack

tell you?  Also, adjusting the value of

/proc/sys/net/ipv4/ip_conntrack_max

may help.

Cheers,
-Jon
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
zzconsumerAuthor Commented:
A few days ago, I set ip_conntrack_max to 65535. After that the problem did not occur again. Anyway, I have taken a look into /proc/net/ip_conntrack. This "file" is about 3 MBs(!) long at the moment, and still contains connections marked as ESTABLISHED that should heve been closed at least 12 hours ago. This might cause the problem, so I think the question should be how can I prevent IP_CONNTRACK to run out of its limit again, e.g. by closing the connections when they are not needed any more?

All lines in ip_conntrack look like this:
tcp      6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
0
 
The--CaptainCommented:
With ipchains, you could specify the timeout values for NAT entries, since it had no stateful translation engine, and hence could not determine when you were done using an entry - I imagine the thinking behind iptables is that conntrack is supoosed to notice when connections go away, and remove the entries automatically - have there been some recently implemented firewall rules (or maybe some MTU problems) that are preventing the delivery of FIN packets to your clients?

BTW, is it really src and dest ports of 1234, or are you just munging that?  If so, please only munge the first 2 or 3 octets in the address, and leave everything else the same.  If You haven't munged much, then there is definitely something weird going on - you should not be seeing such obvious patterns of 12345678 in your conntrack table...

Cheers,
-Jon
0
 
zzconsumerAuthor Commented:
Sorry, I'm munging that because I was told never to give any true logs to the public... I only some numbers. ;-)
The entry itself looks OK, so source and destination ports are different as they should be.

Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.

I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.

I hope this information helps a bit.

0
 
The--CaptainCommented:
I think you might be able to set up a logging rule in iptables that will log FIN packets - that will at least tell you if they are being received - I will have do do a bit more reading on conntrack befiore I can absolutely determine all cases (or at least identify most of them) in which stale translation entries remain.

Cheers,
-Jon
0
 
zzconsumerAuthor Commented:
Using IPCHAINS for a while and found this question still opened.
Thanks for your time!
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now