No IP traffic possible: ip_conntrack: table full.

zzconsumer
zzconsumer used Ask the Experts™
on
Since yesterday, my Linux server writes mad things to its logfile (/var/log/messages). Not even a ping to the local IP addresses of the proxy is possible when the problem occurs, not even from its own local console.
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.

Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)

What can I do?

This is an extract from the log file:

May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.

My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?

My SNORT logs are completely(!) empty.

Please help!
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Commented:
I'm no Squid expert but is it possible that Squid is establishing lots of NAT'ed connections and keeping them open much longer than required? Perhaps there are some Squid config directives that control this.

Author

Commented:
I'm not sure if that's a possibility. I didn't find anything according to that in the squid documentation.
When this is happening, what does

cat /proc/net/ip_conntrack

tell you?  Also, adjusting the value of

/proc/sys/net/ipv4/ip_conntrack_max

may help.

Cheers,
-Jon
Build an E-Commerce Site with Angular 5

Learn how to build an E-Commerce site with Angular 5, a JavaScript framework used by developers to build web, desktop, and mobile applications.

Author

Commented:
A few days ago, I set ip_conntrack_max to 65535. After that the problem did not occur again. Anyway, I have taken a look into /proc/net/ip_conntrack. This "file" is about 3 MBs(!) long at the moment, and still contains connections marked as ESTABLISHED that should heve been closed at least 12 hours ago. This might cause the problem, so I think the question should be how can I prevent IP_CONNTRACK to run out of its limit again, e.g. by closing the connections when they are not needed any more?

All lines in ip_conntrack look like this:
tcp      6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
With ipchains, you could specify the timeout values for NAT entries, since it had no stateful translation engine, and hence could not determine when you were done using an entry - I imagine the thinking behind iptables is that conntrack is supoosed to notice when connections go away, and remove the entries automatically - have there been some recently implemented firewall rules (or maybe some MTU problems) that are preventing the delivery of FIN packets to your clients?

BTW, is it really src and dest ports of 1234, or are you just munging that?  If so, please only munge the first 2 or 3 octets in the address, and leave everything else the same.  If You haven't munged much, then there is definitely something weird going on - you should not be seeing such obvious patterns of 12345678 in your conntrack table...

Cheers,
-Jon

Author

Commented:
Sorry, I'm munging that because I was told never to give any true logs to the public... I only some numbers. ;-)
The entry itself looks OK, so source and destination ports are different as they should be.

Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.

I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.

I hope this information helps a bit.

I think you might be able to set up a logging rule in iptables that will log FIN packets - that will at least tell you if they are being received - I will have do do a bit more reading on conntrack befiore I can absolutely determine all cases (or at least identify most of them) in which stale translation entries remain.

Cheers,
-Jon

Author

Commented:
Using IPCHAINS for a while and found this question still opened.
Thanks for your time!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial