Solved

No IP traffic possible: ip_conntrack: table full.

Posted on 2002-05-21
8
1,497 Views
Last Modified: 2010-05-18
Since yesterday, my Linux server writes mad things to its logfile (/var/log/messages). Not even a ping to the local IP addresses of the proxy is possible when the problem occurs, not even from its own local console.
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.

Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)

What can I do?

This is an extract from the log file:

May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.

My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?

My SNORT logs are completely(!) empty.

Please help!
0
Comment
Question by:zzconsumer
  • 4
  • 3
8 Comments
 
LVL 4

Expert Comment

by:MFCRich
ID: 7026805
I'm no Squid expert but is it possible that Squid is establishing lots of NAT'ed connections and keeping them open much longer than required? Perhaps there are some Squid config directives that control this.
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7032956
I'm not sure if that's a possibility. I didn't find anything according to that in the squid documentation.
0
 
LVL 16

Expert Comment

by:The--Captain
ID: 7036725
When this is happening, what does

cat /proc/net/ip_conntrack

tell you?  Also, adjusting the value of

/proc/sys/net/ipv4/ip_conntrack_max

may help.

Cheers,
-Jon
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7036762
A few days ago, I set ip_conntrack_max to 65535. After that the problem did not occur again. Anyway, I have taken a look into /proc/net/ip_conntrack. This "file" is about 3 MBs(!) long at the moment, and still contains connections marked as ESTABLISHED that should heve been closed at least 12 hours ago. This might cause the problem, so I think the question should be how can I prevent IP_CONNTRACK to run out of its limit again, e.g. by closing the connections when they are not needed any more?

All lines in ip_conntrack look like this:
tcp      6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 16

Accepted Solution

by:
The--Captain earned 100 total points
ID: 7036791
With ipchains, you could specify the timeout values for NAT entries, since it had no stateful translation engine, and hence could not determine when you were done using an entry - I imagine the thinking behind iptables is that conntrack is supoosed to notice when connections go away, and remove the entries automatically - have there been some recently implemented firewall rules (or maybe some MTU problems) that are preventing the delivery of FIN packets to your clients?

BTW, is it really src and dest ports of 1234, or are you just munging that?  If so, please only munge the first 2 or 3 octets in the address, and leave everything else the same.  If You haven't munged much, then there is definitely something weird going on - you should not be seeing such obvious patterns of 12345678 in your conntrack table...

Cheers,
-Jon
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7036822
Sorry, I'm munging that because I was told never to give any true logs to the public... I only some numbers. ;-)
The entry itself looks OK, so source and destination ports are different as they should be.

Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.

I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.

I hope this information helps a bit.

0
 
LVL 16

Expert Comment

by:The--Captain
ID: 7086373
I think you might be able to set up a logging rule in iptables that will log FIN packets - that will at least tell you if they are being received - I will have do do a bit more reading on conntrack befiore I can absolutely determine all cases (or at least identify most of them) in which stale translation entries remain.

Cheers,
-Jon
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7355003
Using IPCHAINS for a while and found this question still opened.
Thanks for your time!
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now