?
Solved

No IP traffic possible: ip_conntrack: table full.

Posted on 2002-05-21
8
Medium Priority
?
1,540 Views
Last Modified: 2010-05-18
Since yesterday, my Linux server writes mad things to its logfile (/var/log/messages). Not even a ping to the local IP addresses of the proxy is possible when the problem occurs, not even from its own local console.
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.

Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)

What can I do?

This is an extract from the log file:

May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.

My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?

My SNORT logs are completely(!) empty.

Please help!
0
Comment
Question by:zzconsumer
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
8 Comments
 
LVL 4

Expert Comment

by:MFCRich
ID: 7026805
I'm no Squid expert but is it possible that Squid is establishing lots of NAT'ed connections and keeping them open much longer than required? Perhaps there are some Squid config directives that control this.
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7032956
I'm not sure if that's a possibility. I didn't find anything according to that in the squid documentation.
0
 
LVL 16

Expert Comment

by:The--Captain
ID: 7036725
When this is happening, what does

cat /proc/net/ip_conntrack

tell you?  Also, adjusting the value of

/proc/sys/net/ipv4/ip_conntrack_max

may help.

Cheers,
-Jon
0
Use Filtering Commands to Process Files in Linux

Learn how to manipulate data with the help of various filtering commands such as `cat`, `fmt`, `pr`, and others in Linux.

 
LVL 1

Author Comment

by:zzconsumer
ID: 7036762
A few days ago, I set ip_conntrack_max to 65535. After that the problem did not occur again. Anyway, I have taken a look into /proc/net/ip_conntrack. This "file" is about 3 MBs(!) long at the moment, and still contains connections marked as ESTABLISHED that should heve been closed at least 12 hours ago. This might cause the problem, so I think the question should be how can I prevent IP_CONNTRACK to run out of its limit again, e.g. by closing the connections when they are not needed any more?

All lines in ip_conntrack look like this:
tcp      6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
0
 
LVL 16

Accepted Solution

by:
The--Captain earned 300 total points
ID: 7036791
With ipchains, you could specify the timeout values for NAT entries, since it had no stateful translation engine, and hence could not determine when you were done using an entry - I imagine the thinking behind iptables is that conntrack is supoosed to notice when connections go away, and remove the entries automatically - have there been some recently implemented firewall rules (or maybe some MTU problems) that are preventing the delivery of FIN packets to your clients?

BTW, is it really src and dest ports of 1234, or are you just munging that?  If so, please only munge the first 2 or 3 octets in the address, and leave everything else the same.  If You haven't munged much, then there is definitely something weird going on - you should not be seeing such obvious patterns of 12345678 in your conntrack table...

Cheers,
-Jon
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7036822
Sorry, I'm munging that because I was told never to give any true logs to the public... I only some numbers. ;-)
The entry itself looks OK, so source and destination ports are different as they should be.

Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.

I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.

I hope this information helps a bit.

0
 
LVL 16

Expert Comment

by:The--Captain
ID: 7086373
I think you might be able to set up a logging rule in iptables that will log FIN packets - that will at least tell you if they are being received - I will have do do a bit more reading on conntrack befiore I can absolutely determine all cases (or at least identify most of them) in which stale translation entries remain.

Cheers,
-Jon
0
 
LVL 1

Author Comment

by:zzconsumer
ID: 7355003
Using IPCHAINS for a while and found this question still opened.
Thanks for your time!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Do you want to know how to make a graph with Microsoft Access? First, create a query with the data for the chart. Then make a blank form and add a chart control. This video also shows how to change what data is displayed on the graph as well as form…
Suggested Courses
Course of the Month12 days, 23 hours left to enroll

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question