zzconsumer
asked on
No IP traffic possible: ip_conntrack: table full.
Since yesterday, my Linux server writes mad things to its logfile (/var/log/messages). Not even a ping to the local IP addresses of the proxy is possible when the problem occurs, not even from its own local console.
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.
Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)
What can I do?
This is an extract from the log file:
May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.
My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?
My SNORT logs are completely(!) empty.
Please help!
I searched many websites about this, but the only thing I found out is that many others have the same problem.
When I restart the server, I have several hours left until the problem occurs again. Before, about six months the server ran like hell. I made no configuration changes the last 3 months except some squid acls, which shouldn't have anything to do with this problem.
Has my server been hacked? Is someone running a DoS to my server? (768kbit/s DSL, 350Mhz AMD K6-2, 256MB of RAM)
What can I do?
This is an extract from the log file:
May 20 18:21:56 slxgw kernel: NET: 4 messages suppressed.
May 20 18:21:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:22:16 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:23:59 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:25:06 slxgw last message repeated 10 times
May 20 18:25:31 slxgw last message repeated 9 times
May 20 18:27:07 slxgw last message repeated 4 times
May 20 18:28:06 slxgw last message repeated 4 times
May 20 18:29:10 slxgw last message repeated 8 times
May 20 18:29:58 slxgw last message repeated 9 times
May 20 18:33:34 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:36 slxgw last message repeated 17 times
May 20 18:34:37 slxgw last message repeated 3 times
May 20 18:34:44 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:44 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:46 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:46 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:56 slxgw kernel: NET: 1 messages suppressed.
May 20 18:34:56 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:34:58 slxgw kernel: ip_conntrack: table full, dropping packet.
May 20 18:35:00 slxgw kernel: NET: 1 messages suppressed.
May 20 18:35:00 slxgw kernel: ip_conntrack: table full, dropping packet.
My ip_conntrack stack size is 16348. Since my machine has 256 MB of memory, I think I might use a higher value, but I think then the message may occur only later and the source is not fixed?
My SNORT logs are completely(!) empty.
Please help!
I'm no Squid expert but is it possible that Squid is establishing lots of NAT'ed connections and keeping them open much longer than required? Perhaps there are some Squid config directives that control this.
ASKER
I'm not sure if that's a possibility. I didn't find anything according to that in the squid documentation.
When this is happening, what does
cat /proc/net/ip_conntrack
tell you? Also, adjusting the value of
/proc/sys/net/ipv4/ip_conn track_max
may help.
Cheers,
-Jon
cat /proc/net/ip_conntrack
tell you? Also, adjusting the value of
/proc/sys/net/ipv4/ip_conn
may help.
Cheers,
-Jon
ASKER
A few days ago, I set ip_conntrack_max to 65535. After that the problem did not occur again. Anyway, I have taken a look into /proc/net/ip_conntrack. This "file" is about 3 MBs(!) long at the moment, and still contains connections marked as ESTABLISHED that should heve been closed at least 12 hours ago. This might cause the problem, so I think the question should be how can I prevent IP_CONNTRACK to run out of its limit again, e.g. by closing the connections when they are not needed any more?
All lines in ip_conntrack look like this:
tcp 6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
All lines in ip_conntrack look like this:
tcp 6 123456 ESTABLISHED src=192.168.x.y dst=1.2.3.4 sport=1234 dport=1234 src=5.6.7.8 dst=192.168.a.b sport=5678 dport=9012 [ASSURED] use=1
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Sorry, I'm munging that because I was told never to give any true logs to the public... I only some numbers. ;-)
The entry itself looks OK, so source and destination ports are different as they should be.
Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.
I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.
I hope this information helps a bit.
The entry itself looks OK, so source and destination ports are different as they should be.
Ehem... I'm really no IP Pro, so I just understand half of your question. I only have some basic knowledge.
I am using Kernel 2.4.17, so I'm using IPTables.
All traffic to my clients is allowed as long the destination port is higher than 1024.
The clients themselfes are allowed to connect to any port, except for port 80 is being redirected to 3128, the port on which Squid runs.
There's some incoming traffic allowed below port 1024, depending on some services running at standard ports. All this traffic is not redirected to clients, but redirected to specific servers.
I hope this information helps a bit.
I think you might be able to set up a logging rule in iptables that will log FIN packets - that will at least tell you if they are being received - I will have do do a bit more reading on conntrack befiore I can absolutely determine all cases (or at least identify most of them) in which stale translation entries remain.
Cheers,
-Jon
Cheers,
-Jon
ASKER
Using IPCHAINS for a while and found this question still opened.
Thanks for your time!
Thanks for your time!