[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 281
  • Last Modified:

Linux IP tables down


I have a server handling the incoming requests from Internet. The server is running Centos 5.5, and the IP tables were going down unexpectedly. The log file is attached.

Any advice about why the tables were down is highly appreciated!

  • 4
  • 2
1 Solution
To begin with, your log tells me that your IPTables is receiving more requests than it can track. When enabled through the use of NAT or other stateful inspection rules, netfilter (iptables) under Linux maintains a list of connections passing through the router. Each connection tracking entry contains defined characteristics of the packet, including the source and destination IP address and port number.

The connection tracking entries are ultimately stored in a hash table with a fixed size. For routers with stateful inspection enabled, the number of connections to track may exceed the total number of connections available in the table. If the router reaches the maximum number of connection tracking entries, it will log an error:

"ip_conntrack: table full, dropping packet"

each time that it is unable to store an entry in the connection tracking table. Each instance of this message represents a connection that the router has discarded, typically meaning that the user whose connection was dropped must re-establish their connection.

The maximum size of the connection tracking table can be increased. The maximum size value is stored in the router's proc filesystem in the file /proc/sys/net/ipv4/ip_conntrack_max. Increasing the maximum size of the connection tracking table to a value larger than the total number fo connections will eliminate the error message and prevent the router from dropping connections due to a lack of space in the connection tracking table.

Each connection uses approximately 350 bytes of memory, so 16384 connections would allocate 5.7 MB of RAM.

Following are commands, which you can use to check current configuration, as well as to modify the same.

# This tell you how many sessions arte open right now.
cat /proc/net/ip_conntrack | wc -l
# This tells you the maximum number of conntrack entries you can have in
cat /proc/sys/net/ipv4/ip_conntrack_max

Once the previous number hits beyond the latter, you should start seeing
these messages. I would increase the latter number by calling:

echo "<some_bigger_number>" > /proc/sys/net/ipv4/ip_conntrack_max

or if you want it to span reboots, you can place the following in

sys.net.ipv4.ip_conntrack_max =3D <some_big_number>

Hope this helps!!

Also, on going further down the log I discovered that you are using Linux version 2.6.18-194.el5xen, which is buggy.

I would suggest you to update the kernel to minimum of 2.6.18-238.

Please use yum upgrade kernel-xenp command to upgrade your kernel, considering you are currently using a xen kernel.

wsyyAuthor Commented:
our website was under attacked, but iptables went down after that attack and when there were only a few incoming requests.

I wonder why the tables went down after the attack but during the attack.
Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

If it was not too long after the attack, it is possible that state information was still being maintained in the Iptables tracking database.

wsyyAuthor Commented:

While I am asking you the question, new issues came out (please see the log in the code snippet). I would think they are related.

Please help advise Thanks.
INFO: task syslogd:1013 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslogd       D ffff880001005460     0  1013      1          1016  1005 (NOTLB)
 ffff8800b8b6bd88  0000000000000282  0000000000000000  0000000000000001 
 0000000000000009  ffff8800bde5b7a0  ffffffff804f4b80  00000000000015f1 
 ffff8800bde5b988  ffff8800bfb9b860 
Call Trace:
 [<ffffffff88036d5a>] :jbd:log_wait_commit+0xa3/0xf5
 [<ffffffff8029c60a>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
 [<ffffffff8023138e>] __writeback_single_inode+0x1e9/0x328
 [<ffffffff802d2ff1>] do_readv_writev+0x26e/0x291
 [<ffffffff802e555b>] sync_inode+0x24/0x33
 [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
 [<ffffffff80252276>] do_fsync+0x52/0xa4
 [<ffffffff802d37f5>] __do_fsync+0x23/0x36
 [<ffffffff802602f9>] tracesys+0xab/0xb6

Open in new window

I would be glad to help, but first may I request you to
1. kindly upgrade your kernel!
2. Run a scan for possible rootkits on your system. There are two popular free tools for RootKit Hunting on Linux Environment.

- Rootkit Hunter, and
- ChkRootKit

Both of these list in Top 4 Root Kit Hunters

You can find instructions for installation on their home pages and as well as here http://www.cyberciti.biz/faq/howto-check-linux-rootkist-with-detectors-software/.

One you have done this!  We can look at any remaining issues.


Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now