Linux IP tables down

Posted on 2011-10-29
Last Modified: 2012-06-27

I have a server handling the incoming requests from Internet. The server is running Centos 5.5, and the IP tables were going down unexpectedly. The log file is attached.

Any advice about why the tables were down is highly appreciated!

Question by:wsyy
    LVL 9

    Expert Comment

    To begin with, your log tells me that your IPTables is receiving more requests than it can track. When enabled through the use of NAT or other stateful inspection rules, netfilter (iptables) under Linux maintains a list of connections passing through the router. Each connection tracking entry contains defined characteristics of the packet, including the source and destination IP address and port number.

    The connection tracking entries are ultimately stored in a hash table with a fixed size. For routers with stateful inspection enabled, the number of connections to track may exceed the total number of connections available in the table. If the router reaches the maximum number of connection tracking entries, it will log an error:

    "ip_conntrack: table full, dropping packet"

    each time that it is unable to store an entry in the connection tracking table. Each instance of this message represents a connection that the router has discarded, typically meaning that the user whose connection was dropped must re-establish their connection.

    The maximum size of the connection tracking table can be increased. The maximum size value is stored in the router's proc filesystem in the file /proc/sys/net/ipv4/ip_conntrack_max. Increasing the maximum size of the connection tracking table to a value larger than the total number fo connections will eliminate the error message and prevent the router from dropping connections due to a lack of space in the connection tracking table.

    Each connection uses approximately 350 bytes of memory, so 16384 connections would allocate 5.7 MB of RAM.

    Following are commands, which you can use to check current configuration, as well as to modify the same.

    # This tell you how many sessions arte open right now.
    cat /proc/net/ip_conntrack | wc -l
    # This tells you the maximum number of conntrack entries you can have in
    cat /proc/sys/net/ipv4/ip_conntrack_max

    Once the previous number hits beyond the latter, you should start seeing
    these messages. I would increase the latter number by calling:

    echo "<some_bigger_number>" > /proc/sys/net/ipv4/ip_conntrack_max

    or if you want it to span reboots, you can place the following in
    /etc/sysctl.conf =3D <some_big_number>

    Hope this helps!!

    LVL 9

    Expert Comment

    Also, on going further down the log I discovered that you are using Linux version 2.6.18-194.el5xen, which is buggy.

    I would suggest you to update the kernel to minimum of 2.6.18-238.

    Please use yum upgrade kernel-xenp command to upgrade your kernel, considering you are currently using a xen kernel.


    Author Comment

    our website was under attacked, but iptables went down after that attack and when there were only a few incoming requests.

    I wonder why the tables went down after the attack but during the attack.
    LVL 9

    Expert Comment

    If it was not too long after the attack, it is possible that state information was still being maintained in the Iptables tracking database.


    Author Comment


    While I am asking you the question, new issues came out (please see the log in the code snippet). I would think they are related.

    Please help advise Thanks.
    INFO: task syslogd:1013 blocked for more than 120 seconds.
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    syslogd       D ffff880001005460     0  1013      1          1016  1005 (NOTLB)
     ffff8800b8b6bd88  0000000000000282  0000000000000000  0000000000000001 
     0000000000000009  ffff8800bde5b7a0  ffffffff804f4b80  00000000000015f1 
     ffff8800bde5b988  ffff8800bfb9b860 
    Call Trace:
     [<ffffffff88036d5a>] :jbd:log_wait_commit+0xa3/0xf5
     [<ffffffff8029c60a>] autoremove_wake_function+0x0/0x2e
     [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
     [<ffffffff8023138e>] __writeback_single_inode+0x1e9/0x328
     [<ffffffff802d2ff1>] do_readv_writev+0x26e/0x291
     [<ffffffff802e555b>] sync_inode+0x24/0x33
     [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
     [<ffffffff80252276>] do_fsync+0x52/0xa4
     [<ffffffff802d37f5>] __do_fsync+0x23/0x36
     [<ffffffff802602f9>] tracesys+0xab/0xb6

    Open in new window

    LVL 9

    Accepted Solution

    I would be glad to help, but first may I request you to
    1. kindly upgrade your kernel!
    2. Run a scan for possible rootkits on your system. There are two popular free tools for RootKit Hunting on Linux Environment.

    - Rootkit Hunter, and
    - ChkRootKit

    Both of these list in Top 4 Root Kit Hunters

    You can find instructions for installation on their home pages and as well as here

    One you have done this!  We can look at any remaining issues.


    Featured Post

    Highfive + Dolby Voice = No More Audio Complaints!

    Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

    Join & Write a Comment

    This is the error message I got (CODE) Error caused by incompatible libmp3lame 3.98-2 with ffmpeg I've googled this error message and found out sometimes it attaches this note "can be treated with downgrade libmp3lame to version 3.97 or 3.98" …
    SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
    Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
    Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

    733 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    23 Experts available now in Live!

    Get 1:1 Help Now