?
Solved

iptables pegging CPU?

Posted on 2007-07-24
16
Medium Priority
?
422 Views
Last Modified: 2012-08-13
Hi all.

Setting up a new box running Debian Etch, kernel 2.6.17, iptables v1.3.6.  Dual Opteron 248's, 2x1GB DDR 800 for RAM.  Decided to install PSAD for anal-retentive reporting, and it requires:

iptables -A INPUT -j LOG
iptables -A FORWARD -j LOG

As soon as I add the INPUT rule, network traffic slams down.  No other rules present on any chains.  Don't have console access atm, so I can't tell if it's a filtering issue or my CPUs are throttling, but executing:

iptables -D INPUT 1

clears that rule and traffic restores.  Can't seem to find any known issues or what I'm doing wrong, especially as that's the _only_ rule in any chain and seems to be the only variable here.  Any takers?  Nothing bizarre is showing in dmesg or syslog, as far as I can tell.
0
Comment
Question by:Rurne
  • 8
  • 8
16 Comments
 
LVL 7

Expert Comment

by:ezaton
ID: 19559992
The log rule logs each packet to your syslog subsystem. It means that the CPU load is caused (or could be caused) by syslog logging any packet which passes through. If your traffic is heavy, this is heavy.
0
 
LVL 9

Author Comment

by:Rurne
ID: 19560076
Right, but it's not.  In fact, in syslog, there were ~200 ipfilter events logged within the 10 minutes the gridlock occurred.  I have another machine in production use with the same setup and no issues at all.  I'm wondering if I need to go for a kernel upgrade, but I've had issues with building Areca RAID drivers for >=2.6.18 in the past, so I'd like to avoid that option if at all possible.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19560241
What is the heaviest process when this rule is active? What is the disk performance (the IO wait, especially) when this rule is active?
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 9

Author Comment

by:Rurne
ID: 19560256
Upping the point value to see about generating some interest.
0
 
LVL 9

Author Comment

by:Rurne
ID: 19560338
As far as I can tell, postfix.  Like I said, this is a remote deployment and I don't have console access, so when this locks down, I get a fresh on `top` once a minute.  I'll do a `cat /sys/block/sda/stat` every minute on cron and append it to a logfile to try to see what's going on here with I/O and put the results on ee-stuff in a bit.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19560379
Try running
iostat -kt 5 -x >> /tmp/iostat.out.txt

and maybe:
while true; do
   ps aux >> /tmp/ps.out.txt
   sleep 5
done
0
 
LVL 9

Author Comment

by:Rurne
ID: 19560502
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19560572
Nothing. What is the output of 'dmesg'? Anything interesting there?
What is the output, during such a process of
(just like before):

vmstat 1 >> /tmp/vmstat.out.txt
?
0
 
LVL 9

Author Comment

by:Rurne
ID: 19560653
FYI: dm-1 in iostat is a LV mapped to /var, so I'll try to keep a monitor on that.

Nothing unusual in dmesg; however, vmstat looks odd... big wait time on the CPU.

https://filedb.experts-exchange.com/incoming/ee-stuff/4139-vmstat.out.txt
0
 
LVL 7

Accepted Solution

by:
ezaton earned 1200 total points
ID: 19560683
Could it be some race-bug with your version of the kernel? What about, if you're not into an upgrade, a downgrade of the kernel?
0
 
LVL 9

Author Comment

by:Rurne
ID: 19560722
That's what my initial thought was, since I had to hand-tweak the RAID drivers and compile them in, but I've not had any real performance issues testing the RAID/LVM setups.  I just found it extremely odd how it'd lock on wait, when there were not but 50-100 blocks being written every ~10 seconds.

Looks like a kernel rebuild is in order.  Thanks.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19560729
Good luck!
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19565841
A thought came by - Could you capture traffic with tcpdump? It might be that your iptables is ignoring lots pf broadcasts, but still has to invest CPU time checking them? Do you have many broadcasts there?
0
 
LVL 9

Author Comment

by:Rurne
ID: 19567506
No.  I have another machine on the same subnet, similar configuration (difference being twice as much RAM and different RAID card).  I'm pretty convinced it's a driver/kernel issue, unless LVM is causing some unforeseen idle-wait issues.  Still haven't had any luck on the kernel downgrade as of yet, but I'm trying.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19567570
Never had such issues with LVM, especially that there is no IO involved in the process.
0
 
LVL 9

Author Comment

by:Rurne
ID: 19567646
Neither have I.  It's a complete anomaly, but I can't explain why we get the ~97% idle-wait lockout in vmstat when I flip the logging on, especially as this machine isn't even set into production roles yet while the other I've mentioned is DMZ'ed as a webserver and has no problem with pegging.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Fine Tune your automatic Updates for Ubuntu / Debian
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses

616 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question