johntsai90
asked on
Linux Networking
Hi,
We had this box as our internet gateway server. This Linux box has basically contained everything we need for running business such as firewall, dhcpd, httpd, squid, auth, or anything it need to protect from outsider. However, the way to use internet is initially an user will access to our server and accept for T/C user page in order to connect outside world. However, after some time if the users have been increased a lot, our server starts running off the resource and several processes might just hang.
1. Is any way I can know which process has reach its limitation? Such as firewall, httpd, dhcpd, or hardware overall such as memory are running low.
2. Sometimes, we need to put proxy address at user's laptop in order to by-pass our firewall to access user page or otherwise, user pages will run up for so long or sometimes time out. What is the problem actually?
3. When server are getting high usage or busy, new users might not get the response from our server, not even ping although the IP has been relocated by our server. Is that something wrong with squid, or port_redirector or http?
Thanks,
We had this box as our internet gateway server. This Linux box has basically contained everything we need for running business such as firewall, dhcpd, httpd, squid, auth, or anything it need to protect from outsider. However, the way to use internet is initially an user will access to our server and accept for T/C user page in order to connect outside world. However, after some time if the users have been increased a lot, our server starts running off the resource and several processes might just hang.
1. Is any way I can know which process has reach its limitation? Such as firewall, httpd, dhcpd, or hardware overall such as memory are running low.
2. Sometimes, we need to put proxy address at user's laptop in order to by-pass our firewall to access user page or otherwise, user pages will run up for so long or sometimes time out. What is the problem actually?
3. When server are getting high usage or busy, new users might not get the response from our server, not even ping although the IP has been relocated by our server. Is that something wrong with squid, or port_redirector or http?
Thanks,
Squid or httpd are probably the culprits. Running top -b > processlog will give you an idea of the resource hog (leave it running and look at processlog after a suitable period of time).
ASKER
How can I identify which process has cause the resource off by doing so?
As far as my observation, I haven't seen anything related to httpd or squid.
here is part of output
8:03pm up 7 days, 9:10, 2 users, load average: 2.75, 3.13, 2.97
197 processes: 188 sleeping, 7 running, 0 zombie, 2 stopped
CPU states: 71.9% user, 26.8% system, 1.1% nice, 0.0% idle
Mem: 392736K av, 381836K used, 10900K free, 0K shrd, 154240K buff
Swap: 72256K av, 0K used, 72256K free 95452K cached
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
10369 mysql 13 0 20996 20M 1640 R 0 25.8 5.3 213:00 mysqld
576 mysql 5 0 20996 20M 1640 S 0 20.3 5.3 770:07 mysqld
578 root 3 0 1892 1892 1304 S 0 12.4 0.4 392:02 billcalc
11799 mysql 13 0 20996 20M 1640 R 0 12.4 5.3 62:44 mysqld
10379 root 12 0 2604 2604 1660 R 0 10.8 0.6 83:50 registration
10374 mysql 10 0 20996 20M 1640 R 0 7.6 5.3 24:55 mysqld
11797 root 4 0 2004 2004 1396 R 0 2.3 0.5 14:00 firewall
10378 root 1 0 2604 2604 1660 S 0 1.7 0.6 10:25 registration
4307 root 3 0 1128 1128 852 R 0 1.1 0.2 0:00 top
32355 mysql 0 0 20996 20M 1640 S 0 0.7 5.3 9:09 mysqld
493 root 0 0 592 592 492 S 0 0.7 0.1 14:23 syslogd
11798 mysql 1 0 20996 20M 1640 S 0 0.5 5.3 4:49 mysqld
10372 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:41 mysqld
10376 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:14 mysqld
10387 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:12 mysqld
1 root 0 0 472 472 408 S 0 0.0 0.1 0:05 init
2 root 0 0 0 0 0 SW 0 0.0 0.0 0:00 kflushd
3 root 0 0 0 0 0 SW 0 0.0 0.0 1:06 kupdate
As far as my observation, I haven't seen anything related to httpd or squid.
here is part of output
8:03pm up 7 days, 9:10, 2 users, load average: 2.75, 3.13, 2.97
197 processes: 188 sleeping, 7 running, 0 zombie, 2 stopped
CPU states: 71.9% user, 26.8% system, 1.1% nice, 0.0% idle
Mem: 392736K av, 381836K used, 10900K free, 0K shrd, 154240K buff
Swap: 72256K av, 0K used, 72256K free 95452K cached
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
10369 mysql 13 0 20996 20M 1640 R 0 25.8 5.3 213:00 mysqld
576 mysql 5 0 20996 20M 1640 S 0 20.3 5.3 770:07 mysqld
578 root 3 0 1892 1892 1304 S 0 12.4 0.4 392:02 billcalc
11799 mysql 13 0 20996 20M 1640 R 0 12.4 5.3 62:44 mysqld
10379 root 12 0 2604 2604 1660 R 0 10.8 0.6 83:50 registration
10374 mysql 10 0 20996 20M 1640 R 0 7.6 5.3 24:55 mysqld
11797 root 4 0 2004 2004 1396 R 0 2.3 0.5 14:00 firewall
10378 root 1 0 2604 2604 1660 S 0 1.7 0.6 10:25 registration
4307 root 3 0 1128 1128 852 R 0 1.1 0.2 0:00 top
32355 mysql 0 0 20996 20M 1640 S 0 0.7 5.3 9:09 mysqld
493 root 0 0 592 592 492 S 0 0.7 0.1 14:23 syslogd
11798 mysql 1 0 20996 20M 1640 S 0 0.5 5.3 4:49 mysqld
10372 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:41 mysqld
10376 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:14 mysqld
10387 mysql 0 0 20996 20M 1640 S 0 0.1 5.3 0:12 mysqld
1 root 0 0 472 472 408 S 0 0.0 0.1 0:05 init
2 root 0 0 0 0 0 SW 0 0.0 0.0 0:00 kflushd
3 root 0 0 0 0 0 SW 0 0.0 0.0 1:06 kupdate
It looks to me like you're running some intense transactions on your mysql database. It's accounting for about 65% of your CP usage on this snapshot. You might consider moving that db to another box.
The issue in my opinion is mysql.
However...
You might also check into using stateful firewall rules and look at where in your firewall script you rules are being executed. For instance, if you run
iptables -v -n -L --line-numbers
you'll get a large printout of the 3 sections of rules (INPUT, OUTPUT and FORWARD) Within each section the majority of the packets should be executed at the top of the list. If not, then you are wasting processor cycles. Just find the one that is being executed the most and move the actual rule to the top of that section of your firewall rules. Also, make use of ESTABLISHED, RELATED.
Also, it looks like you have 384MB of memory in your box. That doesn't seem like much considering the amount of processes running on the box. Even though the swap space isn't being used (which I find suspcious) you should consider adding memory to the server.
The issue in my opinion is mysql.
However...
You might also check into using stateful firewall rules and look at where in your firewall script you rules are being executed. For instance, if you run
iptables -v -n -L --line-numbers
you'll get a large printout of the 3 sections of rules (INPUT, OUTPUT and FORWARD) Within each section the majority of the packets should be executed at the top of the list. If not, then you are wasting processor cycles. Just find the one that is being executed the most and move the actual rule to the top of that section of your firewall rules. Also, make use of ESTABLISHED, RELATED.
Also, it looks like you have 384MB of memory in your box. That doesn't seem like much considering the amount of processes running on the box. Even though the swap space isn't being used (which I find suspcious) you should consider adding memory to the server.
ASKER
How do we make use of ESTABLISHED, RELATED on firewall? Any recommandation of how to use swap space as well?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
How do I check if certain resource such as firewall are reached certain limit of its capacity before I can restart it?
I'm not aware of resource limits for a firewall. I have had a box running for a year without rebooting or restarting a service. Of course, you can restart a service any time you want. If you use the defualt iptables setup then type service iptables restart and it will clear all the rules then reapply what you have saved. There are also plently of good scripts that you can adopt from the internet that may suit your purposes. Just find one that might come close then modify it to you liking.
Thanks for the points.
Thanks for the points.