?
Solved

High CPU usage / for apache

Posted on 2006-04-18
28
Medium Priority
?
7,438 Views
Last Modified: 2007-12-19
I run top -c  and I see this :

 PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
29148 apache    25   0  1124 1084   640 R    86,6  0,1 308:04   0 /var/tmp/httpd

the cpu usage is very high..and it seems strange to me of what the httpd file is doing inside the /var/tmp/ folder..
could this be any  kind of hacking ?




0
Comment
Question by:MaRiOsGR
  • 9
  • 8
  • 5
  • +2
27 Comments
 
LVL 14

Expert Comment

by:pablouruguay
ID: 16477251
yep.
the httpd  was in /usr/sbin/httpd

when the httpd or another service execute inside tmp i really think is a hacking work.

check in /var/log/message to see any extrange and execute the command last to see the last loggings but if you dont have a soft to detect intrutions like tripwire or another is really difficult to see what haoppend in your system after a intrution.

i recommend format and install again. an update SO with a firewall
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16483615
Probably you are hacked.
You may see what is your fake httpd doing:
strace -p 29148
But probably it's some kind of computation (due to high CPU usage) and you may see very little or now system calls.
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16484609
download and run rootkit hunter:
http://www.rootkit.nl/

it will let you know if some known rootkit is already in place.

and consider reinstall of the full box and install a current version of your distribution.

0
Get quick recovery of individual SharePoint items

Free tool – Veeam Explorer for Microsoft SharePoint, enables fast, easy restores of SharePoint sites, documents, libraries and lists — all with no agents to manage and no additional licenses to buy.

 
LVL 3

Accepted Solution

by:
jglyons earned 2000 total points
ID: 16492536

That is certainly a hack.
You need to look at

lsof -p 29148 to see what files are being used.

netstat -pan | grep 29148 will show you want port it's listening on.

Using a rootkit hunter is a good start but you'll almost certainly find that you've got an IRC bot running.

Chances are you've got a forum or CMS package on the server thats out of date and has vulnerabilities within it.

Something like mod_security is a good package to add on to try and prevent basic hacks in future.
0
 
LVL 14

Expert Comment

by:pablouruguay
ID: 16669978
and the split??? all points was give to jglyons.. i don't understand
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16671080
I understand mariosgr accepted his answer at the end

it would have been good to read a final comment from him, though
0
 
LVL 14

Expert Comment

by:pablouruguay
ID: 16671148
ohhhhh i understand now... thanks gabriel for clarification........
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16671184
anytime
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16685027
The httpd script u see running from /var/tmp/ is not the apache.
that bot or hacker uploads through some hole , scripts with false names like "named" or "httpd" so I wont see them at first sight.

The strange thinng is how is he or able to run these scripts...
i thinkg that the main source of evil are some phpbb forums that the server has.

Also that "hacker" keeps uploading psyBNC scripts for spamming ofcourse, so I'm blocking through th firewall all the outgoung traffic to undernet etc.

The only problem is that the red hat version we use doesnt have the "lsof" command but im working on it.

p.s.

The manage to update and run scripts on these folders:

/tmp/
/var/tmp/
/dev/shm/
all files are made by user "apache"
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16685801
MariosGR: Yes, in fact, phpbb has had a lot of vulnerabilities.

just upgrade and you should get rid of these new scripts

but once hacked, you cannot trust your system, so... reinstall and upgrade *retty* often.

you can repartition your system and mount /tmp and /var/tmp on a different partition, with atributes that disallow execution. this way you can get rid of most of automated attacks
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16686317
There are so many phpbb scripts, how can I track which of these did the damage?...


its not so easy to re install cause ther are 500 working domains/websites on the box now.

the other idea with the attributes sound nice....


0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16686573
I usually have a separate log for each subdomain...

look at the error log for each subdomain, looking for the string "/tmp". there you will got which one was.
but you need to have them updated.

In order to have all domains apart one from the other, I use a pretty good project called "linux vserver" which makes virtual servers for each client. if one client get compromised, it will not disturb any other client.

it takes more cpu, though, but the extra security pays in this situations... maybe you have a rootkit already!

regards
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16688531
you mean vhosts not subdomains right?

All the vhosts have their own log files and thats a fine idea to look at them..
but I cant imagine how many hours that will take hehe...
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16692835
that's the downside =)
more security = more effort.
if you want to reduce effort, then you need to invest more money =/
0
 
LVL 3

Expert Comment

by:jglyons
ID: 16693233

mod_security should protect the server for you.

0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16697493
Redimido, i mean that if i run the serach the cpu usage goes high and that makes problems.
do u know by the way how can I run a script in low priority ?



jglyons: you checked that and we have planned to install it
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16699405
you can use "nice" to low the priority of all programs started inside the script. add at the beginning of each line something like
nice 10 program parameters etc etc etc

(nice can go from -20 to 20, being -20 something better than the kernel, so you can loss control of your linux box. if positive, your process will be "nice" to all other process having less priority)
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16723472
What do u mean  I can loose control ???? "nice" command is dangerous for the stability of the box ?
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16730730
every command issued by root is dangerous ;-)
nice -20 will almost suspend all other tasks (including ssh, login) and the system may become not responsive.
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16732450
oh now I got you!!!

about the commands that jglyons gave:
lsof -p 29148 to see what files are being used.
netstat -pan | grep 29148 will show you want port it's listening on.

I used  them today cause i saw a strange process and I found this :

[root@linux7 tmp]# /usr/sbin/lsof -p 20624

perl    20624 apache 2020u  IPv4   29026679                TCP  (myserverdomain):56354->81-208-62-219.ip.fastwebnet.it:ircd (SYN_SENT)

and with netstat -pan | grep 20624
tcp        0      1 (myserverip):56710         81.208.62.219:6667          SYN_SENT    20624/[htttpd32]

I've blocked 6667 & 56710 from the server and also the 81.208.62.219 ip ,but how do I kill the connection
that is allready made?
0
 
LVL 19

Expert Comment

by:Gabriel Orozco
ID: 16734434
I wonder if you have mod_proxy enabled. if you do, then change

proxyrequests on

to off, and restart apache.

also, it is convenient to check if your apache needs to be upgraded.

All this because yoy appear to be an open proxy =)
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16738537
Redimido, htttpd32 perl script seems to be not an Apache daemon.
Probably someone uploaded perl script that is an IRC bot.

MaRiOsGR, if you allow users to run Perl scripts and it you are a web hoster, it's better to close outgoing TCP connections for all apache users or to remove all socket related Perl libraries.
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16780507
Nopius thats a good idea, how do I control the sockets ?
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16781486
You are able to close all outgoing connections (except icmp,  dns, probably smtp) with a firewall rules (still allowing all incoming connections).
If you have some web crawlers or http proxies on that server, then you problem is impossible to resolve.

If you like to remove socket library completely from Perl, I don't know how to do it clean, but you may do:
find / \*perl5/\*Socket.so

then move that file to Socket.so.orig, nobody will be able to use sockets from Perl (but still able to do it from any other scripting language, so firewall rules are preferrable way).
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16783321
Lets say that i block all outgoing connectionsfor the websites,
doesnt that lock the ability of some one to download a file that one of the client's website offers?

the server is for web hosting with almost 600 vhosts so it s very important..
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16786359
> Lets say that i block all outgoing connectionsfor the websites,
> doesnt that lock the ability of some one to download a file that one of the client's website offers?
No.
File downloading goes over HTTP or FTP. In both cases client initiates connection, not a server. So the connection is incoming.
Even uploading is incoming. Normally server doesn't initiate any connection (except DNS requests).

> the server is for web hosting with almost 600 vhosts so it s very important..

Privacy and security is also importent. You may collect network traffic for a couple of days for 'SYN' packets. That's are session initiation TCP packets.

Command syntax is:
tcpdump -w /tmp/sessions.dmp  'tcp[13]==2'

The amount of collected data will be relatevely small. Having that dump file you may analize then only outgoing packets:
tcpdump -r /tmp/sessions.dmp src net x.x.x.x/24
where x.x.x.x is your local interface network/mask bits

 
0
 
LVL 2

Author Comment

by:MaRiOsGR
ID: 16806511
we installed mod_security as you all said, and indeed we havent found any scrpts written at /tmp/ or /var/tmp/ anymore, but now we have a different problem.

Apache crashed much ofter, and in the error logs every time we see this

[Thu Jun 01 14:44:37 2006] [error] Cannot configure connection "domainname_gr_Connection_ssl"
[Thu Jun 01 14:44:38 2006] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-conn
ectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
............
[Thu Jun 01 14:44:40 2006] [error] Cannot configure connection "otherdomainname_gr_Connection"
[Thu Jun 01 14:44:40 2006] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-conn
ectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)



..........
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Are you ready to place your question in front of subject-matter experts for more timely responses? With the release of Priority Question, Premium Members, Team Accounts and Qualified Experts can now identify the emergent level of their issue, signal…
Suggested Courses
Course of the Month15 days, 15 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question