High CPU usage / for apache

I run top -c  and I see this :

 PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
29148 apache    25   0  1124 1084   640 R    86,6  0,1 308:04   0 /var/tmp/httpd

the cpu usage is very high..and it seems strange to me of what the httpd file is doing inside the /var/tmp/ folder..
could this be any  kind of hacking ?




LVL 2
MaRiOsGRAsked:
Who is Participating?
 
jglyonsCommented:

That is certainly a hack.
You need to look at

lsof -p 29148 to see what files are being used.

netstat -pan | grep 29148 will show you want port it's listening on.

Using a rootkit hunter is a good start but you'll almost certainly find that you've got an IRC bot running.

Chances are you've got a forum or CMS package on the server thats out of date and has vulnerabilities within it.

Something like mod_security is a good package to add on to try and prevent basic hacks in future.
0
 
pablouruguayCommented:
yep.
the httpd  was in /usr/sbin/httpd

when the httpd or another service execute inside tmp i really think is a hacking work.

check in /var/log/message to see any extrange and execute the command last to see the last loggings but if you dont have a soft to detect intrutions like tripwire or another is really difficult to see what haoppend in your system after a intrution.

i recommend format and install again. an update SO with a firewall
0
 
NopiusCommented:
Probably you are hacked.
You may see what is your fake httpd doing:
strace -p 29148
But probably it's some kind of computation (due to high CPU usage) and you may see very little or now system calls.
0
Cloud Class® Course: CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

 
Gabriel OrozcoSolution ArchitectCommented:
download and run rootkit hunter:
http://www.rootkit.nl/

it will let you know if some known rootkit is already in place.

and consider reinstall of the full box and install a current version of your distribution.

0
 
pablouruguayCommented:
and the split??? all points was give to jglyons.. i don't understand
0
 
Gabriel OrozcoSolution ArchitectCommented:
I understand mariosgr accepted his answer at the end

it would have been good to read a final comment from him, though
0
 
pablouruguayCommented:
ohhhhh i understand now... thanks gabriel for clarification........
0
 
Gabriel OrozcoSolution ArchitectCommented:
anytime
0
 
MaRiOsGRAuthor Commented:
The httpd script u see running from /var/tmp/ is not the apache.
that bot or hacker uploads through some hole , scripts with false names like "named" or "httpd" so I wont see them at first sight.

The strange thinng is how is he or able to run these scripts...
i thinkg that the main source of evil are some phpbb forums that the server has.

Also that "hacker" keeps uploading psyBNC scripts for spamming ofcourse, so I'm blocking through th firewall all the outgoung traffic to undernet etc.

The only problem is that the red hat version we use doesnt have the "lsof" command but im working on it.

p.s.

The manage to update and run scripts on these folders:

/tmp/
/var/tmp/
/dev/shm/
all files are made by user "apache"
0
 
Gabriel OrozcoSolution ArchitectCommented:
MariosGR: Yes, in fact, phpbb has had a lot of vulnerabilities.

just upgrade and you should get rid of these new scripts

but once hacked, you cannot trust your system, so... reinstall and upgrade *retty* often.

you can repartition your system and mount /tmp and /var/tmp on a different partition, with atributes that disallow execution. this way you can get rid of most of automated attacks
0
 
MaRiOsGRAuthor Commented:
There are so many phpbb scripts, how can I track which of these did the damage?...


its not so easy to re install cause ther are 500 working domains/websites on the box now.

the other idea with the attributes sound nice....


0
 
Gabriel OrozcoSolution ArchitectCommented:
I usually have a separate log for each subdomain...

look at the error log for each subdomain, looking for the string "/tmp". there you will got which one was.
but you need to have them updated.

In order to have all domains apart one from the other, I use a pretty good project called "linux vserver" which makes virtual servers for each client. if one client get compromised, it will not disturb any other client.

it takes more cpu, though, but the extra security pays in this situations... maybe you have a rootkit already!

regards
0
 
MaRiOsGRAuthor Commented:
you mean vhosts not subdomains right?

All the vhosts have their own log files and thats a fine idea to look at them..
but I cant imagine how many hours that will take hehe...
0
 
Gabriel OrozcoSolution ArchitectCommented:
that's the downside =)
more security = more effort.
if you want to reduce effort, then you need to invest more money =/
0
 
jglyonsCommented:

mod_security should protect the server for you.

0
 
MaRiOsGRAuthor Commented:
Redimido, i mean that if i run the serach the cpu usage goes high and that makes problems.
do u know by the way how can I run a script in low priority ?



jglyons: you checked that and we have planned to install it
0
 
Gabriel OrozcoSolution ArchitectCommented:
you can use "nice" to low the priority of all programs started inside the script. add at the beginning of each line something like
nice 10 program parameters etc etc etc

(nice can go from -20 to 20, being -20 something better than the kernel, so you can loss control of your linux box. if positive, your process will be "nice" to all other process having less priority)
0
 
MaRiOsGRAuthor Commented:
What do u mean  I can loose control ???? "nice" command is dangerous for the stability of the box ?
0
 
NopiusCommented:
every command issued by root is dangerous ;-)
nice -20 will almost suspend all other tasks (including ssh, login) and the system may become not responsive.
0
 
MaRiOsGRAuthor Commented:
oh now I got you!!!

about the commands that jglyons gave:
lsof -p 29148 to see what files are being used.
netstat -pan | grep 29148 will show you want port it's listening on.

I used  them today cause i saw a strange process and I found this :

[root@linux7 tmp]# /usr/sbin/lsof -p 20624

perl    20624 apache 2020u  IPv4   29026679                TCP  (myserverdomain):56354->81-208-62-219.ip.fastwebnet.it:ircd (SYN_SENT)

and with netstat -pan | grep 20624
tcp        0      1 (myserverip):56710         81.208.62.219:6667          SYN_SENT    20624/[htttpd32]

I've blocked 6667 & 56710 from the server and also the 81.208.62.219 ip ,but how do I kill the connection
that is allready made?
0
 
Gabriel OrozcoSolution ArchitectCommented:
I wonder if you have mod_proxy enabled. if you do, then change

proxyrequests on

to off, and restart apache.

also, it is convenient to check if your apache needs to be upgraded.

All this because yoy appear to be an open proxy =)
0
 
NopiusCommented:
Redimido, htttpd32 perl script seems to be not an Apache daemon.
Probably someone uploaded perl script that is an IRC bot.

MaRiOsGR, if you allow users to run Perl scripts and it you are a web hoster, it's better to close outgoing TCP connections for all apache users or to remove all socket related Perl libraries.
0
 
MaRiOsGRAuthor Commented:
Nopius thats a good idea, how do I control the sockets ?
0
 
NopiusCommented:
You are able to close all outgoing connections (except icmp,  dns, probably smtp) with a firewall rules (still allowing all incoming connections).
If you have some web crawlers or http proxies on that server, then you problem is impossible to resolve.

If you like to remove socket library completely from Perl, I don't know how to do it clean, but you may do:
find / \*perl5/\*Socket.so

then move that file to Socket.so.orig, nobody will be able to use sockets from Perl (but still able to do it from any other scripting language, so firewall rules are preferrable way).
0
 
MaRiOsGRAuthor Commented:
Lets say that i block all outgoing connectionsfor the websites,
doesnt that lock the ability of some one to download a file that one of the client's website offers?

the server is for web hosting with almost 600 vhosts so it s very important..
0
 
NopiusCommented:
> Lets say that i block all outgoing connectionsfor the websites,
> doesnt that lock the ability of some one to download a file that one of the client's website offers?
No.
File downloading goes over HTTP or FTP. In both cases client initiates connection, not a server. So the connection is incoming.
Even uploading is incoming. Normally server doesn't initiate any connection (except DNS requests).

> the server is for web hosting with almost 600 vhosts so it s very important..

Privacy and security is also importent. You may collect network traffic for a couple of days for 'SYN' packets. That's are session initiation TCP packets.

Command syntax is:
tcpdump -w /tmp/sessions.dmp  'tcp[13]==2'

The amount of collected data will be relatevely small. Having that dump file you may analize then only outgoing packets:
tcpdump -r /tmp/sessions.dmp src net x.x.x.x/24
where x.x.x.x is your local interface network/mask bits

 
0
 
MaRiOsGRAuthor Commented:
we installed mod_security as you all said, and indeed we havent found any scrpts written at /tmp/ or /var/tmp/ anymore, but now we have a different problem.

Apache crashed much ofter, and in the error logs every time we see this

[Thu Jun 01 14:44:37 2006] [error] Cannot configure connection "domainname_gr_Connection_ssl"
[Thu Jun 01 14:44:38 2006] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-conn
ectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
............
[Thu Jun 01 14:44:40 2006] [error] Cannot configure connection "otherdomainname_gr_Connection"
[Thu Jun 01 14:44:40 2006] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-conn
ectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)



..........
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.