Server with with 100% occupancy in Red Hat Linux

Dear Experts

I have an environment with Red Hat Linux 1.5 and JBoss Java and Oracle 10g. In this environment runs a java application.
After 45 or 48 working days, the server gets the two processors with 100% occupancy.
Any idea?

Note : the JBoss and Oracle is shutdown in this moment . The  top process is crond. ~90% process
alcionebernardiAsked:
Who is Participating?
 
joolsCommented:
You need to be looking at the log files as well.

Did you put the port scanner on the system or was it someone else?
0
 
slightwv (䄆 Netminder) Commented:
Sort the list by cpu usage.  We need to see what processes are using the CPU.
0
 
dbauermannCommented:
What the content of crontab (crontab -l or cat /etc/crontab)?
0
Cloud Class® Course: CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

 
woolmilkporcCommented:
Which files does crond hold open? Check with

lsof -p PID

Replace PID with the process ID of crond.

What do you find in the log (/var/log/cron)?

Do you maybe have some huge file in /etc/cron.d which should not be there, e.g. a core dump?

wmp



0
 
CEHJCommented:
Unfortunately that shot isn't very useful. We need to see a tree. You can toggle that with F5 but there's no way you're going to get a proper screenshot. I suggest you begin with the following command, then post the file 'ee.txt'
Make sure you do it in such a way as a monospaced font is used
pstree >ee.txt

Open in new window

0
 
CEHJCommented:
(Or you can simply attach the file to obviate font problems)
0
 
alcionebernardiAuthor Commented:
0
 
JRoyseCommented:
restart crond?
0
 
alcionebernardiAuthor Commented:
no
0
 
CEHJCommented:
Looks like things have changed - instead of a massive number of java processes (first screenshot), you now seem to have a massive number of perl-started port scanning processes ...
0
 
joolsCommented:
defo looks like there are a load of portscan processes running now.

Not something you would want of a server so you have to ask if you installed it or if someone compromised your system and is scanning your network.

0
 
joolsCommented:
Can you see if this file exists;
   /tmp/sess_0088025413980486928597bff29123

...and have a look whats in it;
   strings /tmp/sess_0088025413980486928597bff29123

... and post the output to the following;
   ps -ealf > ff.txt
0
 
hossamshaabanCommented:
you should restart the cron service

# service crond restart
0
 
JRoyseCommented:
You can also search through your web logs to see if there is anything malicious.  Check the web application for updates/patches.  Also you may limiting the webserver access to malicious ip addresses:
[server]$ cat .htaccess
# Block list example
order Allow,Deny
Deny from 217.23.
Deny from 194.44.
Deny from 24.185.
Deny from 186.
Deny from 78.89.
Deny from 85.25.95.
Allow from all

Open in new window

0
 
alcionebernardiAuthor Commented:
hi

I found a pearl script, which was some ips and port scanning.
Disable this script and everything went back to work again.
0
 
joolsCommented:
so has the problem now been fixed?

If you run atop you can monitor the processes and find out what is running to hog the system.

J
0
 
JRoyseCommented:
You are seriously going to do an audit to prevent the malicious person from controlling the server again.  Most likely there are vulnerabilities in the web application that allow remote code to be allowed to execute on the server.  You may have a new user on your server see the below for examples:

find . -ctime -15 -print #created
find . -mtime -15 -print #modified

Open in new window


Will find created/modified files in 15 days.  This may help identify changed files on your server.
0
 
alcionebernardiAuthor Commented:
dear all
the problem occurred again
Now it seems that the processes are Oracle, any ideas?
oraprocess.png
0
 
slightwv (䄆 Netminder) Commented:
Those processes are database connections.  you need to get the Parent PID and see what app/process is connecting to the database.

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.