?
Solved

Server with with 100% occupancy in Red Hat Linux

Posted on 2011-11-01
19
Medium Priority
?
626 Views
Last Modified: 2012-06-22
Dear Experts

I have an environment with Red Hat Linux 1.5 and JBoss Java and Oracle 10g. In this environment runs a java application.
After 45 or 48 working days, the server gets the two processors with 100% occupancy.
Any idea?

Note : the JBoss and Oracle is shutdown in this moment . The  top process is crond. ~90% process
0
Comment
Question by:alcionebernardi
  • 4
  • 4
  • 3
  • +5
19 Comments
 
LVL 78

Assisted Solution

by:slightwv (䄆 Netminder)
slightwv (䄆 Netminder) earned 333 total points
ID: 37062132
Sort the list by cpu usage.  We need to see what processes are using the CPU.
0
 
LVL 3

Assisted Solution

by:dbauermann
dbauermann earned 168 total points
ID: 37062139
What the content of crontab (crontab -l or cat /etc/crontab)?
0
 
LVL 68

Assisted Solution

by:woolmilkporc
woolmilkporc earned 168 total points
ID: 37062179
Which files does crond hold open? Check with

lsof -p PID

Replace PID with the process ID of crond.

What do you find in the log (/var/log/cron)?

Do you maybe have some huge file in /etc/cron.d which should not be there, e.g. a core dump?

wmp



0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 86

Assisted Solution

by:CEHJ
CEHJ earned 333 total points
ID: 37062183
Unfortunately that shot isn't very useful. We need to see a tree. You can toggle that with F5 but there's no way you're going to get a proper screenshot. I suggest you begin with the following command, then post the file 'ee.txt'
Make sure you do it in such a way as a monospaced font is used
pstree >ee.txt

Open in new window

0
 
LVL 86

Assisted Solution

by:CEHJ
CEHJ earned 333 total points
ID: 37062186
(Or you can simply attach the file to obviate font problems)
0
 

Author Comment

by:alcionebernardi
ID: 37062489
0
 
LVL 6

Expert Comment

by:JRoyse
ID: 37062561
restart crond?
0
 

Author Comment

by:alcionebernardi
ID: 37062566
no
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 37062601
Looks like things have changed - instead of a massive number of java processes (first screenshot), you now seem to have a massive number of perl-started port scanning processes ...
0
 
LVL 19

Expert Comment

by:jools
ID: 37066956
defo looks like there are a load of portscan processes running now.

Not something you would want of a server so you have to ask if you installed it or if someone compromised your system and is scanning your network.

0
 
LVL 19

Expert Comment

by:jools
ID: 37067027
Can you see if this file exists;
   /tmp/sess_0088025413980486928597bff29123

...and have a look whats in it;
   strings /tmp/sess_0088025413980486928597bff29123

... and post the output to the following;
   ps -ealf > ff.txt
0
 
LVL 5

Expert Comment

by:hossamshaaban
ID: 37068886
you should restart the cron service

# service crond restart
0
 
LVL 6

Assisted Solution

by:JRoyse
JRoyse earned 330 total points
ID: 37074200
You can also search through your web logs to see if there is anything malicious.  Check the web application for updates/patches.  Also you may limiting the webserver access to malicious ip addresses:
[server]$ cat .htaccess
# Block list example
order Allow,Deny
Deny from 217.23.
Deny from 194.44.
Deny from 24.185.
Deny from 186.
Deny from 78.89.
Deny from 85.25.95.
Allow from all

Open in new window

0
 

Author Comment

by:alcionebernardi
ID: 37101560
hi

I found a pearl script, which was some ips and port scanning.
Disable this script and everything went back to work again.
0
 
LVL 19

Expert Comment

by:jools
ID: 37103080
so has the problem now been fixed?

If you run atop you can monitor the processes and find out what is running to hog the system.

J
0
 
LVL 6

Assisted Solution

by:JRoyse
JRoyse earned 330 total points
ID: 37108157
You are seriously going to do an audit to prevent the malicious person from controlling the server again.  Most likely there are vulnerabilities in the web application that allow remote code to be allowed to execute on the server.  You may have a new user on your server see the below for examples:

find . -ctime -15 -print #created
find . -mtime -15 -print #modified

Open in new window


Will find created/modified files in 15 days.  This may help identify changed files on your server.
0
 

Author Comment

by:alcionebernardi
ID: 37108378
dear all
the problem occurred again
Now it seems that the processes are Oracle, any ideas?
oraprocess.png
0
 
LVL 78

Assisted Solution

by:slightwv (䄆 Netminder)
slightwv (䄆 Netminder) earned 333 total points
ID: 37108432
Those processes are database connections.  you need to get the Parent PID and see what app/process is connecting to the database.

0
 
LVL 19

Accepted Solution

by:
jools earned 168 total points
ID: 37108765
You need to be looking at the log files as well.

Did you put the port scanner on the system or was it someone else?
0

Featured Post

Upgrade your Question Security!

Add Premium security features to your question to ensure its privacy or anonymity. Learn more about your ability to control Question Security today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.
This video teaches viewers about errors in exception handling.
Suggested Courses
Course of the Month14 days, 16 hours left to enroll

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question