Server with with 100% occupancy in Red Hat Linux

Dear Experts

I have an environment with Red Hat Linux 1.5 and JBoss Java and Oracle 10g. In this environment runs a java application.
After 45 or 48 working days, the server gets the two processors with 100% occupancy.
Any idea?

Note : the JBoss and Oracle is shutdown in this moment . The  top process is crond. ~90% process
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

slightwv (䄆 Netminder) Commented:
Sort the list by cpu usage.  We need to see what processes are using the CPU.
What the content of crontab (crontab -l or cat /etc/crontab)?
Which files does crond hold open? Check with

lsof -p PID

Replace PID with the process ID of crond.

What do you find in the log (/var/log/cron)?

Do you maybe have some huge file in /etc/cron.d which should not be there, e.g. a core dump?


Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Unfortunately that shot isn't very useful. We need to see a tree. You can toggle that with F5 but there's no way you're going to get a proper screenshot. I suggest you begin with the following command, then post the file 'ee.txt'
Make sure you do it in such a way as a monospaced font is used
pstree >ee.txt

Open in new window

(Or you can simply attach the file to obviate font problems)
alcionebernardiAuthor Commented:
restart crond?
alcionebernardiAuthor Commented:
Looks like things have changed - instead of a massive number of java processes (first screenshot), you now seem to have a massive number of perl-started port scanning processes ...
joolsSenior Systems AdministratorCommented:
defo looks like there are a load of portscan processes running now.

Not something you would want of a server so you have to ask if you installed it or if someone compromised your system and is scanning your network.

joolsSenior Systems AdministratorCommented:
Can you see if this file exists;

...and have a look whats in it;
   strings /tmp/sess_0088025413980486928597bff29123

... and post the output to the following;
   ps -ealf > ff.txt
you should restart the cron service

# service crond restart
You can also search through your web logs to see if there is anything malicious.  Check the web application for updates/patches.  Also you may limiting the webserver access to malicious ip addresses:
[server]$ cat .htaccess
# Block list example
order Allow,Deny
Deny from 217.23.
Deny from 194.44.
Deny from 24.185.
Deny from 186.
Deny from 78.89.
Deny from 85.25.95.
Allow from all

Open in new window

alcionebernardiAuthor Commented:

I found a pearl script, which was some ips and port scanning.
Disable this script and everything went back to work again.
joolsSenior Systems AdministratorCommented:
so has the problem now been fixed?

If you run atop you can monitor the processes and find out what is running to hog the system.

You are seriously going to do an audit to prevent the malicious person from controlling the server again.  Most likely there are vulnerabilities in the web application that allow remote code to be allowed to execute on the server.  You may have a new user on your server see the below for examples:

find . -ctime -15 -print #created
find . -mtime -15 -print #modified

Open in new window

Will find created/modified files in 15 days.  This may help identify changed files on your server.
alcionebernardiAuthor Commented:
dear all
the problem occurred again
Now it seems that the processes are Oracle, any ideas?
slightwv (䄆 Netminder) Commented:
Those processes are database connections.  you need to get the Parent PID and see what app/process is connecting to the database.

joolsSenior Systems AdministratorCommented:
You need to be looking at the log files as well.

Did you put the port scanner on the system or was it someone else?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Oracle Database

From novice to tech pro — start learning today.