Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win


How can I diagnose my log file to work out why my server load was 9.6?

Posted on 2011-02-20
Medium Priority
Last Modified: 2012-05-11
My Apache web server died so I ran the following command

killall -9 sys-snap.sh

And then restarted httpd. My log file is here:


I can't make much sense of it I still don't understand why the server died
Question by:wordswithfriends
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 16

Expert Comment

by:Joseph Gan
ID: 34939453
I've noticed the mysql daemon was running over 876 hours, is this normal for your server?

mysql    22229  3.0  2.3 164164 18496 ?        Sl   Jan31 876:49  \_ /usr/libexec/mysqld

Author Comment

ID: 34939490
It doesn't sound normal although I only have a very passing knowledge of how the daemon is supposed to work.  Let's say the user opens a web page that runs a SELECT query that takes a few seconds.  In this case would the server open and quickly close a MySQL demon?  If this is true, then 876 hours would be very unusual

On the other hand if there is a single mysql that sits in the background and processes all queries then maybe it's possible.  Running

ps -aux | grep mysql

I get

[root@wor system-snapshot]# ps -aux | grep mysql
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root     22151  0.0  0.1  11932  1416 ?        S    Jan31   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql    22229  3.1  2.3 164164 18560 ?        Sl   Jan31 922:49 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock

It looks like it is still going.  What is normal behavior?

Expert Comment

ID: 34939497
Usually it's normal to have the MySQL daemon running for that long (or even much more) if you have a running DB on the server.
Apparently there's nothing wrong in the log you posted. You had free memory, the sum of %CPU in all the processes is below 10-15% cpu power. What is missing is disk space info, so you might want to check /tmp or /var free space.
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  


Author Comment

ID: 34939523
heaps of disk space

 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs              30G  1.8G   29G   6% /
LVL 80

Expert Comment

ID: 34940412
How long has the system been up? (uptime)

There are not enough hours from Jan 31 to today to account for the 960hours reflected.  
Look at the /var/log/httpd/error_log to see if there is a clue to why it crashed. Check if you have a core dump in the web directory
find /path/to/web -name 'core'

If you have, you could analyze the core dump in an attempt to determine the cause of apach's crash.

You could setup a cron to collect data iostat/vmstat/memstat/etc. this way should apache crash again, you will have a trend of data leading up to it.
Depending on what you have i.e. php, etc., you may not have allocated enough resources.

The apache log should have a clue as to why it crashed.

Author Comment

ID: 34940436
 23:07:55 up 21 days,  5:21,  2 users,  load average: 0.82, 0.35, 0.23

SIGTERM was me restarting Apache.  Doesn't seem like anything interesting before then.

[Sun Feb 20 06:06:46 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:07:55 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:09:04 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:10:14 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:11:41 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:12:50 2011] [client] File does not exist: /var/www/html/whm-server-status
[Sun Feb 20 06:21:55 2011] [notice] caught SIGTERM, shutting down

find /var/www -name 'core' doesn't return any results.

Could you give more details about the cron job?
LVL 80

Expert Comment

ID: 34942795
You can setup a cron job for a script that runs vmstat 5 5, iostat 5 5
top -n 1

If you have another linux/unix box on which you can setup cacti or set it up on this one and by enabling snmp on the web/mysql/mail/courier,etc. system you can have cacti collect the data as well which will then be represented in graphical term CPU, memory, HD, and there are application templates for apache.
Should it get stuck in the same way, use strace -f -p <pid_of_apache_parent> to see what it is doing.
You may have allocated too few children or your system was experiencing a DoS attack.
Check the access_log to see the number of queries it was getting per second i.e. was there a spike in the number of requests per second it was recording in the log.
The time stamp is in unix time format (epoch number of elapsed seconds since 1/1/1970)

another option is to tabulate how many requests were being seen from the same source.

Author Comment

ID: 35073386
This was too complicated for me to follow
LVL 80

Expert Comment

ID: 35079107
To determine the underlying cause, you need to collect data such that when the issue reoccurs, you can look at the collected data to see whether there is something there that can explain the situation.

Author Comment

ID: 35088989
Understood.  But the following instructions

You can setup a cron job for a script that runs vmstat 5 5, iostat 5 5

Whilst may becorrect is not particularly easy to follow
LVL 80

Accepted Solution

arnold earned 1000 total points
ID: 35108970
Do you know how to setup a cron job?

Create a script: datacollection.sh
Add it into the cron job
*/5 * * * * /path/to/script/datacollection.sh iostat
*/5 * * * * /path/to/script/datacollection.sh vmstat

The script will add entries into the iostat.txt or vmstat.txt with the start date/time and end  date/time
Similarly you could add additional parameters into the script for collection purposes i.e. top -n 1 to get the currently top active process list. ps -ef to get a complete list of all processes on the system. etc.

if  [ ! -z "$1"  ] ; then 
    if [  "$1" = 'vmstat' -o "$1" = 'iostat'  ]  ; then
(echo start `date +"%Y%m%e%H%M%S"`;
$1 5 5; 
echo end `date +"%Y%m%e%H%M%S"`) >> /some/location/$1.txt
echo "Usage : $0 (vmstat|iostat)"
exit 1;

Open in new window


Featured Post

Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month11 days, 21 hours left to enroll

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question