Server load high since re-boot - mysql, apache troubleshoot

fulcan
fulcan used Ask the Experts™
on
Hi All,

My VPS has been running fine for some time but for some reason late last week was having trouble loading pages.  After restarting mysql, apache etc, it would still not improve.  We therefore rebooted the server.

However, now we have the problem that we are getting really high server loads which we did not have before.

We are looking for for some help to troubleshoot the issue.

The hardware is a VPS and here are the hardware specs.
Host Server CPU        Poweredge® 2950 III
Dual Xeon® Quad Core
Guaranteed RAM 2048 MB
Burstable RAM 4096 MB
Memory Management       SLM

Here are the ps aux results when the server load was at 32!

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10348   744 ?        Ss   Jan27   0:02 init [3]
nobody    1724  2.0  1.0 158392 42740 ?        S    11:05   0:05 /usr/local/apache/bin/httpd -k start -DSSL
nobody    1736  0.7  0.8 151264 36704 ?        S    11:05   0:02 /usr/local/apache/bin/httpd -k start -DSSL
nobody    1737  1.1  0.8 151540 36944 ?        S    11:05   0:03 /usr/local/apache/bin/httpd -k start -DSSL
nobody    1739  1.2  0.7 146608 32004 ?        S    11:05   0:03 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3515  1.4  0.8 148436 33856 ?        S    11:06   0:03 /usr/local/apache/bin/httpd -k start -DSSL
retailbl  3629  0.0  0.0  15508  1044 ?        S    Jan28   0:06 /usr/lib/courier-imap/bin/imapd /home/retailbl/mail/mydomain.com/sales
root      3689  0.0  0.1  29584  4500 ?        S    Jan29   0:11 tailwatchd
root      3787  0.0  0.0  48064  1616 ?        S    11:07   0:00 crond
root      3790  0.0  0.1  60352  8320 ?        Rs   11:07   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/status.php
nobody    3828  0.6  0.7 145024 30400 ?        S    11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3829  2.0  0.8 152804 37692 ?        S    11:07   0:03 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3830  2.4  1.0 157356 42132 ?        R    11:07   0:03 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3843  1.4  0.7 146568 31840 ?        S    11:07   0:02 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3857  1.3  0.9 154528 38740 ?        S    11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
root      3864  0.1  6.7 388548 284160 ?       Ssl  Jan29   2:55 /usr/sbin/clamd
mailnull  3874  0.0  0.0  64264  3312 ?        Ss   Jan29   0:01 /usr/sbin/exim -bd -q60m
nobody    3891  1.2  0.9 157376 41360 ?        R    11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3893  1.5  0.9 159180 41616 ?        S    11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    3894  1.0  0.9 154492 38720 ?        S    11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
root      3940  0.0  0.1  43012  8012 ?        S    11:07   0:00 tailwatchd
retailbl  3957  0.0  0.0  13416   784 ?        S    11:08   0:00 /usr/lib/courier-imap/bin/imapd /home/retailbl/mail/mydomain.com/sean
nobody    3965  0.6  0.8 149412 34808 ?        S    11:08   0:00 /usr/local/apache/bin/httpd -k start -DSSL
root      3977  0.0  0.0  48064  1616 ?        S    11:08   0:00 crond
root      3979  0.0  0.0  48064  1616 ?        S    11:08   0:00 crond
root      4023  0.0  0.2  60636  8844 ?        Ss   11:08   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/spam_cronjob.p
root      4024  0.0  0.1  59252  7344 ?        Ss   11:08   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/status.php
nobody    4032  2.5  1.0 158424 42028 ?        S    11:08   0:02 /usr/local/apache/bin/httpd -k start -DSSL
nobody    4034  2.6  0.8 150804 35960 ?        S    11:08   0:02 /usr/local/apache/bin/httpd -k start -DSSL
nobody    4035  0.3  0.7 147968 31940 ?        R    11:08   0:00 /usr/local/apache/bin/httpd -k start -DSSL
root      4041  0.0  0.3 125620 13452 ?        S    09:57   0:00 /usr/local/apache/bin/httpd -k start -DSSL
retailbl  4068  0.0  0.0  15508   988 ?        S    11:08   0:00 /usr/lib/courier-imap/bin/imapd /home/retailbl/mail/mydomain.com/sean
nobody    4069  2.7  0.9 157880 41772 ?        R    11:08   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    4070  1.8  0.7 145020 30412 ?        S    11:08   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    4071  2.7  0.9 157104 41284 ?        S    11:08   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody    4072  0.0  0.3 134476 16556 ?        S    11:08   0:00 /usr/local/apache/bin/httpd -k start -DSSL
root      5146  0.0  0.0  48064  1616 ?        S    11:09   0:00 crond
root      5187  0.0  0.0  48064  1616 ?        S    11:09   0:00 crond
root      5232  0.0  0.0      0     0 ?        Z    11:09   0:00 [sh] <defunct>
root      5263  0.0  0.0  10724   984 pts/0    R+   11:09   0:00 ps aux
root      5264  0.6  0.1  57036  4472 ?        Rs   11:09   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/update_email.p
root      5265  2.5  0.0  28464  3732 ?        R    11:09   0:00 /usr/bin/perl /scripts/restartsrv_mysql --check
root      5270  0.0  0.0   8668  1060 ?        S    11:09   0:00 sh -c ps aux -w -w | grep "update_email.php" | grep -v "sh -c" | grep -vc "g
root      5271  0.0  0.0   8668  1052 ?        R    11:09   0:00 sh -c /sbin/ifconfig -a | grep "inet "  | awk '{print $2}'
root      5272  0.0  0.0   8668   484 ?        R    11:09   0:00 sh -c /sbin/ifconfig -a | grep "inet "  | awk '{print $2}'
root      5274  0.0  0.0   6020   592 ?        S    11:09   0:00 grep inet
root      5275  0.0  0.0   8668   152 ?        R    11:09   0:00 sh -c /sbin/ifconfig -a | grep "inet "  | awk '{print $2}'
root      5278  3.0  0.1  59252  7304 ?        Ss   11:09   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/status.php
root      5279  0.0  0.0  59252  4132 ?        R    11:09   0:00 /usr/local/cpanel/3rdparty/bin/php-cgi /usr/local/assp/deluxe/status.php
root      5281  0.0  0.0   8668   488 ?        R    11:09   0:00 sh -c ps aux -w -w | grep "update_email.php" | grep -v "sh -c" | grep -vc "g
root      5282  0.0  0.0   6020   596 ?        S    11:09   0:00 grep -v sh -c
root      5283  0.0  0.0   8668   488 ?        R    11:09   0:00 sh -c ps aux -w -w | grep "update_email.php" | grep -v "sh -c" | grep -vc "g
root      5519  0.0  0.0  12604   676 ?        S<s  Jan27   0:00 /sbin/udevd -d
root      5950  0.0  0.0   3788   444 ?        S    Jan27   0:00 /usr/sbin/courierlogger -pid=/var/spool/authdaemon/pid -facility=mail -start
root      5951  0.0  0.0  14384   700 ?        S    Jan27   0:00 /usr/libexec/courier-authlib/authdaemond
root      5958  0.0  0.0 155896  1420 ?        Sl   Jan27   0:07 rsyslogd -m 0
root      5959  0.0  0.0  14384   404 ?        S    Jan27   0:00 /usr/libexec/courier-authlib/authdaemond
root      5960  0.0  0.0  14384   408 ?        S    Jan27   0:00 /usr/libexec/courier-authlib/authdaemond
root      5964  0.0  0.0   3804   420 ?        Ss   Jan27   0:00 rklogd -x
root      5975  0.0  0.0  62616  1208 ?        Ss   Jan27   0:02 /usr/sbin/sshd
root      5987  0.0  0.0  21644   932 ?        Ss   Jan27   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root      6050  0.0  0.0   3788   444 ?        S    Jan27   0:02 /usr/sbin/courierlogger -pid=/var/run/imapd.pid -start -name=imapd /usr/lib/
root      6051  0.0  0.0   7992   604 ?        S    Jan27   0:03 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=50 -maxperip=
root      6057  0.0  0.0   3788   348 ?        S    Jan27   0:00 /usr/sbin/courierlogger -pid=/var/run/imapd-ssl.pid -start -name=imapd-ssl /
root      6058  0.0  0.0   7992   588 ?        S    Jan27   0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=50 -maxperip=
root      6063  0.0  0.0   3788   440 ?        S    Jan27   0:00 /usr/sbin/courierlogger -pid=/var/run/pop3d.pid -start -name=pop3d /usr/lib/
root      6064  0.0  0.0   7992   608 ?        S    Jan27   0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=50 -maxperip=
root      6069  0.0  0.0   3788   344 ?        S    Jan27   0:00 /usr/sbin/courierlogger -pid=/var/run/pop3d-ssl.pid -start -name=pop3d-ssl /
root      6070  0.0  0.0   7992   588 ?        S    Jan27   0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=50 -maxperip=
root      7216  0.0  0.0  39328  1664 ?        Ss   Jan27   0:00 pure-ftpd (SERVER)
root      7218  0.0  0.0  37008  1592 ?        S    Jan27   0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
root      7231  0.0  0.0  20880  1216 ?        Ss   Jan27   0:13 crond
root      7445  0.0  0.0  46736   820 ?        Ss   Jan27   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root      7446  0.0  0.0  46736   552 ?        S    Jan27   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root      7657  0.0  0.0   3812   440 ?        Ss   Jan27   0:00 /usr/sbin/portsentry -tcp
root      8055  0.0  0.0  28004  2972 ?        S    Jan29   0:00 cPhulkd - processor
root      8061  0.0  0.3  91816 12984 ?        S    Jan29   0:00 cpdavd - accepting connections on 2077 and 2078
root      8064  0.0  0.1  32208  5240 ?        S    Jan29   0:06 queueprocd - wait to process a task
root      8071  0.0  0.0  23464  2112 ?        SN   Jan29   0:00 cpanellogd - sleeping for logs
root      8077  0.0  0.1  29876  4240 ?        S    Jan27   0:07 /etc/authlib/authProg
root      8080  0.0  1.2 129244 53412 ?        S    Jan27   3:22 /usr/bin/perl /usr/local/assp/assp.pl /usr/local/assp
nobody    9390  1.9  1.0 161268 45172 ?        S    10:38   0:37 /usr/local/apache/bin/httpd -k start -DSSL
nobody    9393  1.9  1.1 162768 47064 ?        R    10:38   0:35 /usr/local/apache/bin/httpd -k start -DSSL
nobody    9401  1.5  1.0 160700 44040 ?        S    10:38   0:29 /usr/local/apache/bin/httpd -k start -DSSL
root      9911  0.0  0.1  38776  6592 ?        Ss   Jan28   0:06 /usr/sbin/munin-node
root     13415  0.0  0.3  86608 12964 ?        S    Jan29   0:09 cpsrvd - waiting for connections
root     13451  0.0  0.1  29876  4240 ?        S    Jan27   0:06 /etc/authlib/authProg
retailbl 18191  0.0  0.0  13044  1344 ?        S    Jan27   0:05 /usr/libexec/gam_server
retailbl 18212  0.0  0.0  15880  1420 ?        S    Jan27   0:54 /usr/lib/courier-imap/bin/imapd /home/retailbl/mail/mydomain.com/support
nobody   19622  2.2  0.9 154340 40120 ?        S    10:48   0:29 /usr/local/apache/bin/httpd -k start -DSSL
nobody   19642  1.4  1.0 158180 44124 ?        S    10:48   0:18 /usr/local/apache/bin/httpd -k start -DSSL
root     20054  0.0  0.0  10764  1272 ?        S    Jan29   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/my
mysql    20087 15.7 10.2 589160 430212 ?       Sl   Jan29 346:35 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-fil
root     20398  0.0  0.5 134476 21592 ?        Ss   Jan29   0:14 /usr/local/apache/bin/httpd -k start -DSSL
root     24167  0.0  0.0  92176  3392 ?        Ss   10:19   0:00 sshd: root@pts/0
named    24344  0.0  0.1 251592  8204 ?        Ssl  Jan27   0:39 /usr/sbin/named -u named
retailbl 24396  0.0  0.0  16184  1712 ?        S    03:21   0:08 /usr/lib/courier-imap/bin/imapd /home/retailbl/mail/mydomain.com/sean
root     25886  0.0  0.0  12164  1740 pts/0    Ss   10:20   0:00 -bash
nobody   26061  1.9  1.1 162332 47620 ?        R    10:56   0:15 /usr/local/apache/bin/httpd -k start -DSSL
nobody   26065  2.0  1.0 158932 43308 ?        S    10:56   0:16 /usr/local/apache/bin/httpd -k start -DSSL
nobody   26067  1.6  1.0 158392 43964 ?        S    10:56   0:13 /usr/local/apache/bin/httpd -k start -DSSL
nobody   27772  1.8  0.9 155596 39432 ?        R    10:22   0:51 /usr/local/apache/bin/httpd -k start -DSSL
nobody   32070  1.4  1.0 157112 42504 ?        S    11:02   0:06 /usr/local/apache/bin/httpd -k start -DSSL

Also here are the results from mysqltuner for reference.

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.87-community-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated -InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 2G (Tables: 778)
[!!] Total fragmented tables: 36

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 12h 58m 31s (19M q [146.729 qps], 86K conn, TX: 12B, RX: 3B)
[--] Reads / Writes: 79% / 21%
[--] Total buffers: 378.0M global + 7.2M per thread (500 max threads)
[!!] Maximum possible memory usage: 3.9G (97% of installed RAM)
[OK] Slow queries: 0% (3K/19M)
[OK] Highest usage of available connections: 14% (73/500)
[OK] Key buffer size / total MyISAM indexes: 256.0M/1.3G
[OK] Key buffer hit rate: 100.0% (3B cached / 1M reads)
[OK] Query cache efficiency: 88.1% (15M cached / 18M selects)
[!!] Query cache prunes per day: 987643
[OK] Sorts requiring temporary tables: 0% (136 temp sorts / 891K sorts)
[!!] Joins performed without indexes: 59463
[OK] Temporary tables created on disk: 15% (278K on disk / 1M total)
[OK] Thread cache hit rate: 99% (73 created / 86K connections)
[OK] Table cache hit rate: 35% (1K open / 3K opened)
[OK] Open file limit used: 68% (1K/2K)
[OK] Table locks acquired immediately: 99% (3M immediate / 3M locks)

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Reduce your overall MySQL memory footprint for system stability
    Adjust your join queries to always utilize indexes
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (> 96M)
    join_buffer_size (> 1.0M, or always use indexes with joins)


I have optimized the tables and that did not improve performance.  Query cache was previously set at 64MB and was increased to 96MB.  I don't think it would be wise to increase this further without more ram - which I can get on this server.

I am aware of some poorly written select statements by more shopping cart developer and I am getting this reviewed, but these have not changed from before the reboot to now.

If anyone could provide some next steps for troubleshooting that would be great.

Thanks
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Artysystem administrator
Top Expert 2007

Commented:
> [!!] Maximum possible memory usage: 3.9G (97% of installed RAM)

 When server goes to swap you have very high load averate.
This seems to be  major problem. You need more memory. And consider moving to dedicated host, not VPS.
You are out of resources.
Top Expert 2009
Commented:
As your VPS config only guarantees 2GB ram you should configure your mysql/apache with that limit in mind and not 4GB, as the other 2GB might not be available to you when you need it and therefore have to swap excessively. Reduce the amount of connections or the buffers per connection. With 500 max connections and the current buffer settings you require 3.6GB/ram which you don't necessarily have.

Depending on your disk configuration, you might also have a network (if the mysql data directory is on a NAS) or disk issue (read/write errors) which causes this load. Check the output of dmesg and /var/log/warn for disk errors just to make sure you don't have hardware issues. This is unlikely but does not hurt to check out.

Author

Commented:
Thanks guys.  I have reduced the number of connections from 500 to 400.  In terms of my buffers per connection these have already been set based on 2MB of ram, as has my query cache (I took this back down to 64mb despite the mysqltuner suggestion to continuously increase it).  As outlined I have made the reduction in the number of connections but this has not provided me with any improvement.  The settings I have are also those that I had prior to the restart of the server so I don't think the issue is hardware issue, I believe it has to be a software issue because something has changed to take my load from an average of 0.5 to closer to 2 now and spiking at 40 or higher.

Further even with the spike my RAM being used is only sitting around the 1GB range so I still have spare memory.

Since my initial post we have run mysql slow query for two days and we are looking at that data and contacting the relevant vendors to discuss  their select statement etc.  However, as with above, these were in place previously so while we are checking I still suspect that the issue is a software issue and possibly not just mysql.
 
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Author

Commented:
Some further info.  We have ran the following top -b -i -n 20 >> ./top_procs in order to try and see what might be causing the load.  Here is the download.

top - 14:07:36 up 4 days, 15:51,  1 user,  load average: 15.44, 9.15, 7.79
Tasks:  97 total,  22 running,  73 sleeping,   0 stopped,   2 zombie
Cpu(s): 87.9%us, 12.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4194304k total,  1130052k used,  3064252k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7280 nobody    16   0  162m  48m 4624 R  7.8  1.2   0:02.22 httpd
 7298 nobody    16   0  161m  48m 5496 R  5.2  1.2   0:04.25 httpd
 5504 nobody    16   0  161m  49m 5248 R  4.2  1.2   0:12.64 httpd
 7283 nobody    15   0  157m  45m 4660 R  4.2  1.1   0:02.29 httpd
 9810 mysql     15   0  507m 172m 4296 R  4.2  4.2  22:07.51 mysqld
 7292 nobody    15   0  154m  42m 4664 R  1.6  1.0   0:03.30 httpd
 5976 nobody    16   0  152m  40m 4700 R  0.3  1.0   0:05.27 httpd
 6119 nobody    15   0  160m  47m 5564 R  0.3  1.2   0:03.77 httpd
 6051 root      18   0  7992  604  508 R  0.0  0.0   0:04.28 couriertcpd
 7281 nobody    16   0  153m  42m 4676 R  0.0  1.0   0:02.88 httpd
 7295 nobody    16   0  166m  53m 4564 R  0.0  1.3   0:00.91 httpd
 7824 nobody    17   0  162m  49m 4636 R  0.0  1.2   0:03.24 httpd
 9418 root      18   0 48064 1620  900 R  0.0  0.0   0:00.00 crond
 9419 root      19  -5 62376  10m 3812 R  0.0  0.2   0:00.12 php-cgi
 9618 root      15   0 12736 1252  936 R  0.0  0.0   0:00.10 top
 9675 munin     30  10 92940  11m 4340 R  0.0  0.3   0:00.50 munin-graph
 9905 root      18   0     0    0    0 R  0.0  0.0   0:00.00 grep
 9909 retailbl  18   0 12136  744  600 R  0.0  0.0   0:00.00 imaplogin
 9911 retailbl  20   0 12136  744  600 R  0.0  0.0   0:00.00 imaplogin
 9913 root      18   0 10044  576  464 R  0.0  0.0   0:00.00 imaplogin
 9914 retailbl  18   0 12136  748  600 R  0.0  0.0   0:00.00 imaplogin
13415 root      18   0 86608  12m 1496 R  0.0  0.3   0:15.48 cpsrvd-ssl

and

top - 14:09:19 up 4 days, 15:53,  1 user,  load average: 28.32, 15.60, 10.26
Tasks:  94 total,  25 running,  68 sleeping,   0 stopped,   1 zombie
Cpu(s): 18.7%us,  2.8%sy,  0.3%ni, 76.3%id,  1.9%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4194304k total,  1264172k used,  2930132k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7282 nobody    16   0  161m  49m 5160 R 11.4  1.2   0:03.27 httpd
10027 nobody    16   0  164m  48m 4588 R  8.2  1.2   0:01.50 httpd
 7280 nobody    16   0  159m  47m 4632 R  4.9  1.2   0:03.36 httpd
 7283 nobody    16   0  164m  52m 4732 R  4.9  1.3   0:05.34 httpd
 9675 munin     32  10 92936  11m 4352 R  4.9  0.3   0:00.98 munin-graph
 5976 nobody    16   0  151m  39m 4732 R  1.6  1.0   0:05.84 httpd
 5504 nobody    16   0  161m  49m 5248 R  0.0  1.2   0:13.67 httpd
 5989 nobody    16   0  162m  49m 4656 R  0.0  1.2   0:06.29 httpd
 7231 root      18   0 20880 1216  592 R  0.0  0.0   0:17.42 crond
 7275 nobody    16   0  163m  50m 5156 R  0.0  1.2   0:05.70 httpd
 7281 nobody    16   0  152m  40m 4684 R  0.0  1.0   0:04.19 httpd
 7298 nobody    16   0  160m  47m 5496 R  0.0  1.2   0:05.84 httpd
 7797 nobody    16   0  155m  43m 4700 R  0.0  1.1   0:01.84 httpd
 7821 nobody    16   0  164m  49m 4684 R  0.0  1.2   0:05.31 httpd
10155 nobody    16   0  154m  42m 4460 R  0.0  1.0   0:00.44 httpd
10196 nobody    16   0  142m  29m 3932 R  0.0  0.7   0:00.12 httpd
10211 root      35  18 33412 4304 1204 R  0.0  0.1   0:00.03 cpanellogd
11281 root      25   0  8668  224   72 R  0.0  0.0   0:00.00 sh
11319 root      15   0 12604 1160  860 R  0.0  0.0   0:00.00 top
11327 root      18   0 20880  624    0 R  0.0  0.0   0:00.00 crond
11329 root      18   0 28776 4044 1388 R  0.0  0.1   0:00.03 restartsrv_name
11336 root      18   0 60096 4956  172 R  0.0  0.1   0:00.00 php-cgi
11337 root      18   0 60372 5184  172 R  0.0  0.1   0:00.00 php-cgi
13415 root      18   0 86608  12m 1496 R  0.0  0.3   0:15.48 cpsrvd-ssl
32430 root      15   0 29600 4520 1044 R  0.0  0.1   0:00.83 tailwatchd

Both of these are only a few minutes apart.  As you can see Apache is using a lot of cpu resources.

If anyone has any suggestions for trouble shooting Apache?
Artysystem administrator
Top Expert 2007
Commented:
You can enable Extended statistics page for apache and see what pages take most cpu time:
http://httpd.apache.org/docs/2.2/mod/mod_status.html
then access your status page on http://yourhost.com/server-status URL

Next turn on logging of slow SQL queries:
http://search.mysql.com/search?q=Slow_query_log.html&lr=lang_en

I guess if this slow down became after reboot, your DB files may be corrupted (or having corrupted indexes) if so, repair your db (myisamchk tool, --myisam-recover option of mysqld if you are using MyISAM) or recreate indexes.

Author

Commented:
The suggestions partially helped resolve the issue with the balance coming from other sources.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial