High CPU load, slow page serving

I hope I am posting this in the right section, but I do believe the reason for the problem is Apache/MySQL.

I'm running rh9 on a Celeron 1.7. I'm currently serving about 100,000 pages per day, all of them are php pages and rely (more or less heavily) on a MySQL database. My problem is that with this volume (roughly translates to 3 gigs per day), the CPU is reaching 100% and processes start building up during peak times, pushing the server load beyond 10 up to as much as 50 (according to top).

 18:55:22  up 5 days, 22:56,  1 user,  load average: 7.31, 6.30, 5.92
162 processes: 141 sleeping, 21 running, 0 zombie, 0 stopped
CPU states:  84.0% user  15.9% system   0.0% nice   0.0% iowait   0.0% idle
Mem:  1022796k av,  792168k used,  230628k free,       0k shrd,    7900k buff
                    470300k actv,  122056k in_d,   18972k in_c
Swap: 1052248k av,   77848k used,  974400k free                  497896k cached

This snapshot is rather harmless, it can get much worse. As you can see, I have 1 gig of RAM installed and no, I am not swapping (so memory doesn't seem to be the problem). I do use the server for my own mail (qmail) and have to fight off a lot of spam (using spam assassin), but a basic and reliable test (apachectl stop :) reveals that the vast majority of the load is due to Apache/mySQL.

I am aware that php is CPU intensive, and my CPU is not the fastest around, but I believe it should be able to handle the load without the problems I am having. True?

P.S. I have installed Zend Accelerator to solve the problem - that get's me up to 150,00 pageviews and 4,5 gigs per day, but then the problem resurfaces.
Who is Participating?
Computer101Connect With a Mentor Commented:
PAQed, with points refunded (500)

E-E Admin
hi yosmc,

Since you pretty much isolated that Apache may be the the main contributor to the excessive load on the server, perhaps you may want to take a look at how Apache is doing, in term of it's processes.  Apache comes with a built-in "monitor", which is available using mod_status.  Take a look at Apache docs on mod_status (http://httpd.apache.org/docs-2.0/mod/mod_status.html) to enable it, and how to interpret the output.

Another thing that you could take a look at is Apache general tuning tips (http://httpd.apache.org/docs-2.0/misc/perf-tuning.html) which should give you an overview on what to look for in optimizing your Apache for performance.  I personally found the pages a bit too technical (at some times), but it is a good place to start with.

Some pages that I can search thru (http://www.google.com/search?q=%22apache+tuning%22+performance&hl=en&lr=&ie=UTF-8&start=10&sa=N)
- http://www.ece.concordia.ca/~daniel/tips/apache_tuning.html
- http://builder.com.com/5100-6387-1045523.html
- http://phplens.com/phpeverywhere/tuning-apache-php
- http://www.redhat.com/archives/tux-list/2001-November/msg00062.html
- http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-httpd-tuning-settings.html
- http://people.redhat.com/alikins/system_tuning.html#apache

* and a bunch of other.  I hope it could give you a good kick-start.

I personally would be happy to learn the best way to *boost* apache to the max :)

yosmcAuthor Commented:
Here goes my server-status (it roughly corresponds to s server-load of 7 via top):

Current Time: Wednesday, 25-Aug-2004 12:51:30 CDT
Restart Time: Wednesday, 25-Aug-2004 10:57:21 CDT
Parent Server Generation: 30
Server uptime: 1 hour 54 minutes 8 seconds
Total accesses: 205869 - Total Traffic: 669.8 MB
CPU Usage: u104.46 s8.27 cu.02 cs.01 - 1.65% CPU load
30.1 requests/sec - 100.2 kB/second - 3411 B/request
18 requests currently being processed, 10 idle workers

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M CPU  SS Req Conn Child Slot Client VHost Request
0-30 8913 0/627/7143 _  9.71 0 0 0.0 1.67 22.96 www.mysite.com GET / HTTP/1.1
1-30 9556 0/0/8520 _  1.10 0 0 0.0 0.00 28.03 www.mysite.com GET /forums/templates/subSilver/images/icon_mini_profile.gif HT
2-30 9010 1/451/8145 C  8.28 0 0 0.7 1.21 25.04 www.mysite.com GET /files/8right_up.gif HTTP/1.1
3-30 9540 0/19/7068 R  0.39 0 0 0.0 0.04 24.06  ? ? ..reading..  
4-30 8536 0/19/6125 W  0.34 889 0 0.0 0.03 19.61 www.mysite.com GET /forums/images/avatars/60.gif HTTP/1.
5-30 9541 1/17/5595 C  0.30 0 0 0.1 0.03 18.01 www.mysite.com GET /files/8ugang2.gif HTTP/1.1
6-30 9167 1/287/6997 C  5.14 0 0 0.4 0.65 22.64 www.mysite.com GET /files/sign.gif HTTP/1.1
7-30 9542 1/17/5596 C  0.07 0 0 0.2 0.02 16.39 www.mysite.com GET /files/links.gif HTTP/1.1
8-30 9145 1/262/5190 C  6.78 0 0 0.3 0.81 16.97 www.mysite.com GET /files/chips.gif HTTP/1.1

---SNIP--- (the list continues through 39)

As far as tuning is concerned, I believe I'm pretty much upto date, at least what the basic stuff is concerned. Here's a snippet from my httpd.conf:

KeepAlive Off
<IfModule prefork.c>
StartServers       5
MinSpareServers    5
MaxSpareServers   15
MaxClients       40
MaxRequestsPerChild  10000
HostnameLookups Off

I even avoid checking for .htaccess files to make things faster, and for the same reason I don't check follow SymLinks. Maybe I should add at this point that output is gzipped, which I guess puts some more strain on the CPU.
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

yosmcAuthor Commented:
UPDATE: I ran chkrootkit today just to be on the safe side, but nothing bad was found.

Here's a snippet from my ps -aux output:

apache   24752  2.2  1.3 56084 13504 ?       S    18:30   0:04 /usr/sbin/httpd -k start
apache   24778  3.5  1.1 55076 12032 ?       S    18:31   0:04 /usr/sbin/httpd -k start
apache   24779  1.5  1.2 55324 13032 ?       S    18:31   0:01 /usr/sbin/httpd -k start
apache   24780  2.8  1.2 55256 12516 ?       S    18:31   0:03 /usr/sbin/httpd -k start
apache   24795  3.3  1.2 55776 12496 ?       S    18:32   0:03 /usr/sbin/httpd -k start
apache   24796  1.9  1.2 55436 12672 ?       S    18:32   0:01 /usr/sbin/httpd -k start
apache   24806  1.5  1.1 55276 11804 ?       S    18:32   0:01 /usr/sbin/httpd -k start

As you can see, each Apache process takes 2-3% of CPU time. So if I get my maths right, with mysql and an occasional qmail process running, it takes 30 clients to eat up the rest of the CPU. Would you agree that I'm at least barking up the right tree?

I would suspect that you are right.  However, i can comment much on that since I do not have much expertise on Linux/Unix admin (only knew a few command -- "rm -rf *" :)

but what are there so many "httpd -k start".  It appears that you apache is spawning itself (dies, and restarting).

Try to compare the ps -aux output, and those on /server-status page, and compare the PID?

yosmcAuthor Commented:
Um, I didn't abandon the question, but the number of responses was limited. ;) (samri was very kind, but his input didn't help much.) I have concluded that my celeron processor simply wasn't good enough to handle the load, so I am upgrading to a faster CPU. Would it seem appropriate to close this thread and refund the points?
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.