Solved

High CPU load, slow page serving

Posted on 2004-08-24
8
242 Views
Last Modified: 2010-03-04
I hope I am posting this in the right section, but I do believe the reason for the problem is Apache/MySQL.

I'm running rh9 on a Celeron 1.7. I'm currently serving about 100,000 pages per day, all of them are php pages and rely (more or less heavily) on a MySQL database. My problem is that with this volume (roughly translates to 3 gigs per day), the CPU is reaching 100% and processes start building up during peak times, pushing the server load beyond 10 up to as much as 50 (according to top).

 18:55:22  up 5 days, 22:56,  1 user,  load average: 7.31, 6.30, 5.92
162 processes: 141 sleeping, 21 running, 0 zombie, 0 stopped
CPU states:  84.0% user  15.9% system   0.0% nice   0.0% iowait   0.0% idle
Mem:  1022796k av,  792168k used,  230628k free,       0k shrd,    7900k buff
                    470300k actv,  122056k in_d,   18972k in_c
Swap: 1052248k av,   77848k used,  974400k free                  497896k cached

This snapshot is rather harmless, it can get much worse. As you can see, I have 1 gig of RAM installed and no, I am not swapping (so memory doesn't seem to be the problem). I do use the server for my own mail (qmail) and have to fight off a lot of spam (using spam assassin), but a basic and reliable test (apachectl stop :) reveals that the vast majority of the load is due to Apache/mySQL.

I am aware that php is CPU intensive, and my CPU is not the fastest around, but I believe it should be able to handle the load without the problems I am having. True?

P.S. I have installed Zend Accelerator to solve the problem - that get's me up to 150,00 pageviews and 4,5 gigs per day, but then the problem resurfaces.
0
Comment
Question by:yosmc
  • 3
  • 2
8 Comments
 
LVL 15

Expert Comment

by:samri
ID: 11894320
hi yosmc,

Since you pretty much isolated that Apache may be the the main contributor to the excessive load on the server, perhaps you may want to take a look at how Apache is doing, in term of it's processes.  Apache comes with a built-in "monitor", which is available using mod_status.  Take a look at Apache docs on mod_status (http://httpd.apache.org/docs-2.0/mod/mod_status.html) to enable it, and how to interpret the output.

Another thing that you could take a look at is Apache general tuning tips (http://httpd.apache.org/docs-2.0/misc/perf-tuning.html) which should give you an overview on what to look for in optimizing your Apache for performance.  I personally found the pages a bit too technical (at some times), but it is a good place to start with.

Some pages that I can search thru (http://www.google.com/search?q=%22apache+tuning%22+performance&hl=en&lr=&ie=UTF-8&start=10&sa=N)
- http://www.ece.concordia.ca/~daniel/tips/apache_tuning.html
- http://builder.com.com/5100-6387-1045523.html
- http://phplens.com/phpeverywhere/tuning-apache-php
- http://www.redhat.com/archives/tux-list/2001-November/msg00062.html
- http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-httpd-tuning-settings.html
- http://people.redhat.com/alikins/system_tuning.html#apache

* and a bunch of other.  I hope it could give you a good kick-start.

I personally would be happy to learn the best way to *boost* apache to the max :)

Cheers
0
 

Author Comment

by:yosmc
ID: 11895461
Here goes my server-status (it roughly corresponds to s server-load of 7 via top):

Current Time: Wednesday, 25-Aug-2004 12:51:30 CDT
Restart Time: Wednesday, 25-Aug-2004 10:57:21 CDT
Parent Server Generation: 30
Server uptime: 1 hour 54 minutes 8 seconds
Total accesses: 205869 - Total Traffic: 669.8 MB
CPU Usage: u104.46 s8.27 cu.02 cs.01 - 1.65% CPU load
30.1 requests/sec - 100.2 kB/second - 3411 B/request
18 requests currently being processed, 10 idle workers
__CRWCCCC.W__CR____C...R..C._WW..W.WR._.........................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M CPU  SS Req Conn Child Slot Client VHost Request
0-30 8913 0/627/7143 _  9.71 0 0 0.0 1.67 22.96  192.234.106.2 www.mysite.com GET / HTTP/1.1
1-30 9556 0/0/8520 _  1.10 0 0 0.0 0.00 28.03  198.103.111.110 www.mysite.com GET /forums/templates/subSilver/images/icon_mini_profile.gif HT
2-30 9010 1/451/8145 C  8.28 0 0 0.7 1.21 25.04  66.43.162.74 www.mysite.com GET /files/8right_up.gif HTTP/1.1
3-30 9540 0/19/7068 R  0.39 0 0 0.0 0.04 24.06  ? ? ..reading..  
4-30 8536 0/19/6125 W  0.34 889 0 0.0 0.03 19.61  66.36.74.231 www.mysite.com GET /forums/images/avatars/60.gif HTTP/1.
5-30 9541 1/17/5595 C  0.30 0 0 0.1 0.03 18.01  66.43.162.74 www.mysite.com GET /files/8ugang2.gif HTTP/1.1
6-30 9167 1/287/6997 C  5.14 0 0 0.4 0.65 22.64  199.0.65.2 www.mysite.com GET /files/sign.gif HTTP/1.1
7-30 9542 1/17/5596 C  0.07 0 0 0.2 0.02 16.39  199.0.65.2 www.mysite.com GET /files/links.gif HTTP/1.1
8-30 9145 1/262/5190 C  6.78 0 0 0.3 0.81 16.97  199.0.65.2 www.mysite.com GET /files/chips.gif HTTP/1.1

---SNIP--- (the list continues through 39)

As far as tuning is concerned, I believe I'm pretty much upto date, at least what the basic stuff is concerned. Here's a snippet from my httpd.conf:

KeepAlive Off
<IfModule prefork.c>
StartServers       5
MinSpareServers    5
MaxSpareServers   15
MaxClients       40
MaxRequestsPerChild  10000
</IfModule>
HostnameLookups Off

I even avoid checking for .htaccess files to make things faster, and for the same reason I don't check follow SymLinks. Maybe I should add at this point that output is gzipped, which I guess puts some more strain on the CPU.
0
 

Author Comment

by:yosmc
ID: 11908567
UPDATE: I ran chkrootkit today just to be on the safe side, but nothing bad was found.

Here's a snippet from my ps -aux output:

apache   24752  2.2  1.3 56084 13504 ?       S    18:30   0:04 /usr/sbin/httpd -k start
apache   24778  3.5  1.1 55076 12032 ?       S    18:31   0:04 /usr/sbin/httpd -k start
apache   24779  1.5  1.2 55324 13032 ?       S    18:31   0:01 /usr/sbin/httpd -k start
apache   24780  2.8  1.2 55256 12516 ?       S    18:31   0:03 /usr/sbin/httpd -k start
apache   24795  3.3  1.2 55776 12496 ?       S    18:32   0:03 /usr/sbin/httpd -k start
apache   24796  1.9  1.2 55436 12672 ?       S    18:32   0:01 /usr/sbin/httpd -k start
apache   24806  1.5  1.1 55276 11804 ?       S    18:32   0:01 /usr/sbin/httpd -k start

As you can see, each Apache process takes 2-3% of CPU time. So if I get my maths right, with mysql and an occasional qmail process running, it takes 30 clients to eat up the rest of the CPU. Would you agree that I'm at least barking up the right tree?
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 15

Expert Comment

by:samri
ID: 11917211
yosmc,

I would suspect that you are right.  However, i can comment much on that since I do not have much expertise on Linux/Unix admin (only knew a few command -- "rm -rf *" :)

but what are there so many "httpd -k start".  It appears that you apache is spawning itself (dies, and restarting).

Try to compare the ps -aux output, and those on /server-status page, and compare the PID?

0
 

Author Comment

by:yosmc
ID: 12145898
Um, I didn't abandon the question, but the number of responses was limited. ;) (samri was very kind, but his input didn't help much.) I have concluded that my celeron processor simply wasn't good enough to handle the load, so I am upgrading to a faster CPU. Would it seem appropriate to close this thread and refund the points?
0
 
LVL 1

Accepted Solution

by:
Computer101 earned 0 total points
ID: 12207778
PAQed, with points refunded (500)

Computer101
E-E Admin
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

In my time as an SEO for the last 2 years and in the questions I have assisted with on here I have always seen the need to redirect from non-www urls to their www versions. For instance redirecting http://domain.com (http://domain.com) to http…
As Wikipedia explains 'robots.txt' as -- the robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a websit…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now