?
Solved

Apache High Server Load

Posted on 2009-12-28
10
Medium Priority
?
654 Views
Last Modified: 2012-05-08
Hello,

Recently I've been experiencing very high server loads on all 10 web servers. My normal load is about 9 but 2 times a day it shoots up to anywhere from 80 to 150
on all web servers. These web servers basically runs php and it currently is configured to run 7 virtual hosts .  From what i read online it could be i/o or memory issues but
I don't see any wait or swapping when i run top and iostat. All servers normally have about  15 established httpd connection but when the load shoots up the ESTABLISHED connection
is about 150 with about 200 httpd process running. I don't see any increase in traffic, when i parse the access log. The count is about the same as the week before.  When this happens
I notice the memcache connections spikes and the SELECT_SCAN line from my cacti graph shoots up and look exactly like the web server load graph. Also the Threads Created spikes along with the high load. So in order to resolve this load issue, I have to restart all the web servers. I also notice that if I get lucky and restart the offending web server, I don't have to restart the rest and the load starts dropping.   This sounds like a mysql bottle neck or lots of bad sql queries with full scan but why all restart apache on 1 server can help reduce the load issue? These are 16 proc machines with with very little writes, all it does is take in connections from the loadbalancer, connect to memcache or mysql.  Any suggestion on what else I should look into?
0
Comment
Question by:drlchen
10 Comments
 
LVL 29

Expert Comment

by:fibo
ID: 26136830
- is this happening for any server or are some of them "better" candidates?
- any cron job running to these sites?
- is there some IP address which is connected when these troubles occur? it might be some genuine spider, or some site aspirator; or some site monitoring from a 3rd party, checking if there are any news in the RSS stream, or if your site is alive

Now to the sites themselves:
- there are lots of discussion as to permament connections to MySQL
- for my part, I usually prefer to handle myself, and so I do NOT use permanent connections BUT close the MySQL link as soon as this connection is no more needed in the php/html code

For the sites who experiment the problem: are they running some cms, blog or e-commerce package?  
0
 
LVL 29

Expert Comment

by:fibo
ID: 26136859
For the sites, or rather the databases: have you explored if they are correctly indexed? are the sites running with different databases or are some sites sharing the same database(s)? if so, how did you plan for potential conflicts/ contention?
0
 
LVL 41

Expert Comment

by:noci
ID: 26136880
Have you checked if malware has been injected on your system? (they can run as apache user & create a process called httpd...)

rkhunter & chkrootkit
http://rkhunter.sf.net/
http://www.chkrootkit.org/

If the bursts are predictable, maybe try to have a tcpdump run at the time it occurs and inspect the port accesses at that time?
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:drlchen
ID: 26140235
Hello:

- is this happening for any server or are some of them "better" candidates?
  This happens randomly any servers.

- any cron job running to these sites?
  No we don't have any cron jobs running on these servers.

- is there some IP address which is connected when these troubles occur? it might be some genuine spider, or some site aspirator; or some site monitoring from a 3rd party, checking if there are any news in the RSS stream, or if your site is alive
 I tried to look for the ip address through the access logs but can't really find much. But I guess what I didn't is review the IP addresses that is currently connected when the issues occurred. I'll keep that one in mind when it happens again.

Now to the sites themselves:
- there are lots of discussion as to permament connections to MySQL
- for my part, I usually prefer to handle myself, and so I do NOT use permanent connections BUT close the MySQL link as soon as this connection is no more needed in the php/html code:

 I checked with the programmers and they said they don't use mysql_pconnect which i believe is the persistent connection you are talking about.  But what i notice is that they seems to have persistent connection to the memcache server, though. Could this cause the problem?

For the sites who experiment the problem: are they running some cms, blog or e-commerce package?
We do run a wordpress package as for our websites but as for other activities it's all customs php codes.  

We parse the slow query logs last night at 5pm and rewrote some slow queries. I'm not sure this solves the problem but for about 16 hours now, there are no spikes.  

0
 
LVL 25

Expert Comment

by:madunix
ID: 26144972
try to use pastmon
http://www.crypt.gen.nz/papers/pastmon.html
http://www.linux.com/archive/feature/153983

PasTmon is a good at capturing application performance info not only to find problems but also to capture and compare data before and after an upgrade, the database storage makes it easy to extract and analyze the data.


madunix
0
 
LVL 29

Accepted Solution

by:
fibo earned 2000 total points
ID: 26151034
Thx for your answers. All seems right..

<<We parse the slow query logs last night at 5pm and rewrote some slow queries. I'm not sure this solves the problem but for about 16 hours now, there are no spikes.  >>
Some hope of a glimpse...
let's wait to see if this happens again...

On top of query optimization, start thinking to some cacheing process (eg, keep a query and its results for 120S before requerying)
That might lower the number of queries, and maybe contribute dissolving bottlenecks
0
 

Author Comment

by:drlchen
ID: 26163549
Guys thanks for the help.  The queries might be the bottleneck, as of now still haven't any problems with the load after the tuning.  I'll take a look at pastmon and caching. Thanks.
0
 
LVL 29

Expert Comment

by:fibo
ID: 26164926
Glad you've got some relief!

Remember thinking about caching queries for some time, so that the same page/query does not generate a complete new query.
B-) but I have no easy solution in mind
You might get a similar effect by creating a summarized query (and leaving it as flat table in MySQL) from your (presumably) complex complete query
0
 

Author Closing Comment

by:drlchen
ID: 31670629
After tuning the queries the problems complete went away its been gone for about a week now.
0
 
LVL 29

Expert Comment

by:fibo
ID: 26196217
B-) Thx for the points and grade
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post contains step-by-step instructions for setting up alerting in Percona Monitoring and Management (PMM) using Grafana.
In this blog, we’ll look at how improvements to Percona XtraDB Cluster improved IST performance.
In this video, Percona Solution Engineer Dimitri Vanoverbeke discusses why you want to use at least three nodes in a database cluster. To discuss how Percona Consulting can help with your design and architecture needs for your database and infras…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses
Course of the Month17 days, 13 hours left to enroll

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question