drlchen
asked on
Apache High Server Load
Hello,
Recently I've been experiencing very high server loads on all 10 web servers. My normal load is about 9 but 2 times a day it shoots up to anywhere from 80 to 150
on all web servers. These web servers basically runs php and it currently is configured to run 7 virtual hosts . From what i read online it could be i/o or memory issues but
I don't see any wait or swapping when i run top and iostat. All servers normally have about 15 established httpd connection but when the load shoots up the ESTABLISHED connection
is about 150 with about 200 httpd process running. I don't see any increase in traffic, when i parse the access log. The count is about the same as the week before. When this happens
I notice the memcache connections spikes and the SELECT_SCAN line from my cacti graph shoots up and look exactly like the web server load graph. Also the Threads Created spikes along with the high load. So in order to resolve this load issue, I have to restart all the web servers. I also notice that if I get lucky and restart the offending web server, I don't have to restart the rest and the load starts dropping. This sounds like a mysql bottle neck or lots of bad sql queries with full scan but why all restart apache on 1 server can help reduce the load issue? These are 16 proc machines with with very little writes, all it does is take in connections from the loadbalancer, connect to memcache or mysql. Any suggestion on what else I should look into?
Recently I've been experiencing very high server loads on all 10 web servers. My normal load is about 9 but 2 times a day it shoots up to anywhere from 80 to 150
on all web servers. These web servers basically runs php and it currently is configured to run 7 virtual hosts . From what i read online it could be i/o or memory issues but
I don't see any wait or swapping when i run top and iostat. All servers normally have about 15 established httpd connection but when the load shoots up the ESTABLISHED connection
is about 150 with about 200 httpd process running. I don't see any increase in traffic, when i parse the access log. The count is about the same as the week before. When this happens
I notice the memcache connections spikes and the SELECT_SCAN line from my cacti graph shoots up and look exactly like the web server load graph. Also the Threads Created spikes along with the high load. So in order to resolve this load issue, I have to restart all the web servers. I also notice that if I get lucky and restart the offending web server, I don't have to restart the rest and the load starts dropping. This sounds like a mysql bottle neck or lots of bad sql queries with full scan but why all restart apache on 1 server can help reduce the load issue? These are 16 proc machines with with very little writes, all it does is take in connections from the loadbalancer, connect to memcache or mysql. Any suggestion on what else I should look into?
For the sites, or rather the databases: have you explored if they are correctly indexed? are the sites running with different databases or are some sites sharing the same database(s)? if so, how did you plan for potential conflicts/ contention?
Have you checked if malware has been injected on your system? (they can run as apache user & create a process called httpd...)
rkhunter & chkrootkit
http://rkhunter.sf.net/
http://www.chkrootkit.org/
If the bursts are predictable, maybe try to have a tcpdump run at the time it occurs and inspect the port accesses at that time?
rkhunter & chkrootkit
http://rkhunter.sf.net/
http://www.chkrootkit.org/
If the bursts are predictable, maybe try to have a tcpdump run at the time it occurs and inspect the port accesses at that time?
ASKER
Hello:
- is this happening for any server or are some of them "better" candidates?
This happens randomly any servers.
- any cron job running to these sites?
No we don't have any cron jobs running on these servers.
- is there some IP address which is connected when these troubles occur? it might be some genuine spider, or some site aspirator; or some site monitoring from a 3rd party, checking if there are any news in the RSS stream, or if your site is alive
I tried to look for the ip address through the access logs but can't really find much. But I guess what I didn't is review the IP addresses that is currently connected when the issues occurred. I'll keep that one in mind when it happens again.
Now to the sites themselves:
- there are lots of discussion as to permament connections to MySQL
- for my part, I usually prefer to handle myself, and so I do NOT use permanent connections BUT close the MySQL link as soon as this connection is no more needed in the php/html code:
I checked with the programmers and they said they don't use mysql_pconnect which i believe is the persistent connection you are talking about. But what i notice is that they seems to have persistent connection to the memcache server, though. Could this cause the problem?
For the sites who experiment the problem: are they running some cms, blog or e-commerce package?
We do run a wordpress package as for our websites but as for other activities it's all customs php codes.
We parse the slow query logs last night at 5pm and rewrote some slow queries. I'm not sure this solves the problem but for about 16 hours now, there are no spikes.
- is this happening for any server or are some of them "better" candidates?
This happens randomly any servers.
- any cron job running to these sites?
No we don't have any cron jobs running on these servers.
- is there some IP address which is connected when these troubles occur? it might be some genuine spider, or some site aspirator; or some site monitoring from a 3rd party, checking if there are any news in the RSS stream, or if your site is alive
I tried to look for the ip address through the access logs but can't really find much. But I guess what I didn't is review the IP addresses that is currently connected when the issues occurred. I'll keep that one in mind when it happens again.
Now to the sites themselves:
- there are lots of discussion as to permament connections to MySQL
- for my part, I usually prefer to handle myself, and so I do NOT use permanent connections BUT close the MySQL link as soon as this connection is no more needed in the php/html code:
I checked with the programmers and they said they don't use mysql_pconnect which i believe is the persistent connection you are talking about. But what i notice is that they seems to have persistent connection to the memcache server, though. Could this cause the problem?
For the sites who experiment the problem: are they running some cms, blog or e-commerce package?
We do run a wordpress package as for our websites but as for other activities it's all customs php codes.
We parse the slow query logs last night at 5pm and rewrote some slow queries. I'm not sure this solves the problem but for about 16 hours now, there are no spikes.
try to use pastmon
http://www.crypt.gen.nz/papers/pastmon.html
http://www.linux.com/archive/feature/153983
PasTmon is a good at capturing application performance info not only to find problems but also to capture and compare data before and after an upgrade, the database storage makes it easy to extract and analyze the data.
madunix
http://www.crypt.gen.nz/papers/pastmon.html
http://www.linux.com/archive/feature/153983
PasTmon is a good at capturing application performance info not only to find problems but also to capture and compare data before and after an upgrade, the database storage makes it easy to extract and analyze the data.
madunix
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Guys thanks for the help. The queries might be the bottleneck, as of now still haven't any problems with the load after the tuning. I'll take a look at pastmon and caching. Thanks.
Glad you've got some relief!
Remember thinking about caching queries for some time, so that the same page/query does not generate a complete new query.
B-) but I have no easy solution in mind
You might get a similar effect by creating a summarized query (and leaving it as flat table in MySQL) from your (presumably) complex complete query
Remember thinking about caching queries for some time, so that the same page/query does not generate a complete new query.
B-) but I have no easy solution in mind
You might get a similar effect by creating a summarized query (and leaving it as flat table in MySQL) from your (presumably) complex complete query
ASKER
After tuning the queries the problems complete went away its been gone for about a week now.
B-) Thx for the points and grade
- any cron job running to these sites?
- is there some IP address which is connected when these troubles occur? it might be some genuine spider, or some site aspirator; or some site monitoring from a 3rd party, checking if there are any news in the RSS stream, or if your site is alive
Now to the sites themselves:
- there are lots of discussion as to permament connections to MySQL
- for my part, I usually prefer to handle myself, and so I do NOT use permanent connections BUT close the MySQL link as soon as this connection is no more needed in the php/html code
For the sites who experiment the problem: are they running some cms, blog or e-commerce package?