Recently I've been experiencing very high server loads on all 10 web servers. My normal load is about 9 but 2 times a day it shoots up to anywhere from 80 to 150
on all web servers. These web servers basically runs php and it currently is configured to run 7 virtual hosts . From what i read online it could be i/o or memory issues but
I don't see any wait or swapping when i run top and iostat. All servers normally have about 15 established httpd connection but when the load shoots up the ESTABLISHED connection
is about 150 with about 200 httpd process running. I don't see any increase in traffic, when i parse the access log. The count is about the same as the week before. When this happens
I notice the memcache connections spikes and the SELECT_SCAN line from my cacti graph shoots up and look exactly like the web server load graph. Also the Threads Created spikes along with the high load. So in order to resolve this load issue, I have to restart all the web servers. I also notice that if I get lucky and restart the offending web server, I don't have to restart the rest and the load starts dropping. This sounds like a mysql bottle neck or lots of bad sql queries with full scan but why all restart apache on 1 server can help reduce the load issue? These are 16 proc machines with with very little writes, all it does is take in connections from the loadbalancer, connect to memcache or mysql. Any suggestion on what else I should look into?