PHP dealing with high load spikes.

Posted on 2011-09-13
Medium Priority
Last Modified: 2013-11-08
We have several servers each with arround 50 -80 sites on them. Each site runs an instace of our large ecommerce system.
The problem we are getting is dealing with load on the server, but this seems mainly to be becuase of backups.
The backups start very early in the morning arround 4am, but some days have been continueing running right up till 1 or 2pm. at random times during the backups, the load will spike and the application will not be able to run, but then there are other times when the load is really high and the app can sort of still run, but 1. its slow, and 2. its probably not helping as it may be helping it reach such high loads. Load can spike from 3.4 - 5 right up to 500!
Aside from any ideas of how to fix the issue with the load from the backups, does anyone have any ideas about how the application should handle this. E.g. Should we try to detect the load at the start of the php application, and get it to sleep for a bit till its better, or exit/redirect to a static error page?
I realize that dealing with it at all at the php app it self level seems like avoiding the problem, but i think we may always get spikes in the load anyway from maybe spikes in visitors at random times and so this would help with all cases.
Thanks is anyone has any ideas.
Question by:stilliard
LVL 24

Accepted Solution

slyong earned 668 total points
ID: 36533710

I think you understand the disadvantage dealing with such situation using PHP.  However, if you want to have a script to check server load and redirect, you might want to try:

$load = sys_getloadavg();
if ($load[0] > 80) {
    header('HTTP/1.1 503 Too busy, try again later');
    die('Server too busy. Please try again later.');

Open in new window

ref: http://de.php.net/manual/en/function.sys-getloadavg.php
LVL 24

Expert Comment

ID: 36533714
Sorry another way to do it is to set a timeout and register_shutdown_function to make PHP redirect to somewhere:

function shutdown () {
    if ( ! defined('FINISHED')) {
        header('Location: /timeout');


while (true) {}

define('FINISHED', 1);

Open in new window

ref: http://stackoverflow.com/questions/6759735/can-i-configure-apache-to-redirect-at-a-specified-timeout
LVL 10

Assisted Solution

ray-solomon earned 664 total points
ID: 36534496
Some random insights/tips that may help.

I also have several servers as big as yours. The process of backing up sites to tar.gz files is very CPU intense and mysql consumes a lot of memory when databases are being dumped. This is normal. So during the backup process I make sure there's no other script on any of our sites that is running any big mysql queries.

I also adjust my backup to start late at night at 10pm when traffic is lower because it takes about 9 hours to complete. Better than during the day.

Make sure you are not actually running out of RAM and using swap during the backup process. If so, add more RAM or else apache and other things will start dying randomly. That will help keep your sites operating normally during a backup. Can't have too much ram.

I adjust most cron jobs around the backup process.

As for the php applications. If your applications are doing database queries, make sure you have the proper indexes on your tables where applicable. A lot of people I help get this wrong.

Raid 10 is the way to go if you can afford it. Otherwise throwing more RAM and better CPU is useless if your Hard drive read/write speeds cant keep up with all those sites doing things during the backup or heavy traffic.
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

LVL 26

Expert Comment

ID: 36538744
What are you backing up?

Author Comment

ID: 36542002
@slyong Thanks sys_getloadavg is great for telling the point when the app may need to be stoped, thanks i'll look into it more to decide the best cut off point.
@ray-solomon Thanks for the tips, i'll definetly be looking to through more RAM at it and look at some better hardware to support it.
@arober11 All the sites files and mysql db backups. However im glad you've asked hat question, as we could definetly look to cut down the parts we actually need backups of in terms of files. The sites currently all run the same system, so i gues the backups would only be needed for there own user files folder and any config files.
LVL 26

Assisted Solution

arober11 earned 668 total points
ID: 36545546
You could also set up another mySQL instance, on a separate server and use basic mySQL replication (master / slave) to maintain a remote copy of your DB's, then simply break the mirror, backup from the slave, then re-enable the replication. Would shift the majority of the load elsewhere along with offering some redundancy.

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Build an array called $myWeek which will hold the array elements Today, Yesterday and then builds up the rest of the week by the name of the day going back 1 week.   (CODE) (CODE) Then you just need to pass your date to the function. If i…
These days socially coordinated efforts have turned into a critical requirement for enterprises.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

600 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question