Solved

PHP dealing with high load spikes.

Posted on 2011-09-13
6
575 Views
Last Modified: 2013-11-08
Hi,
We have several servers each with arround 50 -80 sites on them. Each site runs an instace of our large ecommerce system.
The problem we are getting is dealing with load on the server, but this seems mainly to be becuase of backups.
The backups start very early in the morning arround 4am, but some days have been continueing running right up till 1 or 2pm. at random times during the backups, the load will spike and the application will not be able to run, but then there are other times when the load is really high and the app can sort of still run, but 1. its slow, and 2. its probably not helping as it may be helping it reach such high loads. Load can spike from 3.4 - 5 right up to 500!
Aside from any ideas of how to fix the issue with the load from the backups, does anyone have any ideas about how the application should handle this. E.g. Should we try to detect the load at the start of the php application, and get it to sleep for a bit till its better, or exit/redirect to a static error page?
I realize that dealing with it at all at the php app it self level seems like avoiding the problem, but i think we may always get spikes in the load anyway from maybe spikes in visitors at random times and so this would help with all cases.
Thanks is anyone has any ideas.
0
Comment
Question by:stilliard
6 Comments
 
LVL 24

Accepted Solution

by:
slyong earned 167 total points
Comment Utility
Hi,

I think you understand the disadvantage dealing with such situation using PHP.  However, if you want to have a script to check server load and redirect, you might want to try:

<?php
$load = sys_getloadavg();
if ($load[0] > 80) {
    header('HTTP/1.1 503 Too busy, try again later');
    die('Server too busy. Please try again later.');
}
?>

Open in new window


ref: http://de.php.net/manual/en/function.sys-getloadavg.php
0
 
LVL 24

Expert Comment

by:slyong
Comment Utility
Sorry another way to do it is to set a timeout and register_shutdown_function to make PHP redirect to somewhere:

set_time_limit(1);
function shutdown () {
    if ( ! defined('FINISHED')) {
        header('Location: /timeout');
    }
}

register_shutdown_function('shutdown');

while (true) {}

define('FINISHED', 1);

Open in new window


ref: http://stackoverflow.com/questions/6759735/can-i-configure-apache-to-redirect-at-a-specified-timeout
0
 
LVL 10

Assisted Solution

by:ray-solomon
ray-solomon earned 166 total points
Comment Utility
Some random insights/tips that may help.

I also have several servers as big as yours. The process of backing up sites to tar.gz files is very CPU intense and mysql consumes a lot of memory when databases are being dumped. This is normal. So during the backup process I make sure there's no other script on any of our sites that is running any big mysql queries.

I also adjust my backup to start late at night at 10pm when traffic is lower because it takes about 9 hours to complete. Better than during the day.

Make sure you are not actually running out of RAM and using swap during the backup process. If so, add more RAM or else apache and other things will start dying randomly. That will help keep your sites operating normally during a backup. Can't have too much ram.

I adjust most cron jobs around the backup process.

As for the php applications. If your applications are doing database queries, make sure you have the proper indexes on your tables where applicable. A lot of people I help get this wrong.

Raid 10 is the way to go if you can afford it. Otherwise throwing more RAM and better CPU is useless if your Hard drive read/write speeds cant keep up with all those sites doing things during the backup or heavy traffic.
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 26

Expert Comment

by:arober11
Comment Utility
What are you backing up?
0
 
LVL 6

Author Comment

by:stilliard
Comment Utility
@slyong Thanks sys_getloadavg is great for telling the point when the app may need to be stoped, thanks i'll look into it more to decide the best cut off point.
@ray-solomon Thanks for the tips, i'll definetly be looking to through more RAM at it and look at some better hardware to support it.
@arober11 All the sites files and mysql db backups. However im glad you've asked hat question, as we could definetly look to cut down the parts we actually need backups of in terms of files. The sites currently all run the same system, so i gues the backups would only be needed for there own user files folder and any config files.
0
 
LVL 26

Assisted Solution

by:arober11
arober11 earned 167 total points
Comment Utility
You could also set up another mySQL instance, on a separate server and use basic mySQL replication (master / slave) to maintain a remote copy of your DB's, then simply break the mirror, backup from the slave, then re-enable the replication. Would shift the majority of the load elsewhere along with offering some redundancy.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Developers of all skill levels should learn to use current best practices when developing websites. However many developers, new and old, fall into the trap of using deprecated features because this is what so many tutorials and books tell them to u…
1. Introduction As many people are interested in Linux but not as many are interested or knowledgeable (enough) to install Linux on their system, here is a safe way to try out Linux on your existing (Windows) system. The idea is that you insta…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to count occurrences of each item in an array.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now