Solved

PHP dealing with high load spikes.

Posted on 2011-09-13
6
577 Views
Last Modified: 2013-11-08
Hi,
We have several servers each with arround 50 -80 sites on them. Each site runs an instace of our large ecommerce system.
The problem we are getting is dealing with load on the server, but this seems mainly to be becuase of backups.
The backups start very early in the morning arround 4am, but some days have been continueing running right up till 1 or 2pm. at random times during the backups, the load will spike and the application will not be able to run, but then there are other times when the load is really high and the app can sort of still run, but 1. its slow, and 2. its probably not helping as it may be helping it reach such high loads. Load can spike from 3.4 - 5 right up to 500!
Aside from any ideas of how to fix the issue with the load from the backups, does anyone have any ideas about how the application should handle this. E.g. Should we try to detect the load at the start of the php application, and get it to sleep for a bit till its better, or exit/redirect to a static error page?
I realize that dealing with it at all at the php app it self level seems like avoiding the problem, but i think we may always get spikes in the load anyway from maybe spikes in visitors at random times and so this would help with all cases.
Thanks is anyone has any ideas.
0
Comment
Question by:stilliard
6 Comments
 
LVL 24

Accepted Solution

by:
slyong earned 167 total points
ID: 36533710
Hi,

I think you understand the disadvantage dealing with such situation using PHP.  However, if you want to have a script to check server load and redirect, you might want to try:

<?php
$load = sys_getloadavg();
if ($load[0] > 80) {
    header('HTTP/1.1 503 Too busy, try again later');
    die('Server too busy. Please try again later.');
}
?>

Open in new window


ref: http://de.php.net/manual/en/function.sys-getloadavg.php
0
 
LVL 24

Expert Comment

by:slyong
ID: 36533714
Sorry another way to do it is to set a timeout and register_shutdown_function to make PHP redirect to somewhere:

set_time_limit(1);
function shutdown () {
    if ( ! defined('FINISHED')) {
        header('Location: /timeout');
    }
}

register_shutdown_function('shutdown');

while (true) {}

define('FINISHED', 1);

Open in new window


ref: http://stackoverflow.com/questions/6759735/can-i-configure-apache-to-redirect-at-a-specified-timeout
0
 
LVL 10

Assisted Solution

by:ray-solomon
ray-solomon earned 166 total points
ID: 36534496
Some random insights/tips that may help.

I also have several servers as big as yours. The process of backing up sites to tar.gz files is very CPU intense and mysql consumes a lot of memory when databases are being dumped. This is normal. So during the backup process I make sure there's no other script on any of our sites that is running any big mysql queries.

I also adjust my backup to start late at night at 10pm when traffic is lower because it takes about 9 hours to complete. Better than during the day.

Make sure you are not actually running out of RAM and using swap during the backup process. If so, add more RAM or else apache and other things will start dying randomly. That will help keep your sites operating normally during a backup. Can't have too much ram.

I adjust most cron jobs around the backup process.

As for the php applications. If your applications are doing database queries, make sure you have the proper indexes on your tables where applicable. A lot of people I help get this wrong.

Raid 10 is the way to go if you can afford it. Otherwise throwing more RAM and better CPU is useless if your Hard drive read/write speeds cant keep up with all those sites doing things during the backup or heavy traffic.
0
VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

 
LVL 26

Expert Comment

by:arober11
ID: 36538744
What are you backing up?
0
 
LVL 6

Author Comment

by:stilliard
ID: 36542002
@slyong Thanks sys_getloadavg is great for telling the point when the app may need to be stoped, thanks i'll look into it more to decide the best cut off point.
@ray-solomon Thanks for the tips, i'll definetly be looking to through more RAM at it and look at some better hardware to support it.
@arober11 All the sites files and mysql db backups. However im glad you've asked hat question, as we could definetly look to cut down the parts we actually need backups of in terms of files. The sites currently all run the same system, so i gues the backups would only be needed for there own user files folder and any config files.
0
 
LVL 26

Assisted Solution

by:arober11
arober11 earned 167 total points
ID: 36545546
You could also set up another mySQL instance, on a separate server and use basic mySQL replication (master / slave) to maintain a remote copy of your DB's, then simply break the mirror, backup from the slave, then re-enable the replication. Would shift the majority of the load elsewhere along with offering some redundancy.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to dynamically set the form action using jQuery.

832 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question