Solved

Throttle shell_exec?

Posted on 2014-02-15
10
418 Views
Last Modified: 2014-05-12
We have a script that processes very large files occasionally. (EPS graphics for print). Some of the files reach 500M to 1G.

When we process those files on our development server, the server becomes unresponsive until the processing is complete.

Is there a way to limit the amount of CPU consumed when we use shell_exec to call imagemagick?

Is there a way to limit the cpu usage in any other way of calling imagemagick? (Perhaps spinning off a separate thread of apache or PHP an calling imagemagick the PHP way?)
0
Comment
Question by:DrDamnit
  • 4
  • 4
  • 2
10 Comments
 
LVL 16

Expert Comment

by:hankknight
ID: 39862881
PHP does not control the CPU usage of processes run using shell_exec.

However, you may have better success using  mogrify instead of convert.

http://www.imagemagick.org/www/mogrify.html
0
 
LVL 109

Expert Comment

by:Ray Paseur
ID: 39862895
Full disclosure: I have never tried this, but it seems intuitively feasible, and easy to test.

Try starting the processing script with a cURL POST-method request.  The script should be able to run asynchronously.  You may want some kind of a checkpoint file to indicate that the script is complete.

Here is a teaching example showing how to use cURL POST to start a separate script.

<?php // RAY_curl_post_async.php
error_reporting(E_ALL);


// DEMONSTRATE HOW TO USE CURL POST TO START AN ASYNCHRONOUS PROCESS


function curl_post($url, $post_array=array(), $timeout=2, $error_report=FALSE)
{
    // PREPARE THE POST STRING
    $post_string = NULL;
    foreach ($post_array as $key => $val)
    {
        $post_string .= $key . '=' . urlencode($val) . '&';
    }
    $post_string = rtrim($post_string, '&');

    // PREPARE THE CURL CALL
    $curl = curl_init();
    curl_setopt( $curl, CURLOPT_URL,            $url         );
    curl_setopt( $curl, CURLOPT_HEADER,         FALSE        );
    curl_setopt( $curl, CURLOPT_POST,           TRUE         );
    curl_setopt( $curl, CURLOPT_POSTFIELDS,     $post_string );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout     );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE         );

    // EXECUTE THE CURL CALL
    $htm = curl_exec($curl);
    $err = curl_errno($curl);
    $inf = curl_getinfo($curl);

    // ON FAILURE
    if (!$htm)
    {
        // PROCESS ERRORS HERE
        if ($error_report)
        {
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            echo "<pre>\n";
            var_dump($inf);
            echo "</pre>\n";
        }
        curl_close($curl);
        return FALSE;
    }

    // ON SUCCESS
    curl_close($curl);
    return $htm;
}


// USAGE EXAMPLE CREATES ASSOCIATIVE ARRAY OF KEY=>VALUE PAIRS
$args["name"]  = 'Ray';
$args["email"] = 'Ray.Paseur@Gmail.com';

// ACTIVATE THIS TO SEE THE ARRAY OF ARGS
// var_dump($args);

// SET THE URL
$url = "http://LAPRBass.com/RAY_bounce_post.php";

// CALL CURL TO POST THE DATA
$htm = curl_post($url, $args, 3, TRUE);

// SHOW WHAT CAME BACK, IF ANYTHING
if ($htm)
{
    echo "<pre>";
    echo htmlentities($htm);
}
else
{
    echo "NO RESPONSE YET FROM $url -- MAYBE BECAUSE IT IS RUNNING ASYNCHRONOUSLY";
}

Open in new window

HTH, ~Ray
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39862906
Ray, running this asynchronously will not improve CPU performance.
0
Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

 
LVL 109

Expert Comment

by:Ray Paseur
ID: 39862916
Right, I understand that.  But it may make the task subject to load balancing on the server.  Like I said, it's easy to test (and by implication easy to adopt or discard, depending on the outcome of the tests).
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39862927
0
 
LVL 32

Author Comment

by:DrDamnit
ID: 39865082
@hank - I'll try mogrify, but I think the problem stems from the hashing that is done when we use identify to determine the resolution and other information. We don't need all the information from identify, is there a way to only pull small information like the resolution without pulling all the identify data?

@ray - we'll certainly try this, but I think imagemagick is running the CPU on the dev box to 100% causing it to be unresponsive. Unless there is a way I can throttle the CPU, starting it with cURL would (likewise) run it to 100%.
0
 
LVL 32

Author Comment

by:DrDamnit
ID: 39868776
What do you guys think of this?

1. Setup a separate instance of apache running as a different user and responding to a different port.
2. Use CPULimit (http://www.howtoforge.com/how-to-limit-cpu-usage-of-a-process-with-cpulimit-debian-ubuntu) to limit the cpu usage of this instance to some value less than 100%.
3. Use cURL to trigger the mogrify command on that instance.

Long way around the barn or good solution?
0
 
LVL 32

Accepted Solution

by:
DrDamnit earned 0 total points
ID: 39901145
bump?
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39902085
Your three-step plan just might work.  Give it a try!
0
 
LVL 32

Author Closing Comment

by:DrDamnit
ID: 40059445
This is what we did.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will explain how to display the first page of your Microsoft Word documents (e.g. .doc, .docx, etc...) as images in a web page programatically. I have scoured the web on a way to do this unsuccessfully. The goal is to produce something …
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question