Solved

Throttle shell_exec?

Posted on 2014-02-15
10
411 Views
Last Modified: 2014-05-12
We have a script that processes very large files occasionally. (EPS graphics for print). Some of the files reach 500M to 1G.

When we process those files on our development server, the server becomes unresponsive until the processing is complete.

Is there a way to limit the amount of CPU consumed when we use shell_exec to call imagemagick?

Is there a way to limit the cpu usage in any other way of calling imagemagick? (Perhaps spinning off a separate thread of apache or PHP an calling imagemagick the PHP way?)
0
Comment
Question by:DrDamnit
  • 4
  • 4
  • 2
10 Comments
 
LVL 16

Expert Comment

by:hankknight
ID: 39862881
PHP does not control the CPU usage of processes run using shell_exec.

However, you may have better success using  mogrify instead of convert.

http://www.imagemagick.org/www/mogrify.html
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39862895
Full disclosure: I have never tried this, but it seems intuitively feasible, and easy to test.

Try starting the processing script with a cURL POST-method request.  The script should be able to run asynchronously.  You may want some kind of a checkpoint file to indicate that the script is complete.

Here is a teaching example showing how to use cURL POST to start a separate script.

<?php // RAY_curl_post_async.php
error_reporting(E_ALL);


// DEMONSTRATE HOW TO USE CURL POST TO START AN ASYNCHRONOUS PROCESS


function curl_post($url, $post_array=array(), $timeout=2, $error_report=FALSE)
{
    // PREPARE THE POST STRING
    $post_string = NULL;
    foreach ($post_array as $key => $val)
    {
        $post_string .= $key . '=' . urlencode($val) . '&';
    }
    $post_string = rtrim($post_string, '&');

    // PREPARE THE CURL CALL
    $curl = curl_init();
    curl_setopt( $curl, CURLOPT_URL,            $url         );
    curl_setopt( $curl, CURLOPT_HEADER,         FALSE        );
    curl_setopt( $curl, CURLOPT_POST,           TRUE         );
    curl_setopt( $curl, CURLOPT_POSTFIELDS,     $post_string );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout     );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE         );

    // EXECUTE THE CURL CALL
    $htm = curl_exec($curl);
    $err = curl_errno($curl);
    $inf = curl_getinfo($curl);

    // ON FAILURE
    if (!$htm)
    {
        // PROCESS ERRORS HERE
        if ($error_report)
        {
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            echo "<pre>\n";
            var_dump($inf);
            echo "</pre>\n";
        }
        curl_close($curl);
        return FALSE;
    }

    // ON SUCCESS
    curl_close($curl);
    return $htm;
}


// USAGE EXAMPLE CREATES ASSOCIATIVE ARRAY OF KEY=>VALUE PAIRS
$args["name"]  = 'Ray';
$args["email"] = 'Ray.Paseur@Gmail.com';

// ACTIVATE THIS TO SEE THE ARRAY OF ARGS
// var_dump($args);

// SET THE URL
$url = "http://LAPRBass.com/RAY_bounce_post.php";

// CALL CURL TO POST THE DATA
$htm = curl_post($url, $args, 3, TRUE);

// SHOW WHAT CAME BACK, IF ANYTHING
if ($htm)
{
    echo "<pre>";
    echo htmlentities($htm);
}
else
{
    echo "NO RESPONSE YET FROM $url -- MAYBE BECAUSE IT IS RUNNING ASYNCHRONOUSLY";
}

Open in new window

HTH, ~Ray
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39862906
Ray, running this asynchronously will not improve CPU performance.
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39862916
Right, I understand that.  But it may make the task subject to load balancing on the server.  Like I said, it's easy to test (and by implication easy to adopt or discard, depending on the outcome of the tests).
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39862927
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 32

Author Comment

by:DrDamnit
ID: 39865082
@hank - I'll try mogrify, but I think the problem stems from the hashing that is done when we use identify to determine the resolution and other information. We don't need all the information from identify, is there a way to only pull small information like the resolution without pulling all the identify data?

@ray - we'll certainly try this, but I think imagemagick is running the CPU on the dev box to 100% causing it to be unresponsive. Unless there is a way I can throttle the CPU, starting it with cURL would (likewise) run it to 100%.
0
 
LVL 32

Author Comment

by:DrDamnit
ID: 39868776
What do you guys think of this?

1. Setup a separate instance of apache running as a different user and responding to a different port.
2. Use CPULimit (http://www.howtoforge.com/how-to-limit-cpu-usage-of-a-process-with-cpulimit-debian-ubuntu) to limit the cpu usage of this instance to some value less than 100%.
3. Use cURL to trigger the mogrify command on that instance.

Long way around the barn or good solution?
0
 
LVL 32

Accepted Solution

by:
DrDamnit earned 0 total points
ID: 39901145
bump?
0
 
LVL 16

Expert Comment

by:hankknight
ID: 39902085
Your three-step plan just might work.  Give it a try!
0
 
LVL 32

Author Closing Comment

by:DrDamnit
ID: 40059445
This is what we did.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Introduction Many web sites contain image galleries; a common design for these galleries includes a page with a collection of thumbnail images.  You can click on each of the thumbnail images to see the larger version of the image.  This is easily i…
Generating table dynamically is the most common issue faced by php developers.... So it seems there is a need of an article that explains the basic concept of generating tables dynamically. It just requires a basic knowledge of html and little maths…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now