Solved

php curl loop - making it more efficient

Posted on 2013-06-02
3
1,087 Views
Last Modified: 2013-06-07
Hi experts,

many thanks for taking a look at my question.  Essentially i have a page that gets xml data fed to it, which i then use  SimpleXML to loop through - doing a curl call each loop and giving me back a response that i then decode with json - add to a variable then write a file at the end.

my main issue is that i need to dramatically speed up the process.  I currently have it doing approx 3 calls per second which on large data sets, will take hours to complete.

could someone please show me how i could optimise the loop a little more?  I have already move the init_curl out of the loop so it doesn't do this multiple times any more.

your help is most appreciated.  I have attached part of code below.

date_default_timezone_set("Australia/Brisbane"); 
define("API_BASEURL", "https://rest.test.com/");
define("API_VERSION", "2012-12-01");

//echo "<pre>";
$ch = '';
// data
$tstxml = $_POST["xml_data"];
//echo '<br><br><br>This is the xml_data posted'.$tstxml.'<br><br>';
//get the rest of the variables from the form
	$username = $_POST["user"];
	$password = $_POST["pass"];
	$action = $_POST["action"];
$filename = $_POST["filename"];
$datevar = date('YmdHis');
if ($filename =='') 
{
	$filename = 'data';
}
$filename = $filename.$datevar.'.csv';
//echo $filename.'<br><br>';
//Set the URL for the SMS Submission
$url = "https://rest.test.com/wash";

//echo 'This is the XML Data from the page: '.$tstxml.'<br><br><br>';


// MAKE AN OBJECT
$obj = SimpleXML_Load_String($tstxml);

// ACTIVATE THIS TO SEE THE OBJECT
// var_dump($obj);

// USE AN ITERATOR TO ACCESS PROPERTIES OF THE OBJECT
$xmlresult = '';
$fp = fopen($filename, 'w');
$xmlresult == '';
fwrite($fp, "File Num, Num Chkd, Result,Date, time\r\n");
   $ch = curl_init();
   foreach ($obj as $numver)
{
    $user = $_POST["user"];
	$pass = $_POST["pass"];
	$action = $_POST["action"];
	$dest = (string)$numver->recipient;
    $filenum = (string)$numver->file;
	$postfields = array("destination" => $dest);
	//$str = '?user='.urlencode($user).'&pass='.urlencode($pass).'&action='.urlencode($action).'&destination='.urlencode($dest);
      // Open connection
       // Set up the cURL object
	
curl_setopt($ch, CURLOPT_URL, API_BASEURL . API_VERSION . "/" . $action);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, $user . ":" . $pass);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

// Execute the request
$result = curl_exec($ch);
//var_dump($result);
// Get status info on the cURL request
$response_info = curl_getinfo($ch);

//var_dump($response_info);
if(($response_info["http_code"] >= 200) AND ($response_info["http_code"] <= 299)){      // Success

        // Decode the json response
        $json = json_decode($result);
//echo $json->wash[0]->id;
$jsvar = $json->wash[0]->id;
$datevar = date('d/m/Y');
$datetimenow = date('h:i:s A');
//fwrite($fp, $filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n");
//echo '<br><Br>';
//echo 'JM ws ere<br><br>';
        // json_decode will return null on failure



}
 

$xmlresult = $xmlresult.$filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n";

}
 fwrite($fp, $xmlresult);
 fclose($fp); 
curl_close($ch);

Open in new window

0
Comment
Question by:mavmanau
  • 2
3 Comments
 
LVL 108

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 39215744
After clearing up the extraneous stuff, this appears to be the script you're using.  But it looks like you've changed the URLs so nobody but you can test it.  That's too bad, because we really kind of need the SSCCE to begin to discern why this is so slow.

One thing you might want to consider is whether the elapsed time is coming from the cURL call or whether it's coming from your script.  If it's the external web service, your only recourse will be to try to get the publishers to give you a more efficient API.

You might also try omitting the call to curl_getinfo() and just testing the $result variable for FALSE.  This would reduce the program path by a few instructions.  There are other inefficiencies, like doing this over and over inside a loop: $user = $_POST["user"]; but that sort of thing is not likely to cause much delay.

<?php // RAY_temp_mavmanau.php
error_reporting(E_ALL);

date_default_timezone_set("Australia/Brisbane"); 
define("API_BASEURL", "https://rest.test.com/");
define("API_VERSION", "2012-12-01");

$ch = '';

$tstxml   = $_POST["xml_data"];
$username = $_POST["user"];
$password = $_POST["pass"];
$action   = $_POST["action"];
$filename = $_POST["filename"];
$datevar  = date('YmdHis');
if ($filename =='') 
{
	$filename = 'data';
}
$filename = $filename.$datevar.'.csv';

$url = "https://rest.test.com/wash";

// MAKE AN OBJECT
$obj = SimpleXML_Load_String($tstxml);

// ACTIVATE THIS TO SEE THE OBJECT
// var_dump($obj);

// USE AN ITERATOR TO ACCESS PROPERTIES OF THE OBJECT
$xmlresult = '';
$fp = fopen($filename, 'w');
fwrite($fp, "File Num, Num Chkd, Result,Date, time\r\n");
$ch = curl_init();
foreach ($obj as $numver)
{
    $user    = $_POST["user"];
    $pass    = $_POST["pass"];
    $action  = $_POST["action"];
    $dest    = (string)$numver->recipient;
    $filenum = (string)$numver->file;
    $postfields = array("destination" => $dest);

    curl_setopt($ch, CURLOPT_URL, API_BASEURL . API_VERSION . "/" . $action);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_USERPWD, $user . ":" . $pass);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

    $result = curl_exec($ch);
    $response_info = curl_getinfo($ch);
    if(($response_info["http_code"] >= 200) AND ($response_info["http_code"] <= 299)) 
    {

        $json = json_decode($result);
        $jsvar = $json->wash[0]->id;
        $datevar = date('d/m/Y');
        $datetimenow = date('h:i:s A');
    }

    $xmlresult = $xmlresult.$filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n";
}
fwrite($fp, $xmlresult);
fclose($fp); 
curl_close($ch);

Open in new window

Please see the snippet below to learn how to attach a script timer to all or part of a PHP script.  In addition to the class itself, there are some usage examples below line 160.

<?php // RAY_class_Stopwatch.php
error_reporting(E_ALL);


// DEMONSTRATE A SCRIPT TIMER FOR ALL OR PART OF A SCRIPT PHP 5+
// MAN PAGE http://php.net/manual/en/function.microtime.php


class StopWatch
{
    protected $a; // START TIME
    protected $s; // STATUS - IF RUNNING
    protected $z; // STOP TIME

    public function __construct()
    {
        $this->a = array();
        $this->s = array();
        $this->z = array();
    }

    // A METHOD TO PROVIDE A FINAL READOUT, IF NEEDED
    public function __destruct()
    {
        $ret = $this->readout();
        if (!$ret) return FALSE;
        echo
          __CLASS__
        . '::'
        . __FUNCTION__
        . '() '
        ;
        echo "<b>$ret</b>";
        echo PHP_EOL;
    }

    // A METHOD TO REMOVE A TIMER
    public function reset($name='TIMER')
    {
        // RESET ALL TIMERS
        if ($name == 'TIMER')
        {
            $this->__construct();
        }
        else
        {
            unset($this->a[$name]);
            unset($this->s[$name]);
            unset($this->z[$name]);
        }
    }

    // A METHOD TO CAPTURE THE START TIME
    public function start($name='TIMER')
    {
        $this->a[$name] = microtime(TRUE);
        $this->z[$name] = $this->a[$name];
        $this->s[$name] = 'RUNNING';
    }

    // A METHOD TO CAPTURE THE END TIME
    public function stop($name='TIMER')
    {
        $ret = NULL;

        // STOP ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                // IF THIS TIMER IS STILL RUNNING, STOP IT
                if ($this->s[$name])
                {
                    $this->s[$name] = FALSE;
                    $this->z[$name] = microtime(TRUE);
                }
            }
        }

        // STOP ONLY ONE OF THE TIMERS
        else
        {
            if ($this->s[$name])
            {
                $this->s[$name] = FALSE;
                $this->z[$name] = microtime(TRUE);
            }
            else
            {
                $ret .= "ERROR: CALL TO STOP() METHOD: '$name' IS NOT RUNNING";
            }
        }

        // RETURN AN ERROR MESSAGE, IF ANY
        return $ret;
    }

    // A METHOD TO READ OUT THE TIMER(S)
    public function readout($name='TIMER', $dec=3, $m=1000, $t = 'ms', $eol=PHP_EOL)
    {
        $str = NULL;

        // GET READOUTS FOR ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                $str .= $name;

                // IF THIS TIMER IS STILL RUNNING UPDATE THE END TIME
                if ($this->s[$name])
                {
                    $this->z[$name] = microtime(TRUE);
                    $str .= " RUNNING ";
                }
                else
                {
                    $str .= " STOPPED ";
                }

                // RETURN A DISPLAY STRING
                $lapse_time = $this->z[$name] - $start_time;
                $lapse_msec = $lapse_time * $m;
                $lapse_echo = number_format($lapse_msec, $dec);
                $str .= " $lapse_echo $t";
                $str .= $eol;
            }
            return $str;
        }

        // GET A READOUT FOR ONLY ONE TIMER
        else
        {
            $str .= $name;

            // IF THIS TIME IS STILL RUNNING, UPDATE THE END TIME
            if ($this->s[$name])
            {
                $this->z[$name] = microtime(TRUE);
                $str .= " RUNNING ";
            }
            else
            {
                $str .= " STOPPED ";
            }

            // RETURN A DISPLAY STRING
            $lapse_time = $this->z[$name] - $this->a[$name];
            $lapse_msec = $lapse_time * $m;
            $lapse_echo = number_format($lapse_msec, $dec);
            $str .= " $lapse_echo $t";
            $str .= $eol;
            return $str;
        }
    }
}



// DEMONSTRATE THE USE -- INSTANTIATE THE STOPWATCH OBJECT
$sw  = new Stopwatch;

// SET A STOPWATCH NAME THAT REFLECTS THE PARTS OF THE SCRIPT WE WANT TO TIME
$g_timer = 'GOOGLE TIMER';

// START A TIMER TO GET ELAPSED TIME FOR A CALL TO GOOGLE
$sw->start($g_timer);

// PERFORM SOME ACTIVITY THAT YOU WANT TO TIME (READS GOOGLE WEB PAGE)
$page = 'https://www.google.com';
$html = file_get_contents($page);

// GET A READOUT OF THE TIMER WHILE IT IS STILL RUNNING
echo nl2br($sw->readout($g_timer));
echo "<br/>" . PHP_EOL;

// PERFORM SOME OTHER ACTIVITY (READS GOOGLE WEB PAGE AGAIN)
$page = 'https://www.google.com';
$html = file_get_contents($page);

// STOP THE TIMER AND GET A READOUT WITH SHORT DECIMALS
$x = $sw->stop($g_timer);
echo nl2br($sw->readout($g_timer, 1));
echo "<br/>" . PHP_EOL;




// START A SECOND TIMER
$y_timer = 'YAHOO TIMER';
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT (ONE IS STOPPED AND ONE IS STILL RUNNING)
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

// STOP ALL OF THE STOPWATCHES
$sw->stop();

// REPORT THE STOPWATCHES CONTENT AGAIN
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// TRY TO STOP A TIMER THAT IS NOT RUNNING
$x = $sw->stop($g_timer);
var_dump($x);
echo "<br/>" . PHP_EOL;
echo "<br/>" . PHP_EOL;



// START THIS TIMER OVER AGAIN
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://weather.yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ONE OF THE STOPWATCHES
$sw->reset($g_timer);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ALL OF THE STOPWATCHES
$sw->reset();
echo "ALL STOPWATCHES HAVE BEEN REMOVED";

// REPORT THE STOPWATCHES CONTENT (SHOWS NOTHING)
echo nl2br($sw->readout());

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

// SHOW THE DESTRUCTOR IN ACTION
$sw->start('Foo');
echo "<pre>";
unset($sw);
echo "</pre>";

Open in new window

0
 

Author Closing Comment

by:mavmanau
ID: 39228864
perfect, thank you very much.  have you got any good books you could recommend for php dev work.  it seems that I have been getting more and more of it, lately.
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39228926
Thanks for the points.   I have about 11 shelf-feet of PHP books, since I teach it for Boston University.  Of those, I most frequently recommend these.

Beginner to Intermediate:
http://www.sitepoint.com/books/phpmysql5/ (ignore the hokey name - it's a good book!)
http://www.amazon.com/PHP-MySQL-Web-Development-Edition/dp/0321833899
http://www.amazon.com/PHP-Object-Oriented-Solutions-David-Powers/dp/1430210117/

Getting a little old, but has some good "cookbook" items:
http://www.amazon.com/dp/0672328887

Intermediate to Advanced:
http://www.amazon.com/Objects-Patterns-Practice-Experts-Source/dp/143022925X/

Good online learning resources include almost anything from SitePoint, as well as Tizag, W3Schools, CodeAcademy.

Best regards, ~Ray
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Generating table dynamically is the most common issue faced by php developers.... So it seems there is a need of an article that explains the basic concept of generating tables dynamically. It just requires a basic knowledge of html and little maths…
Since pre-biblical times, humans have sought ways to keep secrets, and share the secrets selectively.  This article explores the ways PHP can be used to hide and encrypt information.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to count occurrences of each item in an array.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now