Solved

php curl loop - making it more efficient

Posted on 2013-06-02
3
1,285 Views
Last Modified: 2013-06-07
Hi experts,

many thanks for taking a look at my question.  Essentially i have a page that gets xml data fed to it, which i then use  SimpleXML to loop through - doing a curl call each loop and giving me back a response that i then decode with json - add to a variable then write a file at the end.

my main issue is that i need to dramatically speed up the process.  I currently have it doing approx 3 calls per second which on large data sets, will take hours to complete.

could someone please show me how i could optimise the loop a little more?  I have already move the init_curl out of the loop so it doesn't do this multiple times any more.

your help is most appreciated.  I have attached part of code below.

date_default_timezone_set("Australia/Brisbane"); 
define("API_BASEURL", "https://rest.test.com/");
define("API_VERSION", "2012-12-01");

//echo "<pre>";
$ch = '';
// data
$tstxml = $_POST["xml_data"];
//echo '<br><br><br>This is the xml_data posted'.$tstxml.'<br><br>';
//get the rest of the variables from the form
	$username = $_POST["user"];
	$password = $_POST["pass"];
	$action = $_POST["action"];
$filename = $_POST["filename"];
$datevar = date('YmdHis');
if ($filename =='') 
{
	$filename = 'data';
}
$filename = $filename.$datevar.'.csv';
//echo $filename.'<br><br>';
//Set the URL for the SMS Submission
$url = "https://rest.test.com/wash";

//echo 'This is the XML Data from the page: '.$tstxml.'<br><br><br>';


// MAKE AN OBJECT
$obj = SimpleXML_Load_String($tstxml);

// ACTIVATE THIS TO SEE THE OBJECT
// var_dump($obj);

// USE AN ITERATOR TO ACCESS PROPERTIES OF THE OBJECT
$xmlresult = '';
$fp = fopen($filename, 'w');
$xmlresult == '';
fwrite($fp, "File Num, Num Chkd, Result,Date, time\r\n");
   $ch = curl_init();
   foreach ($obj as $numver)
{
    $user = $_POST["user"];
	$pass = $_POST["pass"];
	$action = $_POST["action"];
	$dest = (string)$numver->recipient;
    $filenum = (string)$numver->file;
	$postfields = array("destination" => $dest);
	//$str = '?user='.urlencode($user).'&pass='.urlencode($pass).'&action='.urlencode($action).'&destination='.urlencode($dest);
      // Open connection
       // Set up the cURL object
	
curl_setopt($ch, CURLOPT_URL, API_BASEURL . API_VERSION . "/" . $action);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, $user . ":" . $pass);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

// Execute the request
$result = curl_exec($ch);
//var_dump($result);
// Get status info on the cURL request
$response_info = curl_getinfo($ch);

//var_dump($response_info);
if(($response_info["http_code"] >= 200) AND ($response_info["http_code"] <= 299)){      // Success

        // Decode the json response
        $json = json_decode($result);
//echo $json->wash[0]->id;
$jsvar = $json->wash[0]->id;
$datevar = date('d/m/Y');
$datetimenow = date('h:i:s A');
//fwrite($fp, $filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n");
//echo '<br><Br>';
//echo 'JM ws ere<br><br>';
        // json_decode will return null on failure



}
 

$xmlresult = $xmlresult.$filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n";

}
 fwrite($fp, $xmlresult);
 fclose($fp); 
curl_close($ch);

Open in new window

0
Comment
Question by:mavmanau
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 110

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 39215744
After clearing up the extraneous stuff, this appears to be the script you're using.  But it looks like you've changed the URLs so nobody but you can test it.  That's too bad, because we really kind of need the SSCCE to begin to discern why this is so slow.

One thing you might want to consider is whether the elapsed time is coming from the cURL call or whether it's coming from your script.  If it's the external web service, your only recourse will be to try to get the publishers to give you a more efficient API.

You might also try omitting the call to curl_getinfo() and just testing the $result variable for FALSE.  This would reduce the program path by a few instructions.  There are other inefficiencies, like doing this over and over inside a loop: $user = $_POST["user"]; but that sort of thing is not likely to cause much delay.

<?php // RAY_temp_mavmanau.php
error_reporting(E_ALL);

date_default_timezone_set("Australia/Brisbane"); 
define("API_BASEURL", "https://rest.test.com/");
define("API_VERSION", "2012-12-01");

$ch = '';

$tstxml   = $_POST["xml_data"];
$username = $_POST["user"];
$password = $_POST["pass"];
$action   = $_POST["action"];
$filename = $_POST["filename"];
$datevar  = date('YmdHis');
if ($filename =='') 
{
	$filename = 'data';
}
$filename = $filename.$datevar.'.csv';

$url = "https://rest.test.com/wash";

// MAKE AN OBJECT
$obj = SimpleXML_Load_String($tstxml);

// ACTIVATE THIS TO SEE THE OBJECT
// var_dump($obj);

// USE AN ITERATOR TO ACCESS PROPERTIES OF THE OBJECT
$xmlresult = '';
$fp = fopen($filename, 'w');
fwrite($fp, "File Num, Num Chkd, Result,Date, time\r\n");
$ch = curl_init();
foreach ($obj as $numver)
{
    $user    = $_POST["user"];
    $pass    = $_POST["pass"];
    $action  = $_POST["action"];
    $dest    = (string)$numver->recipient;
    $filenum = (string)$numver->file;
    $postfields = array("destination" => $dest);

    curl_setopt($ch, CURLOPT_URL, API_BASEURL . API_VERSION . "/" . $action);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_USERPWD, $user . ":" . $pass);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

    $result = curl_exec($ch);
    $response_info = curl_getinfo($ch);
    if(($response_info["http_code"] >= 200) AND ($response_info["http_code"] <= 299)) 
    {

        $json = json_decode($result);
        $jsvar = $json->wash[0]->id;
        $datevar = date('d/m/Y');
        $datetimenow = date('h:i:s A');
    }

    $xmlresult = $xmlresult.$filenum.",".$dest.",".$jsvar.",".$datevar.",".$datetimenow."\r\n";
}
fwrite($fp, $xmlresult);
fclose($fp); 
curl_close($ch);

Open in new window

Please see the snippet below to learn how to attach a script timer to all or part of a PHP script.  In addition to the class itself, there are some usage examples below line 160.

<?php // RAY_class_Stopwatch.php
error_reporting(E_ALL);


// DEMONSTRATE A SCRIPT TIMER FOR ALL OR PART OF A SCRIPT PHP 5+
// MAN PAGE http://php.net/manual/en/function.microtime.php


class StopWatch
{
    protected $a; // START TIME
    protected $s; // STATUS - IF RUNNING
    protected $z; // STOP TIME

    public function __construct()
    {
        $this->a = array();
        $this->s = array();
        $this->z = array();
    }

    // A METHOD TO PROVIDE A FINAL READOUT, IF NEEDED
    public function __destruct()
    {
        $ret = $this->readout();
        if (!$ret) return FALSE;
        echo
          __CLASS__
        . '::'
        . __FUNCTION__
        . '() '
        ;
        echo "<b>$ret</b>";
        echo PHP_EOL;
    }

    // A METHOD TO REMOVE A TIMER
    public function reset($name='TIMER')
    {
        // RESET ALL TIMERS
        if ($name == 'TIMER')
        {
            $this->__construct();
        }
        else
        {
            unset($this->a[$name]);
            unset($this->s[$name]);
            unset($this->z[$name]);
        }
    }

    // A METHOD TO CAPTURE THE START TIME
    public function start($name='TIMER')
    {
        $this->a[$name] = microtime(TRUE);
        $this->z[$name] = $this->a[$name];
        $this->s[$name] = 'RUNNING';
    }

    // A METHOD TO CAPTURE THE END TIME
    public function stop($name='TIMER')
    {
        $ret = NULL;

        // STOP ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                // IF THIS TIMER IS STILL RUNNING, STOP IT
                if ($this->s[$name])
                {
                    $this->s[$name] = FALSE;
                    $this->z[$name] = microtime(TRUE);
                }
            }
        }

        // STOP ONLY ONE OF THE TIMERS
        else
        {
            if ($this->s[$name])
            {
                $this->s[$name] = FALSE;
                $this->z[$name] = microtime(TRUE);
            }
            else
            {
                $ret .= "ERROR: CALL TO STOP() METHOD: '$name' IS NOT RUNNING";
            }
        }

        // RETURN AN ERROR MESSAGE, IF ANY
        return $ret;
    }

    // A METHOD TO READ OUT THE TIMER(S)
    public function readout($name='TIMER', $dec=3, $m=1000, $t = 'ms', $eol=PHP_EOL)
    {
        $str = NULL;

        // GET READOUTS FOR ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                $str .= $name;

                // IF THIS TIMER IS STILL RUNNING UPDATE THE END TIME
                if ($this->s[$name])
                {
                    $this->z[$name] = microtime(TRUE);
                    $str .= " RUNNING ";
                }
                else
                {
                    $str .= " STOPPED ";
                }

                // RETURN A DISPLAY STRING
                $lapse_time = $this->z[$name] - $start_time;
                $lapse_msec = $lapse_time * $m;
                $lapse_echo = number_format($lapse_msec, $dec);
                $str .= " $lapse_echo $t";
                $str .= $eol;
            }
            return $str;
        }

        // GET A READOUT FOR ONLY ONE TIMER
        else
        {
            $str .= $name;

            // IF THIS TIME IS STILL RUNNING, UPDATE THE END TIME
            if ($this->s[$name])
            {
                $this->z[$name] = microtime(TRUE);
                $str .= " RUNNING ";
            }
            else
            {
                $str .= " STOPPED ";
            }

            // RETURN A DISPLAY STRING
            $lapse_time = $this->z[$name] - $this->a[$name];
            $lapse_msec = $lapse_time * $m;
            $lapse_echo = number_format($lapse_msec, $dec);
            $str .= " $lapse_echo $t";
            $str .= $eol;
            return $str;
        }
    }
}



// DEMONSTRATE THE USE -- INSTANTIATE THE STOPWATCH OBJECT
$sw  = new Stopwatch;

// SET A STOPWATCH NAME THAT REFLECTS THE PARTS OF THE SCRIPT WE WANT TO TIME
$g_timer = 'GOOGLE TIMER';

// START A TIMER TO GET ELAPSED TIME FOR A CALL TO GOOGLE
$sw->start($g_timer);

// PERFORM SOME ACTIVITY THAT YOU WANT TO TIME (READS GOOGLE WEB PAGE)
$page = 'https://www.google.com';
$html = file_get_contents($page);

// GET A READOUT OF THE TIMER WHILE IT IS STILL RUNNING
echo nl2br($sw->readout($g_timer));
echo "<br/>" . PHP_EOL;

// PERFORM SOME OTHER ACTIVITY (READS GOOGLE WEB PAGE AGAIN)
$page = 'https://www.google.com';
$html = file_get_contents($page);

// STOP THE TIMER AND GET A READOUT WITH SHORT DECIMALS
$x = $sw->stop($g_timer);
echo nl2br($sw->readout($g_timer, 1));
echo "<br/>" . PHP_EOL;




// START A SECOND TIMER
$y_timer = 'YAHOO TIMER';
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT (ONE IS STOPPED AND ONE IS STILL RUNNING)
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

// STOP ALL OF THE STOPWATCHES
$sw->stop();

// REPORT THE STOPWATCHES CONTENT AGAIN
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// TRY TO STOP A TIMER THAT IS NOT RUNNING
$x = $sw->stop($g_timer);
var_dump($x);
echo "<br/>" . PHP_EOL;
echo "<br/>" . PHP_EOL;



// START THIS TIMER OVER AGAIN
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://weather.yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ONE OF THE STOPWATCHES
$sw->reset($g_timer);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ALL OF THE STOPWATCHES
$sw->reset();
echo "ALL STOPWATCHES HAVE BEEN REMOVED";

// REPORT THE STOPWATCHES CONTENT (SHOWS NOTHING)
echo nl2br($sw->readout());

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

// SHOW THE DESTRUCTOR IN ACTION
$sw->start('Foo');
echo "<pre>";
unset($sw);
echo "</pre>";

Open in new window

0
 

Author Closing Comment

by:mavmanau
ID: 39228864
perfect, thank you very much.  have you got any good books you could recommend for php dev work.  it seems that I have been getting more and more of it, lately.
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39228926
Thanks for the points.   I have about 11 shelf-feet of PHP books, since I teach it for Boston University.  Of those, I most frequently recommend these.

Beginner to Intermediate:
http://www.sitepoint.com/books/phpmysql5/ (ignore the hokey name - it's a good book!)
http://www.amazon.com/PHP-MySQL-Web-Development-Edition/dp/0321833899
http://www.amazon.com/PHP-Object-Oriented-Solutions-David-Powers/dp/1430210117/

Getting a little old, but has some good "cookbook" items:
http://www.amazon.com/dp/0672328887

Intermediate to Advanced:
http://www.amazon.com/Objects-Patterns-Practice-Experts-Source/dp/143022925X/

Good online learning resources include almost anything from SitePoint, as well as Tizag, W3Schools, CodeAcademy.

Best regards, ~Ray
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developers of all skill levels should learn to use current best practices when developing websites. However many developers, new and old, fall into the trap of using deprecated features because this is what so many tutorials and books tell them to u…
These days socially coordinated efforts have turned into a critical requirement for enterprises.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

628 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question