PHP Curl Multi-exec

Hi experts!

Many thanks for your invaluable assistance.

Basically my question is this.  I upload via a curl post to an api a series of posts.  The problem is on a busy day, there are sometimes up to 1500 items to post and the page times out.  

Can I use multi-exec to upload these posts quicker?

your help is very much appreciated!
mavmanauNetwork Engineer/SysadminAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ray PaseurCommented:
What is the API?  Where is the documentation for the API?  Do you need to upload all 1500 items at the same time, or can you make multiple requests to the API in the same day?
1
mavmanauNetwork Engineer/SysadminAuthor Commented:
Hi,

Yes, I upload to this api using the same method throughout the day.  We are talking about SMS's.  The batch of 1500 uploads (Worst Case) is a batch that is run earlier in the day. It used to work fine but now that the system is getting larger and there are more sms's to upload,

I could paste the loop i use to do the upload?
0
mavmanauNetwork Engineer/SysadminAuthor Commented:
Code I normally use.


$xmlresult = '';
foreach ($obj as $sms)
{
    $user = $_POST["user"];
	$pass = $_POST["pass"];
	$action = $_POST["action"];
	$dest = (string)$sms->recipient;
//	echo "this is the dest: ".$dest."<br><br>";
    $message = (string)$sms->msg;
	$filenum = (string)$sms->file;
	$uidnum = "SMS".$filenum.$dest.$currdatetime;
//	echo "this is the uid: ".$uidnum."<br><br>";
	$opnum = (string)$sms->opers;
	$stats = (string)$sms->status;
	$campaign = (string)$sms->campaign;
	$str = '?USERNAME='.urlencode($user).'&PASSWORD='.urlencode($pass).'&ACTION='.urlencode($action).'&MESSAGE_TEXT='.urlencode($message).'&RECIPIENT='.urlencode($dest).'&ORIGINATOR='.urlencode('XXXXXXXXXX').'&REFERENCE='.urlencode($uidnum).'&CAMPAIGN='.urlencode($campaign);
      // Open connection
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url.$str);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/XXXXXX.com.au.crt");

// curl_exec($ch);
// curl_close($ch);
// echo '<br><br><br>This is the string: '.$str.'<br><br>';
// Execute post
//Re-enable this once ready to test submit.
$result = curl_exec($ch);
//echo "<br><br>";
//var_dump($result);
//echo "<br><br>";
// Check if error was returned
if(curl_errno($ch)) echo 'Curl error: ' . curl_error($ch);
// Close connection
curl_close($ch);

// Dump the result
// echo '<br><br><br>';
//var_dump($result);
$xmlresult = $xmlresult.'<smsresultsnewCEN><file>'.$filenum.'</file><result>'.$uidnum.'</result><op>'.$opnum.'</op><cell>'.$dest.'</cell><message>'.$message.'</message><status>'.$stats.'</status></smsresultsnewCEN>';
//echo "<br><br>This is the result to be imported to collect: ".$xmlresult."<br><br><br>";
}

Open in new window

0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

Ray PaseurCommented:
I understand the problem now.  We have two approaches to this.  The first is a quick fix, but of course I cannot test it because I do not have your API information.  The quick fix is to reset the timer on each call to cURL.  It might work, but I am somewhat skeptical, and you would have to do the tests to see what happened.  PHP is not well suited to long-running scripts, and 1,500 iterations of cURL will of necessity be long-running.
<?php // demo/temp_mavmanau.php
/**
 * http://www.experts-exchange.com/questions/28695565/PHP-Curl-Multi-exec.html
 */
error_reporting(E_ALL);

$xmlresult = '';
foreach ($obj as $sms)
{
    $user = $_POST["user"];
    $pass = $_POST["pass"];
    $action = $_POST["action"];
    $dest = (string)$sms->recipient;

    $message = (string)$sms->msg;
    $filenum = (string)$sms->file;
    $uidnum = "SMS".$filenum.$dest.$currdatetime;

    $opnum = (string)$sms->opers;
    $stats = (string)$sms->status;
    $campaign = (string)$sms->campaign;
    $str = '?USERNAME='.urlencode($user).'&PASSWORD='.urlencode($pass).'&ACTION='.urlencode($action).'&MESSAGE_TEXT='.urlencode($message).'&RECIPIENT='.urlencode($dest).'&ORIGINATOR='.urlencode('XXXXXXXXXX').'&REFERENCE='.urlencode($uidnum).'&CAMPAIGN='.urlencode($campaign);

    $ch = curl_init();
    curl_setopt($ch,CURLOPT_URL,$url.$str);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
    curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/XXXXXX.com.au.crt");

    /**
     * SET THE TIMER TO PREVENT PHP FROM EXCEEDING ITS LIMIT
     */
    set_time_limit(3);

    $result = curl_exec($ch);
    if(curl_errno($ch)) echo 'Curl error: ' . curl_error($ch);
    curl_close($ch);
    $xmlresult = $xmlresult.'<smsresultsnewCEN><file>'.$filenum.'</file><result>'.$uidnum.'</result><op>'.$opnum.'</op><cell>'.$dest.'</cell><message>'.$message.'</message><status>'.$stats.'</status></smsresultsnewCEN>';
}

Open in new window

1
Ray PaseurCommented:
The second approach would be to understand the API and see if batch communications make sense.  To that end, let me ask this again:
What is the API?  Where is the documentation for the API?  Do you need to upload all 1500 items at the same time, or can you make multiple requests to the API in the same day?
0
mavmanauNetwork Engineer/SysadminAuthor Commented:
What about the better solution?  At the moment I am happy to extract the data and feed it through manually (i cut the posts into lots of 300 which works for the moment), I would prefer the long term solution so I don't have to worry about it for a little while.
0
Ray PaseurCommented:
What is the API?  Where is the documentation for the API?  Do you need to upload all 1500 items at the same time, or can you make multiple requests to the API in the same day?
0
mavmanauNetwork Engineer/SysadminAuthor Commented:
Hi,

I think I might have what I need to do, just unsure on how to proceed.

basically they allow lots of 100 in a single upload. Like this:

[{“RECIPIENT”:”61435800957”, “MESSAGE_TEXT”:”Message 1”, “REFERENCE”:”unique01”},
{“RECIPIENT”:”61406040271”, “MESSAGE_TEXT”:“Message 2”, “REFERENCE”:”unique02”}]
0
Ray PaseurCommented:
What is the API?  Where is the documentation for the API?
0
mavmanauNetwork Engineer/SysadminAuthor Commented:
Hi,

This is the documentation.

http://www.smscentral.com.au/sms-api/rest-api/

There is a facility to send 100 messages in a single JSON packet.

We have a database system which can spit out data into CSV or XML, which works fine for single messages and small bursts, but every day there are a bulk group that are sent at 10:30 - yes we want to send these around the same time - they can be multiple bursts etc HOWEVER - the DB system will only spit them out in one burst.
1
Ray PaseurCommented:
OK, so it sounds like there are a few things we need to deal with here.  I'll try to show you a code sample with the general design pattern.  In this example we will collect all of the data elements (SMS messages) into an array and we will chop the array up into bite-sized units as we simulate sending the SMS messages.

Please see http://iconoun.com/demo/temp_mavmanau.php

The collection is in the $arr array variable; the number of SMS messages per API call is in the $num variable.  We use a while() loop to call the simulated sendAPI() function until the $arr array is empty.

Look carefully at the sendAPI() function signature.  See how it uses the ampersand?  

function sendAPI(&$set, $num)

This causes the $set argument to be passed by reference.  Thus we can operate on the original array inside the function.  Normally, without the ampersand, we would be operating on a copy of the array.  With the pass-by-reference notation, we can incrementally reduce the original array, sending small batches, until the array is depleted.

<?php // demo/temp_mavmanau.php

/**
 * http://www.experts-exchange.com/questions/28695565/PHP-Curl-Multi-exec.html
 *
 * Show how to cut up a large data set into smaller parts for
 * incremental processing through an API
 * http://www.smscentral.com.au/sms-api/rest-api/
 * http://php.net/manual/en/function.array-slice.php#112359
 *
 */
error_reporting(E_ALL);
echo '<pre>';


// THIS WILL SIMULATE AN ARRAY COLLECTION OF MANY SMS ELEMENTS
$arr = range('A', 'Z');

// THIS WILL SIMULATE THE NUMBER OF ELEMENTS PER REQUEST
$num = 7;

// CALL THE SENDING FUNCTION AS LONG AS WE STILL HAVE DATA IN THE ARRAY COLLECTION
while (!empty($arr))
{
    sendAPI($arr, $num);
}


// THIS FUNCTION WILL SIMULATE THE API SENDING FUNCTION
function sendAPI(&$set, $num)
{
    // GRAB THE NEXT ELEMENTS FROM OUR COLLECTION
    $subset = array_chop($set, $num);

    // USE AN ITERATOR TO PROCESS EACH ELEMENT OF THE SUBSET
    foreach ($subset as $key => $value)
    {
        echo PHP_EOL . "PROCESSING KEY: $key, WITH VALUE: $value";
    }
    echo PHP_EOL;
}


// A UTILITY FUNCTION TO REDUCE THE ARRAY COLLECTION
function array_chop(&$arr, $num, $keys=TRUE)
{
    $ret = array_slice($arr,    0, $num, $keys);
    $arr = array_slice($arr, $num, NULL, $keys);
    return $ret;
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mavmanauNetwork Engineer/SysadminAuthor Commented:
awesome thank you, I will have a play first thing in the morning!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.