PHP screen scraper running slow - how can i speed it up?

Hi,

I am currently running a web scraper to pull race results but its taking so long that it can time out.

Is there any way I can speed up the process by making the code run faster - cutting out some of the searching ect?

I have seen some posts about multi curl??? would that help???
 

    $url  = "http://form.timeform.betfair.com/daypage?date=20150516"; //WILL NEED TO PULL TOMORROWS DATE AS DD-MM-YYY
    $html = curl($url);
    $dom  = new DOMDocument();
    @$dom->loadHTML($html);
    $dom->preserveWhiteSpace = false;
    $xpath                   = new DOMXPath($dom);
    //pull the individual cards for the day
    //li class="rac-cardsclass="ix ixc"
    $getdropdown             = '//div[contains(@data-location, "RACING_COUNTRY_GB_IE")]//div[contains(@class, "course")]';
    $getdropdown2            = $xpath->query($getdropdown);
    //loop through each individual card
    foreach($getdropdown2 as $dropresults) {
        //loop through and get all the a tags
        $arr = $dropresults->getElementsByTagName("a");
        foreach($arr as $item) {
            //only grab the links which point to the results page
            if(strpos($item->getAttribute('href'), 'raceresult') !== false) {
                //grab the code
                $code = explode("=", $item->getAttribute('href'));
                $code = end($code);
                $url  = "http://form.timeform.betfair.com/raceresult?raceId=" . $code; //WILL NEED TO PULL TOMORROWS DATE AS DD-MM-YYY
                $html = curl($url);
                $dom  = new DOMDocument();
                @$dom->loadHTML($html);
                $dom->preserveWhiteSpace = false;
                $xpath                   = new DOMXPath($dom);
                $spanTexts               = array();
                //get the place name
                $getplacename            = '//span[contains(@class, "locality")]';
                $getplacename2           = $xpath->query($getplacename);
                //loop through each individual card
                foreach($getplacename2 as $getplacename22) {
                    $venue = $getplacename22->textContent;
                } //$getplacename2 as $getplacename22
                $gettime  = '//abbr [contains(@class, "dtstart")]';
                //get the Date and the Time
                $gettime2 = $xpath->query($gettime);
    			
                foreach($gettime2 as $gettime22) {
                    $Dateandtime = date(trim($gettime22->getAttribute('title')), strtotime('+5 hours'));
                } //$gettime2 as $gettime22
                //pull the data for the race e.g going money ect
                $getdropdown22  = '//div[contains(@class, "content")]/p';
                $getdropdown222 = $xpath->query($getdropdown22);
    			$getracename            = '//div[contains(@class, "content")]//p[contains(@class, "race-description")]';
                $getracename2           = $xpath->query($getracename);
    			$racename = "";
    			 foreach($getracename2 as $getracename) {
    			 $racename = trim(str_replace("'", "", $getracename->textContent));
    			}
    			            foreach($getdropdown222 as $dropresults2) {
    			
                         foreach($dropresults2->childNodes as $node) {
                        if(is_object($node) && $node->nodeType === XML_ELEMENT_NODE && strtolower($node->tagName) === 'span') {
                            $spanTexts[] = (string) $node->textContent;
                         
                        } 
    					} 
                    if(count($spanTexts) < 6)
                        continue;
                    list($going, $distance, $age, $prizemoney, $runners, $racetype) = $spanTexts;
                    $going      = str_replace(array(
                        'Â', 
                        'Going:',
                        '|'
                    ), '', $going);
                    $distance   = miletofurlong($distance = trim(GetBetween($distance, ':', 'Â')));
                    $age        = trim(GetBetween($age, ':', 'Â'));
                    $prizemoney = trim(GetBetween($prizemoney, '£', 'Â'));
                    $runners    = trim(GetBetween($runners, ':', 'Â'));
                    $racetype   = trim(GetBetween($racetype, ':', 'Â'));
                } 
                //pull the individual horse data
                $getdropdown  = '//div[contains(@class, "table-container")]//tbody//tr';
                $getdropdown2 = $xpath->query($getdropdown);
                //loop through each individual card
                foreach($getdropdown2 as $dropresults) {
                    $position   = $dropresults->childNodes->item(0)->childNodes->item(1)->textContent;
                    $draw       = str_replace(array('(',')'), '', $dropresults->childNodes->item(0)->childNodes->item(3)->textContent);
                    $losingdist = str_replace('Â', '', trim($dropresults->childNodes->item(2)->textContent));
                    if(strpos($losingdist, '¾') !== false) {
                        $losingdist = str_replace('¾', '.75', $losingdist);
                    } 
                    if(strpos($losingdist, '½') !== false) {
                        $losingdist = str_replace('½', '.5', $losingdist);
                    } 
                    if(strpos($losingdist, '¼') !== false) {
                        $losingdist = str_replace('¼', '.25', $losingdist);
                    } 
                    $losingdist;
                    $horse    = trim(preg_replace("/\([^\)]+\)/","",str_replace("'","",trim($dropresults->childNodes->item(4)->textContent))));
                    $horseage = trim($dropresults->childNodes->item(6)->textContent);
                    $weight   = trim($dropresults->childNodes->item(8)->childNodes->item(1)->textContent);
                    $or       = str_replace(array('(',')'), '', trim($dropresults->childNodes->item(8)->childNodes->item(3)->textContent));
                    str_replace('-', '', $eq = trim($dropresults->childNodes->item(10)->textContent));
                    $jockey              = trim($dropresults->childNodes->item(12)->childNodes->item(1)->textContent);
                    $trainer             = trim($dropresults->childNodes->item(12)->childNodes->item(4)->textContent);
                    $highandlowinrunning = trim($dropresults->childNodes->item(14)->childNodes->item(1)->textContent);
                    $highandlow          = explode("/", $highandlowinrunning);
                    str_replace('-', '', $lowodds = trim($highandlow['1']));
                    str_replace('-', '', $highodds = trim($highandlow['0']));
                    $bfsp      = trim($dropresults->childNodes->item(16)->childNodes->item(1)->textContent);
                    $isp       = trim(str_replace('/', '', $dropresults->childNodes->item(16)->childNodes->item(3)->textContent));
                    $placeodds = trim($dropresults->childNodes->item(18)->textContent);
    				
     $venue = mysqli_real_escape_string($db,	$venue);
     $Dateandtime = mysqli_real_escape_string($db,$Dateandtime);
     $going = mysqli_real_escape_string($db,	$going);
     $distance = mysqli_real_escape_string($db,$distance);
     $age = mysqli_real_escape_string($db,$age);
     $prizemoney = mysqli_real_escape_string($db,$prizemoney);
     $runners = mysqli_real_escape_string($db,$runners );
     $racetype = mysqli_real_escape_string($db,$racetype);
     $position  = mysqli_real_escape_string($db,$position );
     $draw  = mysqli_real_escape_string($db,$draw);
     $losingdist = mysqli_real_escape_string($db,$losingdist);
     $horse = mysqli_real_escape_string($db,$horse );
     $age = mysqli_real_escape_string($db,$age);
     $weight = mysqli_real_escape_string($db,$weight);
     $or = mysqli_real_escape_string($db,$or );
     $eq = mysqli_real_escape_string($db,$eq );
     $jockey = mysqli_real_escape_string($db,$jockey);
     $trainer = mysqli_real_escape_string($db,$trainer);
     $lowodds = mysqli_real_escape_string($db,$lowodds);
     $highodds = mysqli_real_escape_string($db,$highodds);
     $bfsp = mysqli_real_escape_string($db,$bfsp);
     $isp = mysqli_real_escape_string($db,$isp);
     $placeodds = mysqli_real_escape_string($db,$placeodds);
     $racename = mysqli_real_escape_string($db,$racename);

Open in new window


Any advice would be great
runnerjp2005Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ray PaseurCommented:
Well, I looked at the code, and the "slow" part is probably due to the cURL calls, but it would be good to test that and know for certain.  This little script will allow you to attach a timer to any part of your PHP code.  You can find out exactly where the script is slow.

If this were my app, I would go about it by making each of the cURL calls and storing the HTML documents and the URLs in a database table.  You can store these as string data.  Once you've gotten all of the information you need from the "daypage" then move on to making cURL calls to the other pages.  You could start asynchronous scripts for each of these other pages, so no single point of failure would bring the application down.  My instinct would be to avoid the DOM classes, and parse the HTML strings directly.  But I'm not really sure how much overhead the DOM manipulation adds.

<?php // demo/class_Stopwatch.php

/**
 * A Class that allows us to see the elapsed time when running a script
 */
error_reporting(E_ALL);


// CREATE AN INSTANCE OF THE STOPWATCH
$t = new StopWatch;


// START THE TIMER
$t->start();

// DO SOME ACTIVITY THAT YOU WANT TO TIME
$x = file_get_contents('https://google.com/');

// GET A READOUT FROM THE TIMER
echo $t->readout();

// RESET THE TIMER
$t->reset();


// A SCRIPT TIMER FOR ALL OR PART OF A SCRIPT PHP 5+
// MAN PAGE http://php.net/manual/en/function.microtime.php
class StopWatch
{
    protected $a; // START TIME
    protected $s; // STATUS - IF RUNNING
    protected $z; // STOP TIME

    public function __construct()
    {
        $this->a = array();
        $this->s = array();
        $this->z = array();
    }

    // A METHOD TO PROVIDE A FINAL READOUT, IF NEEDED
    public function __destruct()
    {
        $ret = $this->readout();
        if (!$ret) return FALSE;
        echo
          __CLASS__
        . '::'
        . __FUNCTION__
        . '() '
        ;
        echo "<b>$ret</b>";
        echo PHP_EOL;
    }

    // A METHOD TO REMOVE A TIMER
    public function reset($name='TIMER')
    {
        // RESET ALL TIMERS
        if ($name == 'TIMER')
        {
            $this->__construct();
        }
        else
        {
            unset($this->a[$name]);
            unset($this->s[$name]);
            unset($this->z[$name]);
        }
    }

    // A METHOD TO CAPTURE THE START TIME
    public function start($name='TIMER')
    {
        $this->a[$name] = microtime(TRUE);
        $this->z[$name] = $this->a[$name];
        $this->s[$name] = 'RUNNING';
    }

    // A METHOD TO CAPTURE THE END TIME
    public function stop($name='TIMER')
    {
        $ret = NULL;

        // STOP ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                // IF THIS TIMER IS STILL RUNNING, STOP IT
                if ($this->s[$name])
                {
                    $this->s[$name] = FALSE;
                    $this->z[$name] = microtime(TRUE);
                }
            }
        }

        // STOP ONLY ONE OF THE TIMERS
        else
        {
            if ($this->s[$name])
            {
                $this->s[$name] = FALSE;
                $this->z[$name] = microtime(TRUE);
            }
            else
            {
                $ret .= "ERROR: CALL TO STOP() METHOD: '$name' IS NOT RUNNING";
            }
        }

        // RETURN AN ERROR MESSAGE, IF ANY
        return $ret;
    }

    // A METHOD TO READ OUT THE TIMER(S)
    public function readout($name='TIMER', $dec=3, $m=1000, $t = 'ms', $eol=PHP_EOL)
    {
        $str = NULL;

        // GET READOUTS FOR ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                $str .= $name;

                // IF THIS TIMER IS STILL RUNNING UPDATE THE END TIME
                if ($this->s[$name])
                {
                    $this->z[$name] = microtime(TRUE);
                    $str .= " RUNNING ";
                }
                else
                {
                    $str .= " STOPPED ";
                }

                // RETURN A DISPLAY STRING
                $lapse_time = $this->z[$name] - $start_time;
                $lapse_msec = $lapse_time * $m;
                $lapse_echo = number_format($lapse_msec, $dec);
                $str .= " $lapse_echo $t";
                $str .= $eol;
            }
            return $str;
        }

        // GET A READOUT FOR ONLY ONE TIMER
        else
        {
            $str .= $name;

            // IF THIS TIME IS STILL RUNNING, UPDATE THE END TIME
            if ($this->s[$name])
            {
                $this->z[$name] = microtime(TRUE);
                $str .= " RUNNING ";
            }
            else
            {
                $str .= " STOPPED ";
            }

            // RETURN A DISPLAY STRING
            $lapse_time = $this->z[$name] - $this->a[$name];
            $lapse_msec = $lapse_time * $m;
            $lapse_echo = number_format($lapse_msec, $dec);
            $str .= " $lapse_echo $t";
            $str .= $eol;
            return $str;
        }
    }
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.