asked on

How do I add For Each to this PHP/cURL and run through pagination

This script actually works except the pages are not changing, the script just keeps posting the same page over and over. ( page=$i )

for ($i = 1; $i <= 1000; $i++) {
     
    /* RUN CODE AND SCRAPE STUFF IN HERE */
 $url2 = 'http://www.website.com/inventory/catalog.php?classID=&dept=&type=&keywords=&brandID=&price1=&price2=&age=&orderBy=BA&page=$i';
    

	$ch = curl_init(); // initialize curl handle
	curl_setopt($ch, CURLOPT_HEADER, 0);
	curl_setopt($ch, CURLOPT_VERBOSE, 1);
	curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
	curl_setopt($ch, CURLOPT_AUTOREFERER, false);
	curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,7);
	curl_setopt($ch, CURLOPT_REFERER, 'Referer: http://www.website.com/inventory/search.php');
	curl_setopt($ch, CURLOPT_URL,$url2); // set url to post to
	curl_setopt($ch, CURLOPT_FAILONERROR, 0);
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);// allow redirects
	
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
	curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
	curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

	curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); // return into a variable
	curl_setopt($ch, CURLOPT_TIMEOUT, 50); // times out after 50s

	
	
	$buffer2 = curl_exec($ch); // run the whole process
	echo $buffer2;
	if(curl_exec($ch) === false)
	{
	    echo 'Curl error: ' . curl_error($ch);
	    exit;
	}
}

Open in new window

ASKER CERTIFIED SOLUTION

Dave Baldwin

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

lawrence_dev

ASKER

If I run the same URL and change to page=2 or page=3, etc, I get different pages every time which is what I want. The URL I have shown is a search of all of the products.

All I am looking for is the page number to change every time a page completes.

.

Dave Baldwin

That's confusing because that is the only thing that is changing the code you posted above.

lawrence_dev

ASKER

I got it working. Thanks for your help! page='.$i; instead of page=$i.

$url2 = 'http://www.website.com/inventory/catalog.php?classID=&dept=&type=&keywords=&brandID=&price1=&price2=&age=&orderBy=BA&page='.$i;

Dave Baldwin

Oh, good!!