Solved

Please double-check my cURL script- why isn't this working?

Posted on 2008-10-06
2
350 Views
Last Modified: 2012-05-05
Please see my function below:

I am attempting to connect to a page like this:

http://yellowpages.com.au//onlineSolution_moreInfo.do;jsessionid=1FB1ABFBFD3CD9BB5BF919A3AADDEEEC.12?iblId=3543593&iblName=Artizan+Cabinets&listingPosition=2&originatingPage=ListingPage&authToken=11cd37700be|d509da26cd3e1412d3070a9514558873&pageNumber=3&st=cs&z=100004

Page opens fine in any browser, but when I use my script, the same page comes up as "The page you requested has been static for too long and has run out of time."

I have already tried a few USERAGENTs, IE6, FF, and even Googlebot. I get the same with all of them.

$myurl = the page where I grab each entry url (used to spoof the referrer)

I have even enabled cookies.

Can you think of any reason this is happening? Any help you can offer is appreciated.


function file_get_curl($url) {
	global $myurl, $cookiefile;
	$curl_handle=curl_init();
	curl_setopt($curl_handle,CURLOPT_URL,$url);
	curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,60);
	curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1);
	curl_setopt($curl_handle,CURLOPT_AUTOREFERER, false);
    curl_setopt($curl_handle,CURLOPT_REFERER, $myurl);	
	curl_setopt($curl_handle,CURLOPT_COOKIEFILE, $cookiefile);
	curl_setopt($curl_handle,CURLOPT_COOKIEJAR, $cookiefile);
	curl_setopt($curl_handle,CURLOPT_USERAGENT, "Googlebot/2.1 (+http://www.googlebot.com/bot.html)");
	$data = curl_exec($curl_handle);
	curl_close($curl_handle);
	return $data;
}

Open in new window

0
Comment
Question by:jpschreibman
  • 2
2 Comments
 
LVL 10

Accepted Solution

by:
kukno earned 500 total points
ID: 22653663
>Can you think of any reason this is happening? Any help you can offer is appreciated.

it's not a curl problem. The session in the server might have timed out.

There are two session "id's" in your URL.

         jsessionid=1FB1ABFBFD3CD9BB5BF919A3AADDEEEC
         authToken=11cd37700be|d509da26cd3e1412d3070a9514558873

Without any further information about the internals of the page, it's hard to decide what's going wrong. However, I suspect a session timeout in the web application.

What are you trying to do? There are better web automation tools than curl ;-)

Regards
Kurt
0
 
LVL 10

Expert Comment

by:kukno
ID: 22653692
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Color can increase conversions, create feelings of warmth or even incite people to get behind a cause. If you want your website to really impact site visitors, then it is vital to consider the impact color has on them.
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now