?
Solved

Please double-check my cURL script- why isn't this working?

Posted on 2008-10-06
2
Medium Priority
?
380 Views
Last Modified: 2012-05-05
Please see my function below:

I am attempting to connect to a page like this:

http://yellowpages.com.au//onlineSolution_moreInfo.do;jsessionid=1FB1ABFBFD3CD9BB5BF919A3AADDEEEC.12?iblId=3543593&iblName=Artizan+Cabinets&listingPosition=2&originatingPage=ListingPage&authToken=11cd37700be|d509da26cd3e1412d3070a9514558873&pageNumber=3&st=cs&z=100004

Page opens fine in any browser, but when I use my script, the same page comes up as "The page you requested has been static for too long and has run out of time."

I have already tried a few USERAGENTs, IE6, FF, and even Googlebot. I get the same with all of them.

$myurl = the page where I grab each entry url (used to spoof the referrer)

I have even enabled cookies.

Can you think of any reason this is happening? Any help you can offer is appreciated.


function file_get_curl($url) {
	global $myurl, $cookiefile;
	$curl_handle=curl_init();
	curl_setopt($curl_handle,CURLOPT_URL,$url);
	curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,60);
	curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1);
	curl_setopt($curl_handle,CURLOPT_AUTOREFERER, false);
    curl_setopt($curl_handle,CURLOPT_REFERER, $myurl);	
	curl_setopt($curl_handle,CURLOPT_COOKIEFILE, $cookiefile);
	curl_setopt($curl_handle,CURLOPT_COOKIEJAR, $cookiefile);
	curl_setopt($curl_handle,CURLOPT_USERAGENT, "Googlebot/2.1 (+http://www.googlebot.com/bot.html)");
	$data = curl_exec($curl_handle);
	curl_close($curl_handle);
	return $data;
}

Open in new window

0
Comment
Question by:jpschreibman
  • 2
2 Comments
 
LVL 10

Accepted Solution

by:
kukno earned 2000 total points
ID: 22653663
>Can you think of any reason this is happening? Any help you can offer is appreciated.

it's not a curl problem. The session in the server might have timed out.

There are two session "id's" in your URL.

         jsessionid=1FB1ABFBFD3CD9BB5BF919A3AADDEEEC
         authToken=11cd37700be|d509da26cd3e1412d3070a9514558873

Without any further information about the internals of the page, it's hard to decide what's going wrong. However, I suspect a session timeout in the web application.

What are you trying to do? There are better web automation tools than curl ;-)

Regards
Kurt
0
 
LVL 10

Expert Comment

by:kukno
ID: 22653692
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When it comes to security, close monitoring is a must. According to WhiteHat Security annual report, a substantial number of all web applications are vulnerable always. Monitis offers a new product - fully-featured Website security monitoring and pr…
There are times when I have encountered the need to decompress a response from a PHP request. This is how it's done, but you must have control of the request and you can set the Accept-Encoding header.
This video teaches users how to migrate an existing Wordpress website to a new domain.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Suggested Courses
Course of the Month13 days, 20 hours left to enroll

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question