asked on

cURL Help Please?

Hi,

If I were to save a webpage onto a file, I know that the following simple code works (I have tried it):

<?php
$url = "http://www.whateversite.com";
$ch = curl_init($url);
$fp = fopen("myfile.txt", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
?>

However, if the variable $url actually holds a dynamically generated URL (that can display on a browser ONLY AFTER logging to a website), the above would not work. For example, I tried doing $url = "http://us.f621.mail.yahoo.com/ym/ShowFolder?rb=Inbox&reset=1&YY=92315&y5beta=yes&y5beta=yes&inc=25&order=down&sort=date&pos=0&view=a&head=b&box=Inbox"; (which is the URL to my Yahoo mail inbox, which can be accessed obviously only after I log in.)

and it didn't work........

I know that I have to handle cookies somehow (I've tried curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); and the whole deal) without luck. I keep getting a blank file as a result.

Anyone can help?

Sudaraka Wijesinghe

Although I haven't tried this myself, I don't think you can bypass a login page with CURL (at least a single HTTP request), because after the validation process the destination (in your case yahoo) will produce some HTTP redirection headers and a security token (maybe) and send to the browser and CURL will receive these not the actual landing page.

You could however phrase the HTTP data you receive and using the security token request a page again that is protected by the login.

Sorry if any of this is misguiding, I'm just thinking out loud and all this is theory (still).
Your Q is interesting, I'll try this myself when I get some time.

zpivat

ASKER

Hi sudaraka,

Thanks for the response.

When you said that it's not possible to bypass a login page with cURL, did you mean that in a general sense, or does it apply only to Yahoo?
If you look at http://www.wagerank.com/2007/how-to-submit-forms-with-php/ , you can see that it gives you a tutorial that teaches you, by using cURL, how to login to Technorati.com and add a certain website to the favourite blog list. Perhaps it can give you some clues?

Thanks.

ASKER CERTIFIED SOLUTION

Sudaraka Wijesinghe

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

zpivat

ASKER

Thanks.

So I guess what I originally intend to do (as in my first post) is not possible then?

zpivat

ASKER

Oh by the way, I also used this:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
(according to PHP's documentation, CURLOPT_FOLLOWLOCATION is for enabling redirection) and it didn't seem to help either.....
Any experience with it?

zpivat

ASKER

Update: finally I was able to do what I want. It works only if I run the script from localhost (i.e: my own computer), whereas if I upload it to an external webhost (with cURL enabled), it doesn't work. Any ideas why?