We help IT Professionals succeed at work.

cURL Help Please?

zpivat
zpivat asked
on
330 Views
Last Modified: 2012-05-05
Hi,

If I were to save a webpage onto a file, I know that the following simple code works (I have tried it):

<?php
$url = "http://www.whateversite.com";
$ch = curl_init($url);
$fp = fopen("myfile.txt", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
?>

However, if the variable $url actually holds a dynamically generated URL (that can display on a browser ONLY AFTER logging to a website), the above would not work. For example, I tried doing $url = "http://us.f621.mail.yahoo.com/ym/ShowFolder?rb=Inbox&reset=1&YY=92315&y5beta=yes&y5beta=yes&inc=25&order=down&sort=date&pos=0&view=a&head=b&box=Inbox"; (which is the URL to my Yahoo mail inbox, which can be accessed obviously only after I log in.)

and it didn't work........

I know that I have to handle cookies somehow (I've tried curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); and the whole deal) without luck. I keep getting a blank file as a result.

Anyone can help?
Comment
Watch Question

Sudaraka WijesingheWeb Application Programmer

Commented:
Although I haven't tried this myself, I don't think you can bypass a login page with CURL (at least a single HTTP request), because after the validation process the destination (in your case yahoo) will produce some HTTP redirection headers and a security token (maybe) and send to the browser and CURL will receive these not the actual landing page.

You could however phrase the HTTP data you receive and using the security token request a page again that is protected by the login.

Sorry if any of this is misguiding, I'm just thinking out loud and all this is theory (still).
Your Q is interesting, I'll try this myself when I get some time.

Author

Commented:
Hi sudaraka,

Thanks for the response.

When you said that it's not possible to bypass a login page with cURL, did you mean that in a general sense, or does it apply only to Yahoo?
If you look at http://www.wagerank.com/2007/how-to-submit-forms-with-php/ , you can see that it gives you a tutorial that teaches you, by using cURL, how to login to Technorati.com and add a certain website to the favourite blog list. Perhaps it can give you some clues?

Thanks.
Web Application Programmer
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks.

So I guess what I originally intend to do (as in my first post) is not possible then?

Author

Commented:
Oh by the way, I also used this:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
(according to PHP's documentation, CURLOPT_FOLLOWLOCATION is for enabling redirection) and it didn't seem to help either.....
Any experience with it?

Author

Commented:
Update: finally I was able to do what I want. It works only if I run the script from localhost (i.e: my own computer), whereas if I upload it to an external webhost (with cURL enabled), it doesn't work. Any ideas why?
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.