Using cookies with WWW::Mechanize...

I am trying to screen scrape from a web site, my code works fine for sites which don't involve cookies.  however, the web site i am trying to access requires that a browser have cookies enabled to log in.

what is the best way to go about solving this problem.  i have considered:

1. implicitly setting the header of my my agent to the Cookie value required for the site.

2. somehow getting the Mechanize agent to have cookies enabled - but i am not sure how to go about doing this? i have tried giving it an empty cookie_jar object from HTTP::Cookie but that did not seem to work.

I am fairly new to perl and would appreciate any hints and tips anyone might have.

many thanks
james

jamesbuckneyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jmcgOwnerCommented:
That's curious. The code for WWW::Mechanize sets up the UserAgent with a cookie jar by default, so it should behave as if cookies are enabled.

Have you taken a look at the WWW::Mechanize::Examples files?

http://search.cpan.org/~petdance/WWW-Mechanize-0.70/lib/WWW/Mechanize/Examples.pod

There they show some examples of getting past login screens and filling out forms automatically.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jmcgOwnerCommented:
Welcome to Experts-Exchange, James,

I know that there was a problem with email notifications going out around 4 December (I certainly missed quite a few), so perhaps you are thinking we ignored your first question because Experts Exchange never seemed to contact you again. Maybe you'll get a notification this time and revisit your question.
0
jmcgOwnerCommented:
Nothing has happened on this question in more than 7 weeks. It's time for cleanup!

My recommendation, which I will post in the Cleanup topic area, is to
accept answer by jmcg [grade B] (it's correct but whether it solves the problem is hard to know, asker abandoned question).

Please leave any comments here within the next seven days.

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

jmcg
EE Cleanup Volunteer
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.