I am trying to screen scrape from a web site, my code works fine for sites which don't involve cookies. however, the web site i am trying to access requires that a browser have cookies enabled to log in.
what is the best way to go about solving this problem. i have considered:
1. implicitly setting the header of my my agent to the Cookie value required for the site.
2. somehow getting the Mechanize agent to have cookies enabled - but i am not sure how to go about doing this? i have tried giving it an empty cookie_jar object from HTTP::Cookie
but that did not seem to work.
I am fairly new to perl and would appreciate any hints and tips anyone might have.