I asked this question in the Web Languages section previosuly, but didn't get any replies.
I decided in any case that I would like to go with a Java implementation if possible, since I am familiar with this language. Does anyone know of any existing libraries or classes that would make something like this easier? In particular classes that allow establishment of an http connection, storing of cookies, etc. I was thinking about using httpUnit. Is this a good choice?
I need to automate browsing of a particular site. That is, I must be able to programatically download the pages associated with the site for parsing and analysis. Additionally, I need to be able to fill out and submit forms in an automated way, as well as support cookies (the site might require the information in the cookie in order to provide context for certain pages).
Thanks in advance for your suggestions.