I am looking for a way to download the contents of a password protected webpage. I have a legitimate username/password, but I want to setup a script to automatically download the content and add it to a database for my later review. I am going to use an RSS feed to get the links directly to the page containing the relevant content. For example, I would like to download the headlines from the Wall Street Journal via the RSS feed they provide and then run the contents of each article through a filter.
After I get the links I am stuck and I am looking for someone to give me some ideas about how I might be able to log my server into the target site and download the content. I have done something similar in the past using getURL/fopen in PHP, but for sites without a password. I would prefer to use PHP with any added libraries, but I would be willing to try out any solution.
I posted this question in the PHP subtopic, please answer it at the following link: