500 points! Download Content from Webpage with Password/Cookies

Posted on 2006-05-10
Medium Priority
Last Modified: 2010-04-06
Hi Experts,

I am looking for a way to download the contents of a password protected webpage.  I have a legitimate username/password, but I want to setup a script to automatically download the content and add it to a database for my later review. I am going to use an RSS feed to get the links directly to the page containing the relevant content. For example, I would like to download the headlines from the Wall Street Journal via the RSS feed they provide and then run the contents of each article through a filter.

After I get the links I am stuck and I am looking for someone to give me some ideas about how I might be able to log my server into the target site and download the content. I have done something similar in the past using getURL/fopen in PHP, but for sites without a password. I would prefer to use PHP with any added libraries, but I would be willing to try out any solution.

-- Matt

I posted this question in the PHP subtopic, please answer it at the following link:
Question by:aaron_karp
1 Comment
LVL 41

Accepted Solution

HonorGod earned 60 total points
ID: 16691053
Answer upated

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Most of the sites are being standardized with W3C Web Standards. W3C provides lot of web standard services to the web. They have the web specification, process and documentation for all the web standards. You can apply HTML, CSS and Accessibility st…
JavaScript has plenty of pieces of code people often just copy/paste from somewhere but never quite fully understand. Self-Executing functions are just one good example that I'll try to demystify here.
The viewer will learn how to count occurrences of each item in an array.
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question