500 points! Download Content from Webpage with Password/Cookies

Posted on 2006-05-10
Last Modified: 2010-04-06
Hi Experts,

I am looking for a way to download the contents of a password protected webpage.  I have a legitimate username/password, but I want to setup a script to automatically download the content and add it to a database for my later review. I am going to use an RSS feed to get the links directly to the page containing the relevant content. For example, I would like to download the headlines from the Wall Street Journal via the RSS feed they provide and then run the contents of each article through a filter.

After I get the links I am stuck and I am looking for someone to give me some ideas about how I might be able to log my server into the target site and download the content. I have done something similar in the past using getURL/fopen in PHP, but for sites without a password. I would prefer to use PHP with any added libraries, but I would be willing to try out any solution.

-- Matt

I posted this question in the PHP subtopic, please answer it at the following link:
Question by:aaron_karp
    1 Comment
    LVL 41

    Accepted Solution

    Answer upated

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Training Course: Android App Development

    This course will involve creating widgets, customize list view, grid view, spinners, etc. Creating applications using audio, video, and SQLite database. Ending with publication on Google Play.

    When you work with shopping cart / ecommerce relates web sites, you need to pass the certain form post details to the payment gateway process page with required details for the products items you give to order. Also you may need to track the ordered…
    JavaScript has plenty of pieces of code people often just copy/paste from somewhere but never quite fully understand. Self-Executing functions are just one good example that I'll try to demystify here.
    Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
    The viewer will learn how to dynamically set the form action using jQuery.

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    17 Experts available now in Live!

    Get 1:1 Help Now