Auto login & extract data from a website

Posted on 2009-02-23
Last Modified: 2012-06-27
This isn't about any website in particular but various websites require you to login before you can access your account.

I want to know how applications (after you supply your username/password) manage to login into the websites and extract your account details?

I can use a webrequest to request a webpage but how do i login before i do that (programatically)?

Question by:Type25
    LVL 23

    Accepted Solution

    It really depends on the website.

    In many cases, everything is handled using forms - you supply id/password, and the response either sets a cookie or fills in some (generally hidden) field in your form with data that makes sense only to that site.

    One way to do this is to use a scriptable or scripted browser - you can use and control a browser object and handle filling in the login/password information yourself.

    Some sites don't like you to access them that way (you might miss out on seeing advertisements that are important to the site), so they change the way this is done on a regular basis and you will have to keep changing your program).  I.e., for some sites what you are proposing is considered hostile behavior, and will violate their terms-of-service and they might even ban you if they catch you trying to access them in a manner appropriate to an older version of their software (because only a bot would do that).
    LVL 9

    Author Comment

    Okay, that makes sense.

    I have a few ideas kicking about for an iPhone application, but would require me to log into a couple of pretty large organisations websites.

    Probably a no-goer

    LVL 23

    Expert Comment

    by:Jens Fiederer
    Depends on the specifice orgs - the ones that are OK with it might even provide services to make it either (like Google exposing its map API) and give you easier alternatives than logging on.

    If they are NOT, it is definitely a no-go.
    LVL 9

    Author Comment

    Thanks for the advice

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Join & Write a Comment

    Suggested Solutions

    Many of us here at EE write code. Many of us write exceptional code; just as many of us write exception-prone code. As we all should know, exceptions are a mechanism for handling errors which are typically out of our control. From database errors, t…
    Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
    This video is in connection to the article "The case of a missing mobile phone (". It will help one to understand clearly the steps to track a lost android phone.
    In this sixth video of the Xpdf series, we discuss and demonstrate the PDFtoPNG utility, which converts a multi-page PDF file to separate color, grayscale, or monochrome PNG files, creating one PNG file for each page in the PDF. It does this via a c…

    732 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    23 Experts available now in Live!

    Get 1:1 Help Now