Auto login & extract data from a website

Posted on 2009-02-23
Medium Priority
Last Modified: 2012-06-27
This isn't about any website in particular but various websites require you to login before you can access your account.

I want to know how applications (after you supply your username/password) manage to login into the websites and extract your account details?

I can use a webrequest to request a webpage but how do i login before i do that (programatically)?

Question by:Type25
  • 2
  • 2
LVL 23

Accepted Solution

Jens Fiederer earned 2000 total points
ID: 23714327
It really depends on the website.

In many cases, everything is handled using forms - you supply id/password, and the response either sets a cookie or fills in some (generally hidden) field in your form with data that makes sense only to that site.

One way to do this is to use a scriptable or scripted browser - you can use and control a browser object and handle filling in the login/password information yourself.

Some sites don't like you to access them that way (you might miss out on seeing advertisements that are important to the site), so they change the way this is done on a regular basis and you will have to keep changing your program).  I.e., for some sites what you are proposing is considered hostile behavior, and will violate their terms-of-service and they might even ban you if they catch you trying to access them in a manner appropriate to an older version of their software (because only a bot would do that).

Author Comment

ID: 23714398
Okay, that makes sense.

I have a few ideas kicking about for an iPhone application, but would require me to log into a couple of pretty large organisations websites.

Probably a no-goer

LVL 23

Expert Comment

by:Jens Fiederer
ID: 23714453
Depends on the specifice orgs - the ones that are OK with it might even provide services to make it either (like Google exposing its map API) and give you easier alternatives than logging on.

If they are NOT, it is definitely a no-go.

Author Comment

ID: 23719394
Thanks for the advice

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A long time ago (May 2011), I have written an article showing you how to create a DLL using Visual Studio 2005 to be hosted in SQL Server 2005. That was valid at that time and it is still valid if you are still using these versions. You can still re…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question