Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

I need write a script that will script a login process then grab the information from a page once logged in.

Posted on 2004-04-15
3
155 Views
Last Modified: 2010-04-06
Scenario:

I am making a screen scraping program that will gather information every day on a site.  The site does not have an rss feed so i cannot gather information that way, and their page is fairly static so this will allow me to screen scrape the information from the site.  I have proper permission to screen scrape and i have a valid user name and password, but my problem is that i do not know how to script a login.  

I need a script in vb.net or vbscript, that will basically do the following.

1. Open a web browser
2. Go to http://www.HowDoIDoThis.com/Login.asp // Arbitrary page
3. Some how type in a given user name and password
4. Submit the form
5. Once logged in, redirect to http://www.HowDoIDoThis.com/Information.asp // Arbitrary page
6. Grab all of the html from that page.

My problem is that trying to access the Information.asp page is protected, no users without passwords can go directly to it.  You have to login first, which sounds right because they want only their paying customers to view the content which i am so i just need to know how to do this bit!  

thanks ahead of time!

Flyin
0
Comment
Question by:flyin69
3 Comments
 
LVL 1

Accepted Solution

by:
Darix earned 250 total points
ID: 10840937
try to use or rewrite this class:

using System;
using System.Configuration;
using System.Net;
using System.IO;
using System.Text;
using System.Text.RegularExpressions;

namespace Tamas
{
      public class AisPoster
      {
            private AisPoster(){}

            private string baseUrl;
            private string user;
            private string password;
            private string xml;
            private string regex1;

            public AisPoster(string user,string password,string xml)
            {
                  System.Collections.Specialized.NameValueCollection config
                        = System.Configuration.ConfigurationSettings.AppSettings;
                  this.baseUrl=config["baseUrl"];
                  this.regex1=config["regex1"];
                  if(!this.baseUrl.EndsWith("/"))
                        this.baseUrl=this.baseUrl+"/";
                  this.user=user;
                  this.password=password;
                  this.xml=xml;
            }
            /// <summary>
            /// Posts data to AIS site and returns response stream.
            /// </summary>
            /// <returns></returns>
            public System.IO.Stream Post()
            {
                  string loginUrl=baseUrl+"main/login.php";
                  PostWebRequest pReq=new PostWebRequest(loginUrl);
                  pReq.AddData("Action","Login");
                  pReq.AddData("Sub_User",user);
                  pReq.AddData("Sub_Pass",password);

                  string response1=pReq.GetResponseString(pReq.GetResponse());

                  pReq.Url=baseUrl+"import/import_priimti.php";
                  pReq.ContentType=ContentType.Multipart;
                  pReq.AddData("Dokumentas","DAA");
                  pReq.AddData("Tikslas","duomenu_baze");
                  pReq.AddData("Formatas","xml");
                  pReq.AddData("userfile",@"c:\upload.xml",xml);

                  string response=pReq.GetResponseString(pReq.GetResponse());
                  System.Text.RegularExpressions.Regex regex=new Regex(regex1
                        ,RegexOptions.Compiled);
                  Match m=regex.Match(response);
                  if (!m.Success)
                        throw new Exception("Response error");
                  else
                        response=m.Result("${value}");
                  

                  pReq.Url=baseUrl+"aapjp/pj_frm_daa.php";
                  pReq.AddData("Action","import");
                  
                  pReq.AddData("Filename",response);
                  pReq.AddData("Tikslas","duomenu_baze");
                  pReq.AddData("Formatas","xml");
                  pReq.ContentType=ContentType.NonMultipart;
                  string response2=pReq.GetResponseString(pReq.GetResponse());


                  pReq.Url=baseUrl+"import/import_daa.php";
                  pReq.AddData("DokNr","");
                  pReq.AddData("Action","");
                  pReq.AddData("Filename",response);
                  pReq.AddData("Tikslas","duomenu_baze");
                  pReq.AddData("Formatas","xml");
                  pReq.ContentType=ContentType.NonMultipart;
                  return pReq.GetResponse().GetResponseStream();
            
            }
      }
}
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

What is Node.js? Node.js is a server side scripting language much like PHP or ASP but is used to implement the complete package of HTTP webserver and application framework. The difference is that Node.js’s execution engine is asynchronous and event…
Browsers only know CSS so your awesome SASS code needs to be translated into normal CSS. Here I'll try to explain what you should aim for in order to take full advantage of SASS.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
The viewer will learn the basics of jQuery including how to code hide show and toggles. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery…

791 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question