asked on

C# autonomous web browsing

I would like to mine some data on a given page.

I want to be able to be able to download a webpage and load some values into a text box:

example
using (System.Net.WebClient client = new System.Net.WebClient()) {
try {
return client.DownloadString("www.goggle.ca");
}
catch{}
}

The thing is I want to be able to when the page downloads. In the case of google there is a main textbox on the page. I want to be able to programmically enter something into the textbox and then resubmit the page and then redownload the page using the same method above.

The point is I want to be able to write a bunch of operations (go to google.ca, search for cats, choose 3rd one) all programmically..Almost like a crawler. expect one that can interact with the page..

Any ideas on this would be very appreciated.

aibusinesssolutions

It sounds almost like you are looking for a macro bot, since you want it to interact with the interface.

There is a PHP class called snoopy that emulates web browsing, you could look at it and maybe get some ideas. http://sourceforge.net/projects/snoopy/

copyPasteGhost

ASKER

That's cool any C# examples? something more.....NETish?

SOLUTION

aibusinesssolutions

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER CERTIFIED SOLUTION

Tony McCreath

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

copyPasteGhost

ASKER

silly question but..is you solution dependent on whether the form's action is Get or Post? I would like this solution to work with any webpage.

I'm guessing the NameValuePair is the id of the field and the contents of the field right? I also just noticed you are pasting the values into the URL. What if the form method is post? like asp.net pages. will that still work?

Ghost

Tony McCreath

The solution will do get or post based on if you set post to true.

If it posts then the data is added to the data stream while non-post (get) will add the data to the url.

namevalue is related as you state (is it 'id' or 'name' attributes that get posted?)

copyPasteGhost

ASKER

I see. very cool.

I'm going to give it a shot later in the week. I'll get back to you.