Web scraping using VB.NET

I have a url:

    test.com/Search/NumberSearch.aspx


On the page there are a number of controls, one of them is textbox. When the user enters a six digit (approximately) number into the textbox and hits enter, the page goes to another page:


    test.com/Data/DetailsPage.aspx?mynum=123456

on that page there are a number of textboxes and other controls from which I need to scrape the data in addition to a number of links that I need to caputure in my code.

I have tried using VB.NET WebRequest:

    Dim wreq As WebRequest = WebRequest.Create("test.com/Data/DetailsPage.aspx?mynum=" & num)
    Dim wresp As HttpWebResponse = CType(wreq.GetResponse(), HttpWebResponse)  
    Dim dStream As Stream = wresp.GetResponseStream()
    Dim rdr As New StreamReader(dStream)
    Dim respStr As String = rdr.ReadToEnd()

Open in new window


As a result my respStr contains a string with html code but that code is for
   
     test.com/Search/NumberSearch.aspx

 not for the resulting    

    test.com/Data/DetailsPage.aspx?mynum=123456

 page with details.

My goal is to get the details page html programmatically.

I also tried using

    WebClient.DownloadString

but gotten the same result. Can anyone help?
LVL 35
YZlatAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ste5anSenior DeveloperCommented:
Are you getting the correct page, when you enter the URL test.com/Data/DetailsPage.aspx?mynum=123456 directly in you browser?

Is the result still correct when you disable JavaScript before entering  that URL?
YZlatAuthor Commented:
Yes, when entrred directky into the browser, the result is correct
ste5anSenior DeveloperCommented:
Even with disabled JavaScript? In this case use Fiddler to see whether you're doing the same as your browser (cookies for example).

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Visual Basic.NET

From novice to tech pro — start learning today.