VB.NET:  Get an external website

Posted on 2010-03-24
Medium Priority
Last Modified: 2012-05-09
I am trying to get a page of an external website's HTML into a variable.

Here is the code
Public Shared Function GetPageText(ByVal url As String) As String
        Dim request As HttpWebRequest = WebRequest.Create(url)
         request.Timeout = 20000
             Using response As HttpWebResponse = request.GetResponse()
                   Using streamIP As Stream = response.GetResponseStream()
                              Using readerText As New StreamReader(streamIP)
                                                Dim htmlText As String = readerText.ReadToEnd()
                            Return htmlText
                End Using
            End Using
        End Using
    End Function

here is the error it returns:
The remote server returned an error: (406) Not Acceptable.  

The break occurs at this line:    Using response As HttpWebResponse = request.GetResponse()

Any ideas?  
Question by:arthurh88
  • 3

Accepted Solution

darshan_derasari earned 2000 total points
ID: 28431909
Try below code. This code is from my earlier post.

Public Sub GetData()
        Dim sResponse As String = ""
        Dim strResponseText As String = ""
        Dim myWebResponse As WebResponse
        Dim myResponseStream As Stream
        Dim myStreamReader As StreamReader
        Dim myWebRequest As HttpWebRequest
        Dim sSiteURL As String = Me.txtLanguage.Text

            myWebRequest = CType(WebRequest.Create(sSiteURL), HttpWebRequest)
            myWebRequest.Method = "GET"
            myWebRequest.Referer = "http://www.test.com"
            myWebRequest.ContentType = "application/x-www-form-urlencoded"
            myWebRequest.ContentLength = 0

            myWebResponse = myWebRequest.GetResponse()            
            myResponseStream = myWebResponse.GetResponseStream()
            myStreamReader = New StreamReader(myResponseStream)
            sResponse = myStreamReader.ReadToEnd()
        Catch ex As Exception
        End Try
    End Sub

Open in new window


Author Comment

ID: 28435820
hi darshan.  same error.  for example, i could not get it to work with this site:

http://www  <dot>  innerpathways  <dot>  ca

Author Comment

ID: 28468773
i got it to work by doing this:

        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"

Author Closing Comment

ID: 31706470
needed to add this line after the get:

        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In an ASP.NET application, I faced some technical problems. In this article, I list them out and show the solutions that I found.  I hope it will be useful. Problem: After closing a pop-up window, the parent page should be refreshed automaticall…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Stellar Phoenix SQL Database Repair software easily fixes the suspect mode issue of SQL Server database. It is a simple process to bring the database from suspect mode to normal mode. Check out the video and fix the SQL database suspect mode problem.
The video provides a quick and easy steps to migrate MBOX file to well known Outlook PST and Office 365. Besides this, it also supports and migrates more than 20 email clients of MBOX which include AppleMail, Opera, Thunderbird and SeaMonkey effortl…

592 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question