VB.NET: Get an external website

I am trying to get a page of an external website's HTML into a variable.

Here is the code
Public Shared Function GetPageText(ByVal url As String) As String
 
        Dim request As HttpWebRequest = WebRequest.Create(url)
         request.Timeout = 20000
             Using response As HttpWebResponse = request.GetResponse()
                   Using streamIP As Stream = response.GetResponseStream()
                              Using readerText As New StreamReader(streamIP)
                                                Dim htmlText As String = readerText.ReadToEnd()
                            Return htmlText
                End Using
            End Using
        End Using
    End Function



here is the error it returns:
The remote server returned an error: (406) Not Acceptable.  

The break occurs at this line:    Using response As HttpWebResponse = request.GetResponse()

Any ideas?  
arthurh88Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

darshan_derasariCommented:
Try below code. This code is from my earlier post.

http://www.experts-exchange.com/Programming/Languages/.NET/ASP.NET/Q_25350739.html
Public Sub GetData()
        Dim sResponse As String = ""
        Dim strResponseText As String = ""
        Dim myWebResponse As WebResponse
        Dim myResponseStream As Stream
        Dim myStreamReader As StreamReader
        Dim myWebRequest As HttpWebRequest
        Dim sSiteURL As String = Me.txtLanguage.Text

        Try
            myWebRequest = CType(WebRequest.Create(sSiteURL), HttpWebRequest)
            myWebRequest.Method = "GET"
            myWebRequest.Referer = "http://www.test.com"
            myWebRequest.ContentType = "application/x-www-form-urlencoded"
            myWebRequest.ContentLength = 0

            myWebResponse = myWebRequest.GetResponse()            
            myResponseStream = myWebResponse.GetResponseStream()
            myStreamReader = New StreamReader(myResponseStream)
            sResponse = myStreamReader.ReadToEnd()
            myStreamReader.Close()
            myResponseStream.Close()
            myWebResponse.Close()
        Catch ex As Exception
            Console.Write(ex.Message)
        End Try
        
    End Sub

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
arthurh88Author Commented:
hi darshan.  same error.  for example, i could not get it to work with this site:

http://www  <dot>  innerpathways  <dot>  ca
0
arthurh88Author Commented:
i got it to work by doing this:

        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"
0
arthurh88Author Commented:
needed to add this line after the get:


        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Visual Basic.NET

From novice to tech pro — start learning today.