?
Solved

VB.NET:  Get an external website

Posted on 2010-03-24
5
Medium Priority
?
379 Views
Last Modified: 2012-05-09
I am trying to get a page of an external website's HTML into a variable.

Here is the code
Public Shared Function GetPageText(ByVal url As String) As String
 
        Dim request As HttpWebRequest = WebRequest.Create(url)
         request.Timeout = 20000
             Using response As HttpWebResponse = request.GetResponse()
                   Using streamIP As Stream = response.GetResponseStream()
                              Using readerText As New StreamReader(streamIP)
                                                Dim htmlText As String = readerText.ReadToEnd()
                            Return htmlText
                End Using
            End Using
        End Using
    End Function



here is the error it returns:
The remote server returned an error: (406) Not Acceptable.  

The break occurs at this line:    Using response As HttpWebResponse = request.GetResponse()

Any ideas?  
0
Comment
Question by:arthurh88
  • 3
5 Comments
 
LVL 5

Accepted Solution

by:
darshan_derasari earned 2000 total points
ID: 28431909
Try below code. This code is from my earlier post.

http://www.experts-exchange.com/Programming/Languages/.NET/ASP.NET/Q_25350739.html
Public Sub GetData()
        Dim sResponse As String = ""
        Dim strResponseText As String = ""
        Dim myWebResponse As WebResponse
        Dim myResponseStream As Stream
        Dim myStreamReader As StreamReader
        Dim myWebRequest As HttpWebRequest
        Dim sSiteURL As String = Me.txtLanguage.Text

        Try
            myWebRequest = CType(WebRequest.Create(sSiteURL), HttpWebRequest)
            myWebRequest.Method = "GET"
            myWebRequest.Referer = "http://www.test.com"
            myWebRequest.ContentType = "application/x-www-form-urlencoded"
            myWebRequest.ContentLength = 0

            myWebResponse = myWebRequest.GetResponse()            
            myResponseStream = myWebResponse.GetResponseStream()
            myStreamReader = New StreamReader(myResponseStream)
            sResponse = myStreamReader.ReadToEnd()
            myStreamReader.Close()
            myResponseStream.Close()
            myWebResponse.Close()
        Catch ex As Exception
            Console.Write(ex.Message)
        End Try
        
    End Sub

Open in new window

0
 

Author Comment

by:arthurh88
ID: 28435820
hi darshan.  same error.  for example, i could not get it to work with this site:

http://www  <dot>  innerpathways  <dot>  ca
0
 

Author Comment

by:arthurh88
ID: 28468773
i got it to work by doing this:

        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"
0
 

Author Closing Comment

by:arthurh88
ID: 31706470
needed to add this line after the get:


        myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"
0

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In an ASP.NET application, I faced some technical problems. In this article, I list them out and show the solutions that I found.  I hope it will be useful. Problem: After closing a pop-up window, the parent page should be refreshed automaticall…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Stellar Phoenix SQL Database Repair software easily fixes the suspect mode issue of SQL Server database. It is a simple process to bring the database from suspect mode to normal mode. Check out the video and fix the SQL database suspect mode problem.
The video provides a quick and easy steps to migrate MBOX file to well known Outlook PST and Office 365. Besides this, it also supports and migrates more than 20 email clients of MBOX which include AppleMail, Opera, Thunderbird and SeaMonkey effortl…

592 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question