Urgent! - Using HttpWebRequest to Post to web page that submits a form using javascript

Posted on 2006-03-25
Last Modified: 2012-08-14
I'm using HttpWebRequest to post to web page.  That web page submits itself using javascript.  My issue is that it seems that web page doesn't seem to be submitting using the javascript when calling the page with a HttpWebRequest.  When running the page in a brower the javascript submits the form correctly.  Does anyone have any ideas on how to solve this?  When adding some logging to the HttpWebRequest I can see that page is being loaded properly but just doesn't seem to be submitting itself.

Below is some of the code that I've tried but has not worked.  Most of this code has been cobbled together from other posts similar to this topic.

  Public Shared Function RetrieveHttpContent(ByVal Url As String, ByRef ErrorMessage As String) As String
    Dim MergedText As String = ""
    Dim Http As System.Net.WebClient = New System.Net.WebClient()
    ' Download the Web resource and save it into a data buffer.
      Dim Result() As Byte = Http.DownloadData(Url)
      MergedText = Encoding.Default.GetString(Result)
    Catch ex As Exception
      ErrorMessage = ex.Message
      Return Nothing
    End Try

    Return MergedText
  End Function

  Public Shared Function RetrieveHttpContent2(ByVal Url As String) As String
    Dim MergedText As String = ""
    Dim Http As System.Net.HttpWebRequest = WebRequest.Create(Url)
    Http.KeepAlive = False

    'Get the data as an HttpWebResponse object
    Dim resp As HttpWebResponse = Http.GetResponse()
    ' Download the Web resource and save it into a data buffer.
    'Convert the data into a string (assumes that you are requesting text)
    Dim sr As New StreamReader(resp.GetResponseStream())
    Dim results As String = sr.ReadToEnd()
    Return results
  End Function

'I also tried async:
  ' go to the website for the info
  Private Function getSite(ByVal HttpSite As Uri) As String
    Dim wreq As HttpWebRequest = CType(WebRequest.Create(HttpSite), HttpWebRequest)
    Dim rc As IAsyncResult = CType(wreq.BeginGetResponse(New AsyncCallback(AddressOf Me.RespCallback), wreq), IAsyncResult)
    Return "done"
  End Function

  Public Sub RespCallback(ByVal ar As IAsyncResult)
    Dim req As HttpWebRequest = CType(ar.AsyncState, HttpWebRequest)
    Dim resp As HttpWebResponse = CType(req.EndGetResponse(ar), HttpWebResponse)

    Dim BytesRead As Integer = 0
    Dim Buffer() As Char = New Char(512) {}

    Dim Reader As StreamReader = New StreamReader(resp.GetResponseStream(), System.Text.Encoding.UTF8)
    Dim Writer As StringWriter = New StringWriter()

    BytesRead = Reader.Read(Buffer, 0, 512)
    While BytesRead <> 0
      Writer.Write(Buffer, 0, 512)
      BytesRead = Reader.Read(Buffer, 0, 512)
    End While
    Dim htmlpage As String = Writer.ToString()
    'extractInfo(htmlpage, resp.ResponseUri.AbsoluteUri)
  End Sub
Question by:cdaly33
    1 Comment
    LVL 15

    Accepted Solution

    firstly, javascript has nothing do do with the actual submission of data to the web server - all it does is instruct the browser to do a postback.

    By doing the postback with a httpWebRequest the javascript is irrelevant. All that matters is the post data(if its a post request) or the get data in the URL (if its a get request)

    The following code is what I use when getting html pages within VB.NET

    ' Initialise the request
                    Dim webRequest As Net.HttpWebRequest = Net.WebRequest.Create(MyURL)
                    ' Obtain the response
                    Dim Response As Net.HttpWebResponse
                        Response = webRequest.GetResponse()
                        Dim sReader As IO.StreamReader = New IO.StreamReader(Response.GetResponseStream)
                        Dim HTML As String = sReader.ReadToEnd

                        thisHTML = HTML
                        thisResponseHeaders = Response.Headers
                    Catch webex As System.Net.WebException
                        thisErrorMessage = webex.Message
                     End Try

    if the action you are trying to perform involves posting data to the webserver (eg form input boxes) then you need to include a byte array of a string of post variable (eg "id=1&submitted=true&action=find")

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Training Course: Java/J2EE and SOA

    This course will cover both core and advanced Java concepts like Database connectivity, Threads, Exception Handling, Collections, JSP, Servlets, XMLHandling, and more. You'll also learn various Java frameworks like Hibernate and Spring.

    I have developed many web applications with asp & and to add and use a dropdownlist was always a very simple task, but with the new, setting the value is a bit tricky and its not similar to the old traditional method. So in this a…
    In an ASP.NET application, I faced some technical problems. In this article, I list them out and show the solutions that I found.  I hope it will be useful. Problem: After closing a pop-up window, the parent page should be refreshed automaticall…
    Need more eyes on your posted question? Go ahead and follow the quick steps in this video to learn how to Request Attention to your question. *Log into your Experts Exchange account *Find the question you want to Request Attention for *Go to the e…
    Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

    760 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    12 Experts available now in Live!

    Get 1:1 Help Now