Urgent! - Using HttpWebRequest to Post to web page that submits a form using javascript

I'm using HttpWebRequest to post to web page.  That web page submits itself using javascript.  My issue is that it seems that web page doesn't seem to be submitting using the javascript when calling the page with a HttpWebRequest.  When running the page in a brower the javascript submits the form correctly.  Does anyone have any ideas on how to solve this?  When adding some logging to the HttpWebRequest I can see that page is being loaded properly but just doesn't seem to be submitting itself.

Below is some of the code that I've tried but has not worked.  Most of this code has been cobbled together from other posts similar to this topic.

  Public Shared Function RetrieveHttpContent(ByVal Url As String, ByRef ErrorMessage As String) As String
    Dim MergedText As String = ""
    Dim Http As System.Net.WebClient = New System.Net.WebClient()
    ' Download the Web resource and save it into a data buffer.
    Try
      Dim Result() As Byte = Http.DownloadData(Url)
      MergedText = Encoding.Default.GetString(Result)
    Catch ex As Exception
      ErrorMessage = ex.Message
      Return Nothing
    End Try

    Return MergedText
  End Function


  Public Shared Function RetrieveHttpContent2(ByVal Url As String) As String
    Dim MergedText As String = ""
    Dim Http As System.Net.HttpWebRequest = WebRequest.Create(Url)
    Http.KeepAlive = False

    'Get the data as an HttpWebResponse object
    Dim resp As HttpWebResponse = Http.GetResponse()
    ' Download the Web resource and save it into a data buffer.
    'Convert the data into a string (assumes that you are requesting text)
    Dim sr As New StreamReader(resp.GetResponseStream())
    Dim results As String = sr.ReadToEnd()
    sr.Close()
    Return results
  End Function


'I also tried async:
  ' go to the website for the info
  Private Function getSite(ByVal HttpSite As Uri) As String
    Dim wreq As HttpWebRequest = CType(WebRequest.Create(HttpSite), HttpWebRequest)
    Dim rc As IAsyncResult = CType(wreq.BeginGetResponse(New AsyncCallback(AddressOf Me.RespCallback), wreq), IAsyncResult)
    Thread.Sleep(3000)
    Return "done"
  End Function

  Public Sub RespCallback(ByVal ar As IAsyncResult)
    Dim req As HttpWebRequest = CType(ar.AsyncState, HttpWebRequest)
    Dim resp As HttpWebResponse = CType(req.EndGetResponse(ar), HttpWebResponse)
    'resp.StatusCode.ToString();

    Dim BytesRead As Integer = 0
    Dim Buffer() As Char = New Char(512) {}

    Dim Reader As StreamReader = New StreamReader(resp.GetResponseStream(), System.Text.Encoding.UTF8)
    Dim Writer As StringWriter = New StringWriter()

    BytesRead = Reader.Read(Buffer, 0, 512)
    While BytesRead <> 0
      Writer.Write(Buffer, 0, 512)
      BytesRead = Reader.Read(Buffer, 0, 512)
    End While
    Dim htmlpage As String = Writer.ToString()
    'extractInfo(htmlpage, resp.ResponseUri.AbsoluteUri)
  End Sub
LVL 9
cdaly33Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Solar_FlareCommented:
firstly, javascript has nothing do do with the actual submission of data to the web server - all it does is instruct the browser to do a postback.

By doing the postback with a httpWebRequest the javascript is irrelevant. All that matters is the post data(if its a post request) or the get data in the URL (if its a get request)

The following code is what I use when getting html pages within VB.NET

' Initialise the request
                Dim webRequest As Net.HttpWebRequest = Net.WebRequest.Create(MyURL)
                             
                ' Obtain the response
                Dim Response As Net.HttpWebResponse
                Try
                    Response = webRequest.GetResponse()
                    Dim sReader As IO.StreamReader = New IO.StreamReader(Response.GetResponseStream)
                    Dim HTML As String = sReader.ReadToEnd

                    thisHTML = HTML
                    thisResponseHeaders = Response.Headers
                    Response.Close()
                Catch webex As System.Net.WebException
                    thisErrorMessage = webex.Message
                 End Try


if the action you are trying to perform involves posting data to the webserver (eg form input boxes) then you need to include a byte array of a string of post variable (eg "id=1&submitted=true&action=find")
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASP.NET

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.