Solved

Make sure document is downloaded before parsing out elements

Posted on 2006-06-22
7
189 Views
Last Modified: 2010-04-23
I have a problem that took me awhile to track down where my program begins trying to use regex on a document to extract text before the document is completely downloaded. It kept getting only half of the text correctly and dropping the other half until I ran the function again. I used LEN on the document html to figure out that the two files were dramatically different in size.

 Dim EventUrl As String = "http://www.whatismyip.com"

                    Dim sResult As String
                    Dim oHttp As HttpWebRequest
                    Dim objResponse As WebResponse
                    Dim objRequest As WebRequest = System.Net.HttpWebRequest.Create(EventUrl)

                    Dim ProxyAddress As String = "XXX.XXX.XX.X"
                    Dim ProxyPort As Integer = "8088"



                    Dim oProxy As New WebProxy(ProxyAddress, ProxyPort)
                    objRequest.Proxy = oProxy

                    objRequest.Method = "GET"
                    objRequest.Timeout = 1200000 ' 20 sec.
                    objResponse = objRequest.GetResponse
                   
                    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(objResponse.GetResponseStream(), System.Text.Encoding.UTF7)

                    sResult = sr.ReadToEnd()

                    sr.Close()


                    Dim sErr As String
                    Dim FullTempFilePath As String = "C:\TJYTEMP\MAINSEL" & Rnd(500) & Rnd(350) & "TEMP.html"
                    SaveTextToFile(sResult, FullTempFilePath, sErr)

                    Debug.Write("FILE WRITE ERROR: " & sErr.ToString)

                    Dim doc As New HtmlDocument(FullTempFilePath)



                    EStatusLabel.Text = "Digesting"



Heres what the above code does, I had to make an http request through a proxy so I make the request with the proxy then saved the html to a file. This is where I think my problem lies as it must not be waiting for all of the html document to come in before saving. Then I used "Dim doc As New HtmlDocument(FullTempFilePath)" to create a doc from that file I just saved. I need to do this so I can use the mshtml to parse out the elements in the document. I know this must be an easy fix.  500 points because I need to have the skeleton of this program developed in 2 weeks! :D
0
Comment
Question by:JPERKS1985
  • 4
  • 3
7 Comments
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965498
I don't believe that you need to save it to a file to create an HtmlDocument, but I am not 100% sure about that.

If it is true, then you could do something like this:

While Not System.IO.File.Exists(FullTempFilePath)
   Application.DoEvents()
End While

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965765
Hmm, I don't think it'll work because the file will exist only not in its full form. Anyway to make it wait till the file is fully written?
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965797
The file will not exist until it is fully written--try it.

Bob
0
Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965838
didin't work :(. Its as if my code keeps writing over the file it created until its full? I put msgbox (Len(Dochtml)) in the part that parses it and when it finds the remaining elements the len jumps up.
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16966529
Anything that can be done with objResponse.ContentLength? Even though I'm not sure what the full length will be?
0
 
LVL 96

Accepted Solution

by:
Bob Learned earned 500 total points
ID: 16969883
Test:

   Get the response stream into a variable, and check the length of the stream, and see if you are getting the entire response.

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16973185
  While m_document Is Nothing OrElse m_document.readyState <> "complete"
                    Application.DoEvents()
                End While
Is what I used it seems to work good.
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Introduction When many people think of the WebBrowser (http://msdn.microsoft.com/en-us/library/2te2y1x6%28v=VS.85%29.aspx) control, they immediately think of a control which allows the viewing and navigation of web pages. While this is true, it's a…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question