Solved

Make sure document is downloaded before parsing out elements

Posted on 2006-06-22
7
188 Views
Last Modified: 2010-04-23
I have a problem that took me awhile to track down where my program begins trying to use regex on a document to extract text before the document is completely downloaded. It kept getting only half of the text correctly and dropping the other half until I ran the function again. I used LEN on the document html to figure out that the two files were dramatically different in size.

 Dim EventUrl As String = "http://www.whatismyip.com"

                    Dim sResult As String
                    Dim oHttp As HttpWebRequest
                    Dim objResponse As WebResponse
                    Dim objRequest As WebRequest = System.Net.HttpWebRequest.Create(EventUrl)

                    Dim ProxyAddress As String = "XXX.XXX.XX.X"
                    Dim ProxyPort As Integer = "8088"



                    Dim oProxy As New WebProxy(ProxyAddress, ProxyPort)
                    objRequest.Proxy = oProxy

                    objRequest.Method = "GET"
                    objRequest.Timeout = 1200000 ' 20 sec.
                    objResponse = objRequest.GetResponse
                   
                    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(objResponse.GetResponseStream(), System.Text.Encoding.UTF7)

                    sResult = sr.ReadToEnd()

                    sr.Close()


                    Dim sErr As String
                    Dim FullTempFilePath As String = "C:\TJYTEMP\MAINSEL" & Rnd(500) & Rnd(350) & "TEMP.html"
                    SaveTextToFile(sResult, FullTempFilePath, sErr)

                    Debug.Write("FILE WRITE ERROR: " & sErr.ToString)

                    Dim doc As New HtmlDocument(FullTempFilePath)



                    EStatusLabel.Text = "Digesting"



Heres what the above code does, I had to make an http request through a proxy so I make the request with the proxy then saved the html to a file. This is where I think my problem lies as it must not be waiting for all of the html document to come in before saving. Then I used "Dim doc As New HtmlDocument(FullTempFilePath)" to create a doc from that file I just saved. I need to do this so I can use the mshtml to parse out the elements in the document. I know this must be an easy fix.  500 points because I need to have the skeleton of this program developed in 2 weeks! :D
0
Comment
Question by:JPERKS1985
  • 4
  • 3
7 Comments
 
LVL 96

Expert Comment

by:Bob Learned
Comment Utility
I don't believe that you need to save it to a file to create an HtmlDocument, but I am not 100% sure about that.

If it is true, then you could do something like this:

While Not System.IO.File.Exists(FullTempFilePath)
   Application.DoEvents()
End While

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
Comment Utility
Hmm, I don't think it'll work because the file will exist only not in its full form. Anyway to make it wait till the file is fully written?
0
 
LVL 96

Expert Comment

by:Bob Learned
Comment Utility
The file will not exist until it is fully written--try it.

Bob
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 1

Author Comment

by:JPERKS1985
Comment Utility
didin't work :(. Its as if my code keeps writing over the file it created until its full? I put msgbox (Len(Dochtml)) in the part that parses it and when it finds the remaining elements the len jumps up.
0
 
LVL 1

Author Comment

by:JPERKS1985
Comment Utility
Anything that can be done with objResponse.ContentLength? Even though I'm not sure what the full length will be?
0
 
LVL 96

Accepted Solution

by:
Bob Learned earned 500 total points
Comment Utility
Test:

   Get the response stream into a variable, and check the length of the stream, and see if you are getting the entire response.

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
Comment Utility
  While m_document Is Nothing OrElse m_document.readyState <> "complete"
                    Application.DoEvents()
                End While
Is what I used it seems to work good.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Suggested Solutions

Article by: jpaulino
XML Literals are a great way to handle XML files and the community doesn’t use it as much as it should.  An XML Literal is like a String (http://msdn.microsoft.com/en-us/library/system.string.aspx) Literal, only instead of starting and ending with w…
Article by: Kraeven
Introduction Remote Share is a simple remote sharing tool, enabling you to see, add and remove remote or local shares. The application is written in VB.NET targeting the .NET framework 2.0. The source code and the compiled programs have been in…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now