Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Make sure document is downloaded before parsing out elements

Posted on 2006-06-22
7
Medium Priority
?
194 Views
Last Modified: 2010-04-23
I have a problem that took me awhile to track down where my program begins trying to use regex on a document to extract text before the document is completely downloaded. It kept getting only half of the text correctly and dropping the other half until I ran the function again. I used LEN on the document html to figure out that the two files were dramatically different in size.

 Dim EventUrl As String = "http://www.whatismyip.com"

                    Dim sResult As String
                    Dim oHttp As HttpWebRequest
                    Dim objResponse As WebResponse
                    Dim objRequest As WebRequest = System.Net.HttpWebRequest.Create(EventUrl)

                    Dim ProxyAddress As String = "XXX.XXX.XX.X"
                    Dim ProxyPort As Integer = "8088"



                    Dim oProxy As New WebProxy(ProxyAddress, ProxyPort)
                    objRequest.Proxy = oProxy

                    objRequest.Method = "GET"
                    objRequest.Timeout = 1200000 ' 20 sec.
                    objResponse = objRequest.GetResponse
                   
                    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(objResponse.GetResponseStream(), System.Text.Encoding.UTF7)

                    sResult = sr.ReadToEnd()

                    sr.Close()


                    Dim sErr As String
                    Dim FullTempFilePath As String = "C:\TJYTEMP\MAINSEL" & Rnd(500) & Rnd(350) & "TEMP.html"
                    SaveTextToFile(sResult, FullTempFilePath, sErr)

                    Debug.Write("FILE WRITE ERROR: " & sErr.ToString)

                    Dim doc As New HtmlDocument(FullTempFilePath)



                    EStatusLabel.Text = "Digesting"



Heres what the above code does, I had to make an http request through a proxy so I make the request with the proxy then saved the html to a file. This is where I think my problem lies as it must not be waiting for all of the html document to come in before saving. Then I used "Dim doc As New HtmlDocument(FullTempFilePath)" to create a doc from that file I just saved. I need to do this so I can use the mshtml to parse out the elements in the document. I know this must be an easy fix.  500 points because I need to have the skeleton of this program developed in 2 weeks! :D
0
Comment
Question by:JPERKS1985
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965498
I don't believe that you need to save it to a file to create an HtmlDocument, but I am not 100% sure about that.

If it is true, then you could do something like this:

While Not System.IO.File.Exists(FullTempFilePath)
   Application.DoEvents()
End While

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965765
Hmm, I don't think it'll work because the file will exist only not in its full form. Anyway to make it wait till the file is fully written?
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965797
The file will not exist until it is fully written--try it.

Bob
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965838
didin't work :(. Its as if my code keeps writing over the file it created until its full? I put msgbox (Len(Dochtml)) in the part that parses it and when it finds the remaining elements the len jumps up.
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16966529
Anything that can be done with objResponse.ContentLength? Even though I'm not sure what the full length will be?
0
 
LVL 96

Accepted Solution

by:
Bob Learned earned 2000 total points
ID: 16969883
Test:

   Get the response stream into a variable, and check the length of the stream, and see if you are getting the entire response.

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16973185
  While m_document Is Nothing OrElse m_document.readyState <> "complete"
                    Application.DoEvents()
                End While
Is what I used it seems to work good.
0

Featured Post

New benefit for Premium Members - Upgrade now!

Ready to get started with anonymous questions today? It's easy! Learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to create and use a custom WaterMark textbox class.  The custom WaterMark textbox class allows you to set the WaterMark Background Color and WaterMark text at design time.   IMAGE OF WATERMARKS STEPS Create VB …
The ECB site provides FX rates for major currencies since its inception in 1999 in the form of an XML feed. The files have the following format (reducted for brevity) (CODE) There are three files available HERE (http://www.ecb.europa.eu/stats/exch…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
Suggested Courses

664 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question