Solved

Make sure document is downloaded before parsing out elements

Posted on 2006-06-22
7
191 Views
Last Modified: 2010-04-23
I have a problem that took me awhile to track down where my program begins trying to use regex on a document to extract text before the document is completely downloaded. It kept getting only half of the text correctly and dropping the other half until I ran the function again. I used LEN on the document html to figure out that the two files were dramatically different in size.

 Dim EventUrl As String = "http://www.whatismyip.com"

                    Dim sResult As String
                    Dim oHttp As HttpWebRequest
                    Dim objResponse As WebResponse
                    Dim objRequest As WebRequest = System.Net.HttpWebRequest.Create(EventUrl)

                    Dim ProxyAddress As String = "XXX.XXX.XX.X"
                    Dim ProxyPort As Integer = "8088"



                    Dim oProxy As New WebProxy(ProxyAddress, ProxyPort)
                    objRequest.Proxy = oProxy

                    objRequest.Method = "GET"
                    objRequest.Timeout = 1200000 ' 20 sec.
                    objResponse = objRequest.GetResponse
                   
                    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(objResponse.GetResponseStream(), System.Text.Encoding.UTF7)

                    sResult = sr.ReadToEnd()

                    sr.Close()


                    Dim sErr As String
                    Dim FullTempFilePath As String = "C:\TJYTEMP\MAINSEL" & Rnd(500) & Rnd(350) & "TEMP.html"
                    SaveTextToFile(sResult, FullTempFilePath, sErr)

                    Debug.Write("FILE WRITE ERROR: " & sErr.ToString)

                    Dim doc As New HtmlDocument(FullTempFilePath)



                    EStatusLabel.Text = "Digesting"



Heres what the above code does, I had to make an http request through a proxy so I make the request with the proxy then saved the html to a file. This is where I think my problem lies as it must not be waiting for all of the html document to come in before saving. Then I used "Dim doc As New HtmlDocument(FullTempFilePath)" to create a doc from that file I just saved. I need to do this so I can use the mshtml to parse out the elements in the document. I know this must be an easy fix.  500 points because I need to have the skeleton of this program developed in 2 weeks! :D
0
Comment
Question by:JPERKS1985
  • 4
  • 3
7 Comments
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965498
I don't believe that you need to save it to a file to create an HtmlDocument, but I am not 100% sure about that.

If it is true, then you could do something like this:

While Not System.IO.File.Exists(FullTempFilePath)
   Application.DoEvents()
End While

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965765
Hmm, I don't think it'll work because the file will exist only not in its full form. Anyway to make it wait till the file is fully written?
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16965797
The file will not exist until it is fully written--try it.

Bob
0
Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

 
LVL 1

Author Comment

by:JPERKS1985
ID: 16965838
didin't work :(. Its as if my code keeps writing over the file it created until its full? I put msgbox (Len(Dochtml)) in the part that parses it and when it finds the remaining elements the len jumps up.
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16966529
Anything that can be done with objResponse.ContentLength? Even though I'm not sure what the full length will be?
0
 
LVL 96

Accepted Solution

by:
Bob Learned earned 500 total points
ID: 16969883
Test:

   Get the response stream into a variable, and check the length of the stream, and see if you are getting the entire response.

Bob
0
 
LVL 1

Author Comment

by:JPERKS1985
ID: 16973185
  While m_document Is Nothing OrElse m_document.readyState <> "complete"
                    Application.DoEvents()
                End While
Is what I used it seems to work good.
0

Featured Post

Networking for the Cloud Era

Join Microsoft and Riverbed for a discussion and demonstration of enhancements to SteelConnect:
-One-click orchestration and cloud connectivity in Azure environments
-Tight integration of SD-WAN and WAN optimization capabilities
-Scalability and resiliency equal to a data center

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VB.Net - Project Software to help the Developer 7 40
VB.Net - TypeInitializer Error 25 32
Error handling in asp.net site 5 34
Access/Visual Basic Question 3 29
Article by: Kraeven
Introduction Remote Share is a simple remote sharing tool, enabling you to see, add and remove remote or local shares. The application is written in VB.NET targeting the .NET framework 2.0. The source code and the compiled programs have been in…
It’s quite interesting for me as I worked with Excel using vb.net for some time. Here are some topics which I know want to share with others whom this might help. First of all if you are working with Excel then you need to Download the Following …
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question