Solved

Efficiently downloading large number of webpages using Visual Basic 2008

Posted on 2007-11-27
1
153 Views
Last Modified: 2010-04-30
Im currently using this code to download binary data from about 2000 uri's. The code works fine, however, once it gets near the 600th request it starts to slow down considerably. Any help with optomizing my current method or a point towards a different one is greatly appreciated thanks.
Private Function getpiece(ByVal hash As String, ByVal piece As String)
 
        Dim txturi As String = (removed)
        Dim pieceuri As New Uri(txturi)
        Dim piecerequest As System.Net.WebRequest = System.Net.WebRequest.Create(pieceuri)
        Dim pieceresponse As System.Net.WebResponse = piecerequest.GetResponse()
        ' Get the stream containing content returned by the server.
        Dim dataStream As Stream = pieceresponse.GetResponseStream()
        ' Open the stream using a StreamReader for easy access.
        Dim reader As New System.IO.BinaryReader(dataStream)
        ' Read the content.
        Dim responseFromServer() As Byte = reader.ReadBytes(264144)
        Application.DoEvents()
        reader.Close()
        dataStream.Close()
        pieceresponse.Close()
        Return responseFromServer
 
    End Function

Open in new window

0
Comment
Question by:dead1ne
1 Comment
 
LVL 38

Accepted Solution

by:
PaulHews earned 500 total points
ID: 20360930
The only thing I can think of changing is to use

dataStream.Dispose()

instead of closing the stream.  But I'll see if anyone else has anything to add.
0

Featured Post

Networking for the Cloud Era

Join Microsoft and Riverbed for a discussion and demonstration of enhancements to SteelConnect:
-One-click orchestration and cloud connectivity in Azure environments
-Tight integration of SD-WAN and WAN optimization capabilities
-Scalability and resiliency equal to a data center

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
MS Access Search and Replace Using VBA 6 80
Visual Studio 2005 text editor 10 43
Excel Automation VBA 19 88
RUNRMTCMD from AS/400 12 68
When trying to find the cause of a problem in VBA or VB6 it's often valuable to know what procedures were executed prior to the error. You can use the Call Stack for that but it is often inadequate because it may show procedures you aren't intereste…
You can of course define an array to hold data that is of a particular type like an array of Strings to hold customer names or an array of Doubles to hold customer sales, but what do you do if you want to coordinate that data? This article describes…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question