• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 119
  • Last Modified:

Read buffer

I'm trying to read a file in segments and pass it into the function uploadFile(), however when I try and read large files I get memory issues:-

        Dim uploadBytesSize As Integer = 200000

        Dim sr As New IO.StreamReader("C:\A Large File.mp4")
        Dim fileLength As Long = sr.BaseStream.Length
        Dim bytesRemaining As Long = fileLength
        Dim bufferSize As Long
        Dim bytesUploaded As Long = 0

        Do
            If bytesRemaining <= uploadBytesSize Then
                buffersize = bytesRemaining
            Else
                bufferSize = uploadBytesSize
            End If

            Dim buffer(bufferSize - 1) As Byte
            sr.BaseStream.Position = bytesUploaded
            sr.BaseStream.Read(buffer, 0, bufferSize)

            bytesUploaded += bufferSize

            uploadFile(buffer)

        Loop Until bytesRemaining < uploadBytesSize

Open in new window


Seems to work fine for files for a few Mb, but anything over 300Mb it seems to fall over when I try to access the file buffer. I only need to access a small portion of the file at a time (defined by uploadBytesSize)

Does anyone have any ideas how I can access a file in segments?
0
tonelm54
Asked:
tonelm54
1 Solution
 
Shaun KlineLead Software EngineerCommented:
It does not appear that the condition for your loop will be satisfied if the file size is larger than 200000 because you are not refreshing the value of bytesRemaining. You should add a line of code after
bytesUploaded += bufferSize

Open in new window

with
bytesRemaining -= bufferSize

Open in new window

0
 
Robberbaron (robr)Commented:
your code doesnt make complete sense to me as BytesRemaining never changes.

msdn seems to indicate a slightly different method, http://msdn.microsoft.com/en-us/library/9kstw824(v=vs.110).aspx?cs-save-lang=1&cs-lang=vb#code-snippet-2


note that .Read reads up to the buffer size, so no need to change buffersize.
        Dim sr As New IO.StreamReader("C:\A Large File.mp4")
        Dim fileLength As Long = sr.BaseStream.Length

        Dim bufferSize As Integer = 4096 * 50  '4k block size most efficient
        Dim bytesUploaded As Long = 0
        Dim thisreadsize as integer

        Do While sr.Peek() => 0

            Dim buffer(bufferSize) As Byte
            'reads MAX buffer bytes.  returns less if end of file is reached
            thisreadsize = sr.BaseStream.Read(buffer, 0, buffer.Length)

            bytesUploaded += thisreadsize

            uploadFile(buffer)

        Loop 

Open in new window

0
 
tonelm54Author Commented:
Modified the MSDN solution and works good now :--)
0

Featured Post

[Webinar] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now