Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Reclaiming memory from the allocation of a large byte array

Posted on 2004-11-22
32
Medium Priority
?
1,539 Views
Last Modified: 2008-03-17
Under VB 6 you could delcare a dynamic array, make it as big as you wanted it, then Erase it to reclaim the memory allocated to it.

Under VB.NET, I'm allocating a 20mb byte array as a local variable in a function.

 Dim TotalBytes(20000000) As Byte

I've tried everything I can think of to get VB to release the memory allocated to this array once it goes out of scope....

I've tried Dimming it inside of an If/Then/Else block, and Erasing it:

                Erase TotalBytes

I've tried Clearing it:

                TotalBytes.Clear(TotalBytes, 0, 20000000)

I've tried Redimming it:

               Redim TotalBytes(1)

I've tried setting it to Nothing:

              TotalBytes = Nothing

And, at the end of the procedure, I've tried collecting the garbage:

            GC.Collect()

However, if I'm looking at my Available Memory under the Task Manager, I don't get the memory back until I exit the program.  Now, In the past, I've been able to get around things like this with a kludge such as loading a hidden form with this code inside... when the function ends, then unload the form and set it to nothing... that might work, but I'm wondering, does anybody have another technique for reclaiming the memory allocated to a large byte array?

Thanks!
0
Comment
Question by:mdougan
  • 12
  • 11
  • 5
  • +1
32 Comments
 
LVL 96

Expert Comment

by:Bob Learned
ID: 12647383
What is the big picture?  Why allocate 20 MB byte array?  It might be better to look at the other things around this to see why the memory is not being released.

Bob
0
 
LVL 24

Expert Comment

by:Jeff Certain
ID: 12647465
What about using an Array object in place of the Byte()? I suspect part of the issue is that Byte() is an unmanaged type, and may not be collected.
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12647605
mdougan, I find that redim + gc.collect does free the memory right away for my test application.  Without the gc.collect of course you have to wait for the runtime to collect and free the memory.




0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 38

Expert Comment

by:PaulHews
ID: 12647626
I can also confirm that setting = Nothing does not return the memory right away...with or without an explicit gc.collect.
0
 
LVL 18

Author Comment

by:mdougan
ID: 12647631
OK, valid question :)

The scenario is that we're connecting to a server using a TcpClient object, we're sending up what could be a rather large xml request which the server processes and sends back what could be a rather large response...

We currently do this using HTTP Posts, but in a two step, asynchcronous process... send up the requests.... then another program polls and pulls down results whenever they're queued up.  

Since the server is soon to be moved into our LAN, we're looking at turning this into a synchronous process using the NetworkStream and TcpClient objects.  The networkstream's Read method takes a byte array and fills it with data, but if your data is over a certain size, it buffers it, sending you little bits at a time.  

My first tests had me looping and concatenating the data returned in the byte array to a string until I read the terminating xml tag, then , I could load it into a DOM or put it in a textbox or whatever, but in a test with data of about 2mb, the string concatenations failed to hold the entire data returned!  Even though strings no longer have a max size limit.  I could concatenate the data returned with the textbox's text property and get the entire 2mb of data... but I wanted some type of program variable that I could use to eventually load the DOM object with, not a gui control... so, with some experimentation, I found that if I moved the bytes from the byte array used in the Read method to a TotalBytes array, the TotalBytes array would, in fact, hold all the data... and I could then, once all the data has been read, convert it's contents using:

oDom.LoadXml(Encoding.ASCII.GetString(TotalBytes))

That all works great, but now I'd like to reclaim the memory allocated to the TotalBytes array...  we don't know ahead of time exactly how big the result is going to be, so, we can't allocate exactly what we'll need... so, I was testing with allocating something larger than our probable max transmission size to-date.

In VB6 you could create a "dynamic" array, resize it larger as needed, then erase it to reclaim all of the storage allocated, but I think under .NET arrays are no longer dynamic....



0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12647650
Erase and clear I would not expect to work, as they are meant to initialize the array.  Sorry for the multiple posts.  My brain is fragmented right now.
0
 
LVL 24

Expert Comment

by:Jeff Certain
ID: 12647668
Have you tried using stringBuilder in place of the string? For one thing, it performs much better for concatenations...
0
 
LVL 18

Author Comment

by:mdougan
ID: 12647728
PaulHews, just tried again with Redim and GC.Collect, but don't show any memory reclaimed... can you post the code for your test ?

Chaosian, I think that an object array might work, but I'm not really sure of the syntax for allocating that, and/or whether the networkstream.Read method would accept an array that wasn't allocated as a Byte:

        Dim tcpClient As New System.Net.Sockets.TcpClient()
        tcpClient.Connect("120.0.0.1", 1234)

         Dim bytes(tcpClient.ReceiveBufferSize) As Byte
         Dim TotalBytes(20000000) As Byte

        Dim clientdata As String
        Dim BytesRead As Int32
        Dim CurrIndex As Int32 = 0

                Do
                    clientdata = ""
                    BytesRead = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                    clientdata = Encoding.ASCII.GetString(bytes)
                    bytes.CopyTo(TotalBytes, CurrIndex)
                    CurrIndex += BytesRead

                    If InStr(clientdata, "</end_messages>", CompareMethod.Binary) > 0 Then
                        Exit Do
                    End If

                    bytes.Clear(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                Loop Until False

0
 
LVL 18

Author Comment

by:mdougan
ID: 12647742
Yes, I tried the stringbuilder, as I am aware of the performance boost, but it actually failed to hold as much as a when concatenating to a string.... big disappointment
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12647762
This is my test code:

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim TotalBytes(20000000) As Byte
        'ReDim TotalBytes(20000000)
        For i As Integer = 0 To 20000000  'do something with it.  The compiler seems too smart to allocate memory for the unused array.
            TotalBytes(i) = 1
        Next
        ReDim TotalBytes(0)
        'TotalBytes = Nothing
        GC.Collect()
        MsgBox("Done")
    End Sub

Taskmanager shows ~8M at start up.  Then when you click the button, the peak mem usage field shows ~28M, but the current mem usage is still at 8M.
0
 
LVL 24

Expert Comment

by:Jeff Certain
ID: 12647802
Well, the Array shouldn't be an issue -- you're reading into bytes, which can be left as byte(). The CopyTo(bytes.copyto)  function should work as well.
0
 
LVL 18

Author Comment

by:mdougan
ID: 12647919
Paul, I was redimming to 1, not 0, but I tried again with 0 and no results... what are you looking at in the task manager?  I'm looking at the Performance tab, and over at the Physical Memory... particularly the Available counter.  I also brought up perfmon with the same results... available drops until I end the program.
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 12647958
If you look in the help file for BufferedStream, you will find an example of using a BufferedStream with a NetworkStream to read chunks of data.  The buffer size would not have to be 20 MB.  In the example, they have a buffer size of 1K.  You can experiment with this to see how it could help you with memory management.

Bob
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648094
Chaosian, are you suggesting changing my dims as:

         Dim bytes(tcpClient.ReceiveBufferSize) As Object
         Dim TotalBytes(20000000) As Object


When I try that I get syntax errors for the Read and the Encoding calls.
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12648101
>what are you looking at in the task manager?
The processes tab shows memory usage and peak memory usage by process.  That's what I've been using to guage the memory usage.  If I look at the performance tab, I see the change when the program loads, but the subroutine runs too quickly for me to see any change in the total memory when it runs.

I find it very strange that the string concatenation is failing.  Are you certain that you are retrieving the string with the correct encoding?  Are you decoding to ASCII when it should be UTF-8 maybe?
0
 
LVL 24

Expert Comment

by:Jeff Certain
ID: 12648136
dim totalBytes as Array =  Array.CreateInstance(GetType(Byte),20000000)

leave bytes alone.
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648151
TheLearnedOne,  I am effectively buffering my reads with the bytes array being dimmed at tcpClient.ReceiveBufferSize, but the problem is that I need to concatenate all of my results into one big string or array of some sort so that once I've got the whole thing, I can load it into a DOM.  Problem occurs because a string or stringbuilder variable can't seem to handle anything above about 84K (or was that 840K?)
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648231
Chaosian, tried that, when finished, I tried both    Erase totalBytes     and    totalBytes = Nothing    both followed by a GC.Collect and still nothing, no return of the large chunk of available memory until the program ends.
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 12648251
When you create the StringBuilder, do you give it an initial size.  The MaxCapacity for the StringBuilder class is Int32.MaxSize which is significantly larger than 840K.  If you don't give it an initial size, then its efficiency is reduced, because it has to increase its size to fit.

Bob
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 12648268
Also, garbage collection will not work is there is a memory reference, even if you force with GC.Collect.  What is the lifetime of the holding object?  Do you make additional references, circular references, etc.?

Bob
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648312
Paul, I look at available memory only... this drops and stays down until the program ends.

Yes, I was pretty incredulous about the string concatenation too, but I really exhausted all possibilities before arriving at that conclusion...

When I do the encode to ascii the data looks correct, and if I do this:

      Dim tcpClient As New System.Net.Sockets.TcpClient()
        tcpClient.Connect("120.0.0.1", 1234)

         Dim bytes(tcpClient.ReceiveBufferSize) As Byte

        Dim clientdata As String

                Do
                    clientdata = ""
                    networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                    clientdata = Encoding.ASCII.GetString(bytes)
                    txtReceived.Text += clientdata

                    If InStr(clientdata, "</end_messages>", CompareMethod.Binary) > 0 Then
                        Exit Do
                    End If

                    bytes.Clear(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                Loop Until False

The textbox gets the complete stream of data... however, if I do this, then I only get about the first 80K or so, even though the instr eventually gets a hit and I exit the loop:

      Dim tcpClient As New System.Net.Sockets.TcpClient()
        tcpClient.Connect("120.0.0.1", 1234)

         Dim bytes(tcpClient.ReceiveBufferSize) As Byte

        Dim clientdata As String
        Dim sResult as String
                Do
                    clientdata = ""
                    networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                    clientdata = Encoding.ASCII.GetString(bytes)
                    sResult += clientdata  'or do a concatenate, or use a stringbuilder

                    If InStr(clientdata, "</end_messages>", CompareMethod.Binary) > 0 Then
                        Exit Do
                    End If

                    bytes.Clear(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                Loop Until False

                txtResult.text = sResult  ' and the textbox's maxsize has been set to zero
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648453
TheLearnedOne, yes, I tried setting the "capacity" for the stringbuilder before using it... yes, the maxcapacity is 2 billion which would be more than enough... what is weird is that the string builder's length only grew to about 84 (or 840)K when the total string is about 1.6mb.  Didn't notice that stringbuilder has an initialsize property but I could try it...

I thought about the scope of the variables, first, they were Dimmed outside of my main try/catch block, and I'm doing the GC.Collect in the Finally of the try block.   Then, I moved them inside of an If/Then/Else block and before exiting the If statement, I'm trying to set them to nothing or erase them etc.

            If networkStream.CanWrite And networkStream.CanRead Then
                ' Do a simple write.
                Dim bytes(tcpClient.ReceiveBufferSize) As Byte
                Dim TotalBytes(20000000) As Byte

                Do
                    clientdata = ""
                    BytesRead = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12648558
UTF-8 will resemble ASCII until you hit an out of range character (> ASCII code 127.)  Also string concat will fail on chr(0) although *not* for a text box:

   Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim s As String
        Dim i As Int16
        For i = 0 To 9
            TextBox1.Text = TextBox1.Text & "hello" & Chr(0)
            s = s & "hello" & Chr(0)
        Next
        MsgBox(s)
    End Sub


 
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 12648560
Dim sb As StringBuilder = New StringBuilder(Integer.MaxValue)

Bob
0
 
LVL 38

Accepted Solution

by:
PaulHews earned 2000 total points
ID: 12648607
If you are sure that it is ASCII encoded try substituting this:

Dim ReceivedLength As Integer = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
clientdata = System.Text.Encoding.ASCII.GetString(bytes, 0, ReceivedLength)
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12648627
String concat isn't failing on chr(0)...  Just the message box, sorry.
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12648713
I was surprised that the text box concatenation didn't fail, since it probably would in VB 6.  However, assigning the string with embedded chr(0) does cause it to show only the first segment, the same as it would in VB 6.  I wouldn't discount it as a possible cause of the failure, since you would not be able to print, display in a text box, or use in an XMLDoc.Load if there was an embedded zero character.  Maybe a simple .Replace call would fix it.
0
 
LVL 18

Author Comment

by:mdougan
ID: 12648995
Paul, the file that I was experimenting with was a large file full of SQL dumped out of Enterprise Manager as a .sql file... so, I'm assuming that it's ASCII, but I don't know... could be that some special character is blowing things up....

Well, your last suggestion:

                    'clientdata = Encoding.ASCII.GetString(bytes)
                    clientdata = System.Text.Encoding.ASCII.GetString(bytes, 0, BytesRead)

Solved the concatenation problem and also allows me to use the string builder and get the entire 2mb into a single string without having to allocate the huge 20mb TotalBytes buffer... so, goes pretty far to solving my problem..., however, the question still remains how to recover the memory... we've been mainly talking about the receiving back of the response, but I'm also creating a byte array to send out the data request:

Dim sendBytes As [Byte]() = Encoding.ASCII.GetBytes("<my_messages>" & Me.txtSend.Text & "</my_messages>" & vbCrLf)
networkStream.Write(sendBytes, 0, sendBytes.Length)

So, if I have a 2mb request, I'll still be allocating a 2mb byte array that doesn't get released at the end of the procedure...  likewise, my 2mb result string or stringbuilder doesn't seem to be letting go of the memory either.  Even when doing the following:

                Dim sXML As New StringBuilder()
                    sXML.Append(clientdata)


                ReDim sendBytes(0)
                ReDim bytes(0)
                sXML.Remove(0, sXML.Length)  'sXML is a stringbuilder object
                sXML = Nothing
                GC.Collect()

0
 
LVL 18

Author Comment

by:mdougan
ID: 12649046
Oh, Paul, one thing I forgot to mention, I first read the .sql file into another textbox, and I could verify that the entire 1.6mb file loaded fine into the textbox... so, I figured I was safe from any embedded binary zeros... I was even using the same sort of stringbuilder code:

        Dim SReadLine As Stream
        Dim fReader As StreamReader
        Dim sb As New StringBuilder()
        Me.Cursor = Cursors.WaitCursor

        SReadLine = File.OpenRead("C:\Temp.sql")
        fReader = New StreamReader(SReadLine, _
            System.Text.Encoding.ASCII)
        fReader.BaseStream.Seek(0, SeekOrigin.Begin)

        While (fReader.Peek() > -1)
            sb.Append(fReader.ReadLine)
        End While
        fReader.Close()

        txtSend.Text += sb.ToString
        Me.Cursor = Cursors.Default

The responses are basically the same exact text echoed back.... except that I wrap the whole thing with an xml begin and end tag:

<my_messages></my_messages> & vbCrlf

0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12649388
>Oh, Paul, one thing I forgot to mention, I first read the .sql file into another textbox, and I could verify that the entire 1.6mb file loaded fine into the textbox...

Sounds like the embedded 0 characters are coming from trying to encode the entire buffer, which makes sense if the text is smaller than the buffer.  

As for releasing the memory, it's hard for me to try it since I'm not having the same problem in my test code.  Garbage collection is not supposed to be done manually, however, when I've needed to do it, it's been working for me.  One wonder's how accurately Windows reports the inner workings of your program.  

Try using
Debug.WriteLine(GC.GetTotalMemory(True))

Before and after deallocation to see what the change is.  Using the true parameter will force a full collection, so there is no need to call collect.
0
 
LVL 18

Author Comment

by:mdougan
ID: 12650112
OK Paul, I think you're the winner :)

With the debug's the before/after was:

3307450
 184868

which means about 3mb freed up, even though perfmon or the task manager still don't show it as available... I'd image that if some process needed it, that memory could be utilized.  For the record, here is my deallocation code:

                Debug.WriteLine(GC.GetTotalMemory(True))

                Me.txtReceive.Text = CType(sXML.ToString, String).Substring(CType(sXML.ToString, String).Length - 18, 18)

' NOTE: these figures are when run seperately, when run together, total freed shows as 3124358
                Erase sendBytes  ' frees up 3071699, about 38 bytes more than Redim sendBytes(0)
                Erase bytes         ' frees up 7836 bytes
                sXML = Nothing   ' frees up 2097078 bytes

                Debug.WriteLine(GC.GetTotalMemory(True))

The substring stuff was just to ensure that my stringbuilder did, in fact, have the end tag in it...

For helping change my syntax to this to get my stringbuilder working:

Dim ReceivedLength As Integer = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
clientdata = System.Text.Encoding.ASCII.GetString(bytes, 0, ReceivedLength)

and for showing me the GetTotalMemory function to help debug things... PaulHews solved the problelm!

Thanks All, much appreciated!
Mike
0
 
LVL 38

Expert Comment

by:PaulHews
ID: 12650315
Happy to help Mike.  Just remember that the debug statements will go away when you build your release version.

>I'd image that if some process needed it, that memory could be utilized.

A qualified yes to that.
http://weblogs.asp.net/pwilson/archive/2004/02/14/73033.aspx

"Garbage collection occurs rather frequently, which reduces the committed bytes,
but that does NOT mean the memory is given back to the OS for other processes.
Instead, the reserved memory associated with the process stays with the process,
which is why my earlier example often found it acceptable to rise even to 300MB.
Apparently, the reserved memory is only given back to the OS to use for other
processes when the overall available system memory drops to this 32MB threshold."

The whole blog entry has a lot of interesting information on the memory management issue.  Worth reading in full.  
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction When many people think of the WebBrowser (http://msdn.microsoft.com/en-us/library/2te2y1x6%28v=VS.85%29.aspx) control, they immediately think of a control which allows the viewing and navigation of web pages. While this is true, it's a…
1.0 - Introduction Converting Visual Basic 6.0 (VB6) to Visual Basic 2008+ (VB.NET). If ever there was a subject full of murkiness and bad decisions, it is this one!   The first problem seems to be that people considering this task of converting…
The Relationships Diagram is a good way to get an overall view of what a database is keeping track of. It is also where relationships are defined. A relationship specifies how two tables connect to each other. As you build tables in Microsoft Ac…
How can you see what you are working on when you want to see it while you to save a copy? Add a "Save As" icon to the Quick Access Toolbar, or QAT. That way, when you save a copy of a query, form, report, or other object you are modifying, you…
Suggested Courses

578 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question