Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Large Memory Usage - StringBuffer? Leak?

Posted on 2002-04-04
17
Medium Priority
?
583 Views
Last Modified: 2013-12-29
I have a program that is parsing a large file ( ~1 MB )
It parses it into a data structure.
The memory (heap usage) after it is loaded is ~10MB.

I have looked at the code using OptimizeIt and about 5MB of the heap is taken up by allocations to char[] which, when you trace the allocation, all come from StringBuffer.

At certain parts of the program, I'll read some of the file and do something like

readText()
{
    StringBuffer buf = new StringBuffer();

    I'll then append() to buf

    return buf.toString();
}

I would have thought that this allocation would be garbage collected and come out of the heap usage but it doesn't seem to. Is this right?

Running garbage collection with System.gc() makes no difference.

Are these allocations hanging about in the heap?
How do I free the memory?

Help!
0
Comment
Question by:slinky
  • 6
  • 4
  • 3
  • +2
17 Comments
 
LVL 1

Author Comment

by:slinky
ID: 6919358
It's probably worth saying that this all happens at least 18000 times...
0
 
LVL 35

Accepted Solution

by:
girionis earned 600 total points
ID: 6919552
 Calling System.gc() does not necessarily mean that the garbage collector will run. According to the JavaDoc: ?Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all unused objects.?.

  As you can clearly see it states that it has made a best effort to reclaim space not that it has reclaimed it. It is unknown when the garbage collector will actually run and collect the object (it might not even run once during an application¢s lifetime!). Some implementations of the VM run the gc only if more memory is needed.

Besides running the gc manually is a quite heavy and expensive operation (in terms of system resources) since the VM has to pause everything it is doing, run the gc and restart everything it was doing. The only way I would run the gc would be if I were getting an OutOfMemoryError. Otherwise the VM itself will find some time to collect it if it is no longer needed.

  With regards to your question these allocations will hang in heap until the gc grabs them. Setting them to null after you no longer need them will help. Also lazy instantiation will help reducing memory usage. Of course there might be design flaws or memory leaks in your code. If you intend to run your application on a 24/7 basis then you might indeed encounter problems. From personal experience though, I have done large files processing on a 24/7 basis and never had any memory problems.

  You might also want to take a look here: http://www-106.ibm.com/developerworks/library/j-leaks/index.html , http://www-106.ibm.com/developerworks/library/tip-heap-size.html , http://www.javaworld.com/javaworld/jw-01-2002/jw-0111-hotspotgc.html , http://www.javaworld.com/javaworld/javaqa/1999-08/04-qa-leaks.html .

  Hope it helps.
0
 
LVL 92

Expert Comment

by:objects
ID: 6920066
> Are these allocations hanging about in the heap?

Depends whether references to the String returned by readText are maintained, and once there are no references then whether gc has got around to freeing them.

> How do I free the memory?

You don't, that's the job of the garbage collector.

0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:slinky
ID: 6920260
Thanks for comments so far...

I've managed to save some memory by replacing the
buf.toString() with buf.substring(0, buf.length())

I think the problem is that what is returned is maintained in the data structure and therefore the memory hangs about.

If I pass the returned String into the creation of a new Object (that has it's own internal String to store it in), do I end up with two copies of the memory?
0
 
LVL 92

Expert Comment

by:objects
ID: 6920281
> do I end up with two copies of the memory?

no
0
 
LVL 1

Author Comment

by:slinky
ID: 6920303
Looks like I'm dealing with inefficient storage rather than leaking as such. I've not got a 1MB file that is getting stored in about 9MB of heap. That makes it, on average about 500 bytes per record (which isn't bad I suppose, but records are only about 30-50 bytes in the file).

Any ideas of more efficient storage? ByteArrays rather than Strings?

I'm going to use a more radical solution for now...have less data :-)

Thanks again.
0
 
LVL 92

Expert Comment

by:objects
ID: 6920330
yes byte arrays could definitely be more efficient than strings. Remember Java stores all it's strings as Unicode.
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6920383
String obejcts constructed from StringBuffer keep reference to StringBuffer's byte array, which may be pretty big.

    public String (StringBuffer buffer) {
     synchronized(buffer) {
         buffer.setShared();
         this.value = buffer.getValue();
         this.offset = 0;
         this.count = buffer.length();
     }
    }

> Any ideas of more efficient storage? ByteArrays rather than Strings?

byte arrays should work better :)
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 6920600
Are using the empty constructor for StringBuffer? If not, use the constructor that specifies the original buffer size. Make the allocation 2Mb to be on the safe side. Let us know what happens.
0
 
LVL 1

Author Comment

by:slinky
ID: 6920632
I'm going to have a go with ByteArrays, might take a while, others things are getting in the way...
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996265
hmm ... why exactly did you accept girionis comment ?
0
 
LVL 35

Expert Comment

by:girionis
ID: 6996313
 heyhey if you are really concerned about the points then I am asking slinky to subtract them from me and add them to you. I am here to help (and be helped by) people not to gather points and proove I am better than somebody else.

  If slinky agrees then lets ask a moderator to do the change.
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996329
I do not care about points any more - I just do not think that your comment helped to detect or solve the problem at all.

nothing personal :)
0
 
LVL 35

Expert Comment

by:girionis
ID: 6996357
 First of all I have to thank slinky for accepting my comment.

>I would have thought that this allocation would be garbage collected and come out of the heap usage
but it doesn't seem to. Is this right?

>Are these allocations hanging about in the heap?

>How do I free the memory?

  I can clearly see three questions above and I *think* I have helped him with all three of them. Of course I did not give the most optimal solution, that's why I wasn't rewarded with an A grade.
0
 
LVL 1

Author Comment

by:slinky
ID: 6996415
Now, now, calm down everybody...

I've got thousands of points to throw about, I'll ask another question just for heyhey....
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996472
> I'll ask another question just for heyhey....

no no, I do not care about points - I just wanted to know why you chose this comment.

I won't accept additional points for this question.
0
 
LVL 1

Author Comment

by:slinky
ID: 6996756
I suppose I chose it because in relation to what I already knew about the problem and what I didn't, it told me more about things I didn't already know. Also some of the links provided were useful and lead me to find more useful stuff.
I suppose the answer was perhaps not appropriate for everyone but it was OK for me. I'd rather give the points on that basis than just leave the question open. Your answer was useful too but it's not easy on EE to Accept or partially accept two answers...
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

An old method to applying the Singleton pattern in your Java code is to check if a static instance, defined in the same class that needs to be instantiated once and only once, is null and then create a new instance; otherwise, the pre-existing insta…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
This video teaches viewers about errors in exception handling.
Suggested Courses

886 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question