Solved

Large Memory Usage - StringBuffer? Leak?

Posted on 2002-04-04
17
550 Views
Last Modified: 2013-12-29
I have a program that is parsing a large file ( ~1 MB )
It parses it into a data structure.
The memory (heap usage) after it is loaded is ~10MB.

I have looked at the code using OptimizeIt and about 5MB of the heap is taken up by allocations to char[] which, when you trace the allocation, all come from StringBuffer.

At certain parts of the program, I'll read some of the file and do something like

readText()
{
    StringBuffer buf = new StringBuffer();

    I'll then append() to buf

    return buf.toString();
}

I would have thought that this allocation would be garbage collected and come out of the heap usage but it doesn't seem to. Is this right?

Running garbage collection with System.gc() makes no difference.

Are these allocations hanging about in the heap?
How do I free the memory?

Help!
0
Comment
Question by:slinky
  • 6
  • 4
  • 3
  • +2
17 Comments
 
LVL 1

Author Comment

by:slinky
Comment Utility
It's probably worth saying that this all happens at least 18000 times...
0
 
LVL 35

Accepted Solution

by:
girionis earned 200 total points
Comment Utility
 Calling System.gc() does not necessarily mean that the garbage collector will run. According to the JavaDoc: ?Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all unused objects.?.

  As you can clearly see it states that it has made a best effort to reclaim space not that it has reclaimed it. It is unknown when the garbage collector will actually run and collect the object (it might not even run once during an application¢s lifetime!). Some implementations of the VM run the gc only if more memory is needed.

Besides running the gc manually is a quite heavy and expensive operation (in terms of system resources) since the VM has to pause everything it is doing, run the gc and restart everything it was doing. The only way I would run the gc would be if I were getting an OutOfMemoryError. Otherwise the VM itself will find some time to collect it if it is no longer needed.

  With regards to your question these allocations will hang in heap until the gc grabs them. Setting them to null after you no longer need them will help. Also lazy instantiation will help reducing memory usage. Of course there might be design flaws or memory leaks in your code. If you intend to run your application on a 24/7 basis then you might indeed encounter problems. From personal experience though, I have done large files processing on a 24/7 basis and never had any memory problems.

  You might also want to take a look here: http://www-106.ibm.com/developerworks/library/j-leaks/index.html , http://www-106.ibm.com/developerworks/library/tip-heap-size.html , http://www.javaworld.com/javaworld/jw-01-2002/jw-0111-hotspotgc.html , http://www.javaworld.com/javaworld/javaqa/1999-08/04-qa-leaks.html .

  Hope it helps.
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
> Are these allocations hanging about in the heap?

Depends whether references to the String returned by readText are maintained, and once there are no references then whether gc has got around to freeing them.

> How do I free the memory?

You don't, that's the job of the garbage collector.

0
 
LVL 1

Author Comment

by:slinky
Comment Utility
Thanks for comments so far...

I've managed to save some memory by replacing the
buf.toString() with buf.substring(0, buf.length())

I think the problem is that what is returned is maintained in the data structure and therefore the memory hangs about.

If I pass the returned String into the creation of a new Object (that has it's own internal String to store it in), do I end up with two copies of the memory?
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
> do I end up with two copies of the memory?

no
0
 
LVL 1

Author Comment

by:slinky
Comment Utility
Looks like I'm dealing with inefficient storage rather than leaking as such. I've not got a 1MB file that is getting stored in about 9MB of heap. That makes it, on average about 500 bytes per record (which isn't bad I suppose, but records are only about 30-50 bytes in the file).

Any ideas of more efficient storage? ByteArrays rather than Strings?

I'm going to use a more radical solution for now...have less data :-)

Thanks again.
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
yes byte arrays could definitely be more efficient than strings. Remember Java stores all it's strings as Unicode.
0
 
LVL 16

Expert Comment

by:heyhey_
Comment Utility
String obejcts constructed from StringBuffer keep reference to StringBuffer's byte array, which may be pretty big.

    public String (StringBuffer buffer) {
     synchronized(buffer) {
         buffer.setShared();
         this.value = buffer.getValue();
         this.offset = 0;
         this.count = buffer.length();
     }
    }

> Any ideas of more efficient storage? ByteArrays rather than Strings?

byte arrays should work better :)
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
Are using the empty constructor for StringBuffer? If not, use the constructor that specifies the original buffer size. Make the allocation 2Mb to be on the safe side. Let us know what happens.
0
 
LVL 1

Author Comment

by:slinky
Comment Utility
I'm going to have a go with ByteArrays, might take a while, others things are getting in the way...
0
 
LVL 16

Expert Comment

by:heyhey_
Comment Utility
hmm ... why exactly did you accept girionis comment ?
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
 heyhey if you are really concerned about the points then I am asking slinky to subtract them from me and add them to you. I am here to help (and be helped by) people not to gather points and proove I am better than somebody else.

  If slinky agrees then lets ask a moderator to do the change.
0
 
LVL 16

Expert Comment

by:heyhey_
Comment Utility
I do not care about points any more - I just do not think that your comment helped to detect or solve the problem at all.

nothing personal :)
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
 First of all I have to thank slinky for accepting my comment.

>I would have thought that this allocation would be garbage collected and come out of the heap usage
but it doesn't seem to. Is this right?

>Are these allocations hanging about in the heap?

>How do I free the memory?

  I can clearly see three questions above and I *think* I have helped him with all three of them. Of course I did not give the most optimal solution, that's why I wasn't rewarded with an A grade.
0
 
LVL 1

Author Comment

by:slinky
Comment Utility
Now, now, calm down everybody...

I've got thousands of points to throw about, I'll ask another question just for heyhey....
0
 
LVL 16

Expert Comment

by:heyhey_
Comment Utility
> I'll ask another question just for heyhey....

no no, I do not care about points - I just wanted to know why you chose this comment.

I won't accept additional points for this question.
0
 
LVL 1

Author Comment

by:slinky
Comment Utility
I suppose I chose it because in relation to what I already knew about the problem and what I didn't, it told me more about things I didn't already know. Also some of the links provided were useful and lead me to find more useful stuff.
I suppose the answer was perhaps not appropriate for everyone but it was OK for me. I'd rather give the points on that basis than just leave the question open. Your answer was useful too but it's not easy on EE to Accept or partially accept two answers...
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now