Solved

Large Memory Usage - StringBuffer? Leak?

Posted on 2002-04-04
17
557 Views
Last Modified: 2013-12-29
I have a program that is parsing a large file ( ~1 MB )
It parses it into a data structure.
The memory (heap usage) after it is loaded is ~10MB.

I have looked at the code using OptimizeIt and about 5MB of the heap is taken up by allocations to char[] which, when you trace the allocation, all come from StringBuffer.

At certain parts of the program, I'll read some of the file and do something like

readText()
{
    StringBuffer buf = new StringBuffer();

    I'll then append() to buf

    return buf.toString();
}

I would have thought that this allocation would be garbage collected and come out of the heap usage but it doesn't seem to. Is this right?

Running garbage collection with System.gc() makes no difference.

Are these allocations hanging about in the heap?
How do I free the memory?

Help!
0
Comment
Question by:slinky
  • 6
  • 4
  • 3
  • +2
17 Comments
 
LVL 1

Author Comment

by:slinky
ID: 6919358
It's probably worth saying that this all happens at least 18000 times...
0
 
LVL 35

Accepted Solution

by:
girionis earned 200 total points
ID: 6919552
 Calling System.gc() does not necessarily mean that the garbage collector will run. According to the JavaDoc: ?Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all unused objects.?.

  As you can clearly see it states that it has made a best effort to reclaim space not that it has reclaimed it. It is unknown when the garbage collector will actually run and collect the object (it might not even run once during an application¢s lifetime!). Some implementations of the VM run the gc only if more memory is needed.

Besides running the gc manually is a quite heavy and expensive operation (in terms of system resources) since the VM has to pause everything it is doing, run the gc and restart everything it was doing. The only way I would run the gc would be if I were getting an OutOfMemoryError. Otherwise the VM itself will find some time to collect it if it is no longer needed.

  With regards to your question these allocations will hang in heap until the gc grabs them. Setting them to null after you no longer need them will help. Also lazy instantiation will help reducing memory usage. Of course there might be design flaws or memory leaks in your code. If you intend to run your application on a 24/7 basis then you might indeed encounter problems. From personal experience though, I have done large files processing on a 24/7 basis and never had any memory problems.

  You might also want to take a look here: http://www-106.ibm.com/developerworks/library/j-leaks/index.html , http://www-106.ibm.com/developerworks/library/tip-heap-size.html , http://www.javaworld.com/javaworld/jw-01-2002/jw-0111-hotspotgc.html , http://www.javaworld.com/javaworld/javaqa/1999-08/04-qa-leaks.html .

  Hope it helps.
0
 
LVL 92

Expert Comment

by:objects
ID: 6920066
> Are these allocations hanging about in the heap?

Depends whether references to the String returned by readText are maintained, and once there are no references then whether gc has got around to freeing them.

> How do I free the memory?

You don't, that's the job of the garbage collector.

0
ScreenConnect 6.0 Free Trial

Discover new time-saving features in one game-changing release, ScreenConnect 6.0, based on partner feedback. New features include a redesigned UI, app configurations and chat acknowledgement to improve customer engagement!

 
LVL 1

Author Comment

by:slinky
ID: 6920260
Thanks for comments so far...

I've managed to save some memory by replacing the
buf.toString() with buf.substring(0, buf.length())

I think the problem is that what is returned is maintained in the data structure and therefore the memory hangs about.

If I pass the returned String into the creation of a new Object (that has it's own internal String to store it in), do I end up with two copies of the memory?
0
 
LVL 92

Expert Comment

by:objects
ID: 6920281
> do I end up with two copies of the memory?

no
0
 
LVL 1

Author Comment

by:slinky
ID: 6920303
Looks like I'm dealing with inefficient storage rather than leaking as such. I've not got a 1MB file that is getting stored in about 9MB of heap. That makes it, on average about 500 bytes per record (which isn't bad I suppose, but records are only about 30-50 bytes in the file).

Any ideas of more efficient storage? ByteArrays rather than Strings?

I'm going to use a more radical solution for now...have less data :-)

Thanks again.
0
 
LVL 92

Expert Comment

by:objects
ID: 6920330
yes byte arrays could definitely be more efficient than strings. Remember Java stores all it's strings as Unicode.
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6920383
String obejcts constructed from StringBuffer keep reference to StringBuffer's byte array, which may be pretty big.

    public String (StringBuffer buffer) {
     synchronized(buffer) {
         buffer.setShared();
         this.value = buffer.getValue();
         this.offset = 0;
         this.count = buffer.length();
     }
    }

> Any ideas of more efficient storage? ByteArrays rather than Strings?

byte arrays should work better :)
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 6920600
Are using the empty constructor for StringBuffer? If not, use the constructor that specifies the original buffer size. Make the allocation 2Mb to be on the safe side. Let us know what happens.
0
 
LVL 1

Author Comment

by:slinky
ID: 6920632
I'm going to have a go with ByteArrays, might take a while, others things are getting in the way...
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996265
hmm ... why exactly did you accept girionis comment ?
0
 
LVL 35

Expert Comment

by:girionis
ID: 6996313
 heyhey if you are really concerned about the points then I am asking slinky to subtract them from me and add them to you. I am here to help (and be helped by) people not to gather points and proove I am better than somebody else.

  If slinky agrees then lets ask a moderator to do the change.
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996329
I do not care about points any more - I just do not think that your comment helped to detect or solve the problem at all.

nothing personal :)
0
 
LVL 35

Expert Comment

by:girionis
ID: 6996357
 First of all I have to thank slinky for accepting my comment.

>I would have thought that this allocation would be garbage collected and come out of the heap usage
but it doesn't seem to. Is this right?

>Are these allocations hanging about in the heap?

>How do I free the memory?

  I can clearly see three questions above and I *think* I have helped him with all three of them. Of course I did not give the most optimal solution, that's why I wasn't rewarded with an A grade.
0
 
LVL 1

Author Comment

by:slinky
ID: 6996415
Now, now, calm down everybody...

I've got thousands of points to throw about, I'll ask another question just for heyhey....
0
 
LVL 16

Expert Comment

by:heyhey_
ID: 6996472
> I'll ask another question just for heyhey....

no no, I do not care about points - I just wanted to know why you chose this comment.

I won't accept additional points for this question.
0
 
LVL 1

Author Comment

by:slinky
ID: 6996756
I suppose I chose it because in relation to what I already knew about the problem and what I didn't, it told me more about things I didn't already know. Also some of the links provided were useful and lead me to find more useful stuff.
I suppose the answer was perhaps not appropriate for everyone but it was OK for me. I'd rather give the points on that basis than just leave the question open. Your answer was useful too but it's not easy on EE to Accept or partially accept two answers...
0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
ArrayIndexOutOfBoundException 9 81
java 8 lambda expresssions exception handling 3 89
java stored proc example 9 29
login form jsp example 2 25
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
By the end of 1980s, object oriented programming using languages like C++, Simula69 and ObjectPascal gained momentum. It looked like programmers finally found the perfect language. C++ successfully combined the object oriented principles of Simula w…
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question