[Webinar] Streamline your web hosting managementRegister Today

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 589
  • Last Modified:

Large Memory Usage - StringBuffer? Leak?

I have a program that is parsing a large file ( ~1 MB )
It parses it into a data structure.
The memory (heap usage) after it is loaded is ~10MB.

I have looked at the code using OptimizeIt and about 5MB of the heap is taken up by allocations to char[] which, when you trace the allocation, all come from StringBuffer.

At certain parts of the program, I'll read some of the file and do something like

readText()
{
    StringBuffer buf = new StringBuffer();

    I'll then append() to buf

    return buf.toString();
}

I would have thought that this allocation would be garbage collected and come out of the heap usage but it doesn't seem to. Is this right?

Running garbage collection with System.gc() makes no difference.

Are these allocations hanging about in the heap?
How do I free the memory?

Help!
0
slinky
Asked:
slinky
  • 6
  • 4
  • 3
  • +2
1 Solution
 
slinkyAuthor Commented:
It's probably worth saying that this all happens at least 18000 times...
0
 
girionisCommented:
 Calling System.gc() does not necessarily mean that the garbage collector will run. According to the JavaDoc: ?Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all unused objects.?.

  As you can clearly see it states that it has made a best effort to reclaim space not that it has reclaimed it. It is unknown when the garbage collector will actually run and collect the object (it might not even run once during an application¢s lifetime!). Some implementations of the VM run the gc only if more memory is needed.

Besides running the gc manually is a quite heavy and expensive operation (in terms of system resources) since the VM has to pause everything it is doing, run the gc and restart everything it was doing. The only way I would run the gc would be if I were getting an OutOfMemoryError. Otherwise the VM itself will find some time to collect it if it is no longer needed.

  With regards to your question these allocations will hang in heap until the gc grabs them. Setting them to null after you no longer need them will help. Also lazy instantiation will help reducing memory usage. Of course there might be design flaws or memory leaks in your code. If you intend to run your application on a 24/7 basis then you might indeed encounter problems. From personal experience though, I have done large files processing on a 24/7 basis and never had any memory problems.

  You might also want to take a look here: http://www-106.ibm.com/developerworks/library/j-leaks/index.html , http://www-106.ibm.com/developerworks/library/tip-heap-size.html , http://www.javaworld.com/javaworld/jw-01-2002/jw-0111-hotspotgc.html , http://www.javaworld.com/javaworld/javaqa/1999-08/04-qa-leaks.html .

  Hope it helps.
0
 
objectsCommented:
> Are these allocations hanging about in the heap?

Depends whether references to the String returned by readText are maintained, and once there are no references then whether gc has got around to freeing them.

> How do I free the memory?

You don't, that's the job of the garbage collector.

0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
slinkyAuthor Commented:
Thanks for comments so far...

I've managed to save some memory by replacing the
buf.toString() with buf.substring(0, buf.length())

I think the problem is that what is returned is maintained in the data structure and therefore the memory hangs about.

If I pass the returned String into the creation of a new Object (that has it's own internal String to store it in), do I end up with two copies of the memory?
0
 
objectsCommented:
> do I end up with two copies of the memory?

no
0
 
slinkyAuthor Commented:
Looks like I'm dealing with inefficient storage rather than leaking as such. I've not got a 1MB file that is getting stored in about 9MB of heap. That makes it, on average about 500 bytes per record (which isn't bad I suppose, but records are only about 30-50 bytes in the file).

Any ideas of more efficient storage? ByteArrays rather than Strings?

I'm going to use a more radical solution for now...have less data :-)

Thanks again.
0
 
objectsCommented:
yes byte arrays could definitely be more efficient than strings. Remember Java stores all it's strings as Unicode.
0
 
heyhey_Commented:
String obejcts constructed from StringBuffer keep reference to StringBuffer's byte array, which may be pretty big.

    public String (StringBuffer buffer) {
     synchronized(buffer) {
         buffer.setShared();
         this.value = buffer.getValue();
         this.offset = 0;
         this.count = buffer.length();
     }
    }

> Any ideas of more efficient storage? ByteArrays rather than Strings?

byte arrays should work better :)
0
 
CEHJCommented:
Are using the empty constructor for StringBuffer? If not, use the constructor that specifies the original buffer size. Make the allocation 2Mb to be on the safe side. Let us know what happens.
0
 
slinkyAuthor Commented:
I'm going to have a go with ByteArrays, might take a while, others things are getting in the way...
0
 
heyhey_Commented:
hmm ... why exactly did you accept girionis comment ?
0
 
girionisCommented:
 heyhey if you are really concerned about the points then I am asking slinky to subtract them from me and add them to you. I am here to help (and be helped by) people not to gather points and proove I am better than somebody else.

  If slinky agrees then lets ask a moderator to do the change.
0
 
heyhey_Commented:
I do not care about points any more - I just do not think that your comment helped to detect or solve the problem at all.

nothing personal :)
0
 
girionisCommented:
 First of all I have to thank slinky for accepting my comment.

>I would have thought that this allocation would be garbage collected and come out of the heap usage
but it doesn't seem to. Is this right?

>Are these allocations hanging about in the heap?

>How do I free the memory?

  I can clearly see three questions above and I *think* I have helped him with all three of them. Of course I did not give the most optimal solution, that's why I wasn't rewarded with an A grade.
0
 
slinkyAuthor Commented:
Now, now, calm down everybody...

I've got thousands of points to throw about, I'll ask another question just for heyhey....
0
 
heyhey_Commented:
> I'll ask another question just for heyhey....

no no, I do not care about points - I just wanted to know why you chose this comment.

I won't accept additional points for this question.
0
 
slinkyAuthor Commented:
I suppose I chose it because in relation to what I already knew about the problem and what I didn't, it told me more about things I didn't already know. Also some of the links provided were useful and lead me to find more useful stuff.
I suppose the answer was perhaps not appropriate for everyone but it was OK for me. I'd rather give the points on that basis than just leave the question open. Your answer was useful too but it's not easy on EE to Accept or partially accept two answers...
0

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 6
  • 4
  • 3
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now