Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Posted on 2009-05-20
16
Medium Priority
?
1,340 Views
Last Modified: 2012-05-07
Hello,

I get the exception in the title in an application I have written.
It seems to be caused by a large StringBuffer. The settings I
use are -Xms1024m -Xmx1512m and
-XX:+AggressiveOpts -XX:+UseParallelGC.

The machine is running Kubuntu 9.04 64bit and the physical
memory is 2g.

Any ideas on how I could overcome the problem?

Regards,

Nick
0
Comment
Question by:ntzanos
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
  • 4
  • +1
16 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 24430778
>>Any ideas on how I could overcome the problem?

Probably by using the StringBuffer more carefully - they have big gotchas. Please post code
0
 
LVL 92

Expert Comment

by:objects
ID: 24430798
try running it in a 32 bit jvm

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24430867
though there are issues in frequent StringBuffer creation and setting size, mostly the issue will be with String copies created out of the buffer and memory leaks. first try running profiler and see if there are leaks. On the other hand, also check if it is really required to have that big string in memory. check the possibility of reading partial string to memory through stream rather loading the entire string into memory. also consider increasing the heap!
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:ntzanos
ID: 24430989
The following bit of code is the problematic one.
The error is thrown for the line where it says sb.append(line)

Could you suggest some more elaborate solution on an alternative? I am a bit lost at the moment

Many thanks

Nick
String sFile = utils.loadFileToStringWithNewlines(sInputFile);
		ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 
		for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
            StringBuffer sbDef = new StringBuffer();
            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
			hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
		// Start of code that prepares for 10-fold cross validation
 
		int giSize = alLines.size();
		int tenth = giSize / 10;
		LineNumberReader lnr = null;
		try {
			lnr = new LineNumberReader(new FileReader(sInputFile));
		} catch (FileNotFoundException e) {
			System.err.println("File not found..");
		}
 
		StringBuffer sb = new StringBuffer();
		HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
		int counter = 1;
		String line = "";
		try {
			while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
			//while((String line = lnr.readLine()) != null) {
				line = lnr.readLine();
				if (((lnr.getLineNumber() + 1) % tenth) == 0) {
					giChunks.put(counter++, sb.toString()); // Also increment the counter.
					sb = new StringBuffer();
				}
				sb.append(line);
			}
        } catch (IOException e) {
			// put the final tenth of the file into the HashMap
			giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
		}

Open in new window

0
 
LVL 92

Expert Comment

by:objects
ID: 24431011
>                                         sb = new StringBuffer();

you don't need to create a new one each time, just clear it

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431047
>>sb = new StringBuffer();

replace with

sb.delete( 0, sb.length() ) ;
0
 

Author Comment

by:ntzanos
ID: 24431237
I changed it (sb.delete(0,sb.length()) and it still crushes, but it crushes in 1min and 40 sec instead of the 40 secs that it took initially

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431282
That will help but a much worse use of memory is the following:

>>
        String sFile = utils.loadFileToStringWithNewlines(sInputFile);
        ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
>>

You create a massive String then proceed to split that String with a regex in order to put it into a List, when you could have simply read it into the List in the first place
0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431316
can you try this,

String sFile = utils.loadFileToStringWithNewlines(sInputFile);
            ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 StringBuffer sb = new StringBuffer();
StringBuffer sbDef = new StringBuffer();

            for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
if( sbDef.length() > 0 ) sbDef.delete( 0, sbDef.length() ) ;

            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
                  hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
            // Start of code that prepares for 10-fold cross validation
 
            int giSize = alLines.size();
            int tenth = giSize / 10;
            LineNumberReader lnr = null;
            try {
                  lnr = new LineNumberReader(new FileReader(sInputFile));
            } catch (FileNotFoundException e) {
                  System.err.println("File not found..");
            }
 
            
                                if( sb.length() > 0 ) sb.delete( 0, sb.length() ) ;
            HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
            int counter = 1;
            String line = "";
            try {
                  while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
                  //while((String line = lnr.readLine()) != null) {
                        line = lnr.readLine();
                        if (((lnr.getLineNumber() + 1) % tenth) == 0) {
                              giChunks.put(counter++, sb.toString()); // Also increment the counter.
                              sb.delete( 0, sb.length() ) ;
                        }
                        sb.append(line);
                  }
        } catch (IOException e) {
                  // put the final tenth of the file into the HashMap
                  giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
            }
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431337
... in fact the finer details are even worse, but the big picture is bad enough
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24432337
Out of interest - what size is that input file?
0
 
LVL 92

Expert Comment

by:objects
ID: 24437336
use the following to read the lines on the file into a list

http://helpdesk.objects.com.au/java/how-do-i-read-a-text-file-line-by-line-into-a-list

you can adapt it to filter your lines as required

0
 

Author Comment

by:ntzanos
ID: 24443763
The size of the file is about 1mb.
I will try again and let you all know. Maybe I need to reimplement the code because it was a mixture of my own code, with code from another person, so it would be better if I start it from scratch.

I will keep you informed.

Nick
0
 
LVL 92

Accepted Solution

by:
objects earned 1500 total points
ID: 24446786
yes it could definititely be improved.  Have a look at the link I posted above which will allow you to directly read the file into a list and should remove the need for split() and regexp

0
 

Author Closing Comment

by:ntzanos
ID: 31583464
Thanks for all the help. I finally decided that it would be best to reimplement the code. And now it goes much faster (using advice from all people here). Thanks a lot
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24467785
ntzanos, can you tell us why you accepted that answer please? Future searchers need to know
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Viewers learn about the scanner class in this video and are introduced to receiving user input for their programs. Additionally, objects, conditional statements, and loops are used to help reinforce the concepts. Introduce Scanner class: Importing…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
Suggested Courses

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question