Solved

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Posted on 2009-05-20
16
1,335 Views
Last Modified: 2012-05-07
Hello,

I get the exception in the title in an application I have written.
It seems to be caused by a large StringBuffer. The settings I
use are -Xms1024m -Xmx1512m and
-XX:+AggressiveOpts -XX:+UseParallelGC.

The machine is running Kubuntu 9.04 64bit and the physical
memory is 2g.

Any ideas on how I could overcome the problem?

Regards,

Nick
0
Comment
Question by:ntzanos
  • 5
  • 4
  • 4
  • +1
16 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 24430778
>>Any ideas on how I could overcome the problem?

Probably by using the StringBuffer more carefully - they have big gotchas. Please post code
0
 
LVL 92

Expert Comment

by:objects
ID: 24430798
try running it in a 32 bit jvm

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24430867
though there are issues in frequent StringBuffer creation and setting size, mostly the issue will be with String copies created out of the buffer and memory leaks. first try running profiler and see if there are leaks. On the other hand, also check if it is really required to have that big string in memory. check the possibility of reading partial string to memory through stream rather loading the entire string into memory. also consider increasing the heap!
0
Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

 

Author Comment

by:ntzanos
ID: 24430989
The following bit of code is the problematic one.
The error is thrown for the line where it says sb.append(line)

Could you suggest some more elaborate solution on an alternative? I am a bit lost at the moment

Many thanks

Nick
String sFile = utils.loadFileToStringWithNewlines(sInputFile);
		ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 
		for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
            StringBuffer sbDef = new StringBuffer();
            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
			hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
		// Start of code that prepares for 10-fold cross validation
 
		int giSize = alLines.size();
		int tenth = giSize / 10;
		LineNumberReader lnr = null;
		try {
			lnr = new LineNumberReader(new FileReader(sInputFile));
		} catch (FileNotFoundException e) {
			System.err.println("File not found..");
		}
 
		StringBuffer sb = new StringBuffer();
		HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
		int counter = 1;
		String line = "";
		try {
			while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
			//while((String line = lnr.readLine()) != null) {
				line = lnr.readLine();
				if (((lnr.getLineNumber() + 1) % tenth) == 0) {
					giChunks.put(counter++, sb.toString()); // Also increment the counter.
					sb = new StringBuffer();
				}
				sb.append(line);
			}
        } catch (IOException e) {
			// put the final tenth of the file into the HashMap
			giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
		}

Open in new window

0
 
LVL 92

Expert Comment

by:objects
ID: 24431011
>                                         sb = new StringBuffer();

you don't need to create a new one each time, just clear it

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431047
>>sb = new StringBuffer();

replace with

sb.delete( 0, sb.length() ) ;
0
 

Author Comment

by:ntzanos
ID: 24431237
I changed it (sb.delete(0,sb.length()) and it still crushes, but it crushes in 1min and 40 sec instead of the 40 secs that it took initially

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431282
That will help but a much worse use of memory is the following:

>>
        String sFile = utils.loadFileToStringWithNewlines(sInputFile);
        ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
>>

You create a massive String then proceed to split that String with a regex in order to put it into a List, when you could have simply read it into the List in the first place
0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431316
can you try this,

String sFile = utils.loadFileToStringWithNewlines(sInputFile);
            ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 StringBuffer sb = new StringBuffer();
StringBuffer sbDef = new StringBuffer();

            for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
if( sbDef.length() > 0 ) sbDef.delete( 0, sbDef.length() ) ;

            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
                  hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
            // Start of code that prepares for 10-fold cross validation
 
            int giSize = alLines.size();
            int tenth = giSize / 10;
            LineNumberReader lnr = null;
            try {
                  lnr = new LineNumberReader(new FileReader(sInputFile));
            } catch (FileNotFoundException e) {
                  System.err.println("File not found..");
            }
 
            
                                if( sb.length() > 0 ) sb.delete( 0, sb.length() ) ;
            HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
            int counter = 1;
            String line = "";
            try {
                  while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
                  //while((String line = lnr.readLine()) != null) {
                        line = lnr.readLine();
                        if (((lnr.getLineNumber() + 1) % tenth) == 0) {
                              giChunks.put(counter++, sb.toString()); // Also increment the counter.
                              sb.delete( 0, sb.length() ) ;
                        }
                        sb.append(line);
                  }
        } catch (IOException e) {
                  // put the final tenth of the file into the HashMap
                  giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
            }
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431337
... in fact the finer details are even worse, but the big picture is bad enough
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24432337
Out of interest - what size is that input file?
0
 
LVL 92

Expert Comment

by:objects
ID: 24437336
use the following to read the lines on the file into a list

http://helpdesk.objects.com.au/java/how-do-i-read-a-text-file-line-by-line-into-a-list

you can adapt it to filter your lines as required

0
 

Author Comment

by:ntzanos
ID: 24443763
The size of the file is about 1mb.
I will try again and let you all know. Maybe I need to reimplement the code because it was a mixture of my own code, with code from another person, so it would be better if I start it from scratch.

I will keep you informed.

Nick
0
 
LVL 92

Accepted Solution

by:
objects earned 500 total points
ID: 24446786
yes it could definititely be improved.  Have a look at the link I posted above which will allow you to directly read the file into a list and should remove the need for split() and regexp

0
 

Author Closing Comment

by:ntzanos
ID: 31583464
Thanks for all the help. I finally decided that it would be best to reimplement the code. And now it goes much faster (using advice from all people here). Thanks a lot
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24467785
ntzanos, can you tell us why you accepted that answer please? Future searchers need to know
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Java Jpanels and Jframe 8 32
Cipher Configuration on Apache HTTPD 4 62
Oracle SQL syntax check  without executing 6 50
jsp login check 12 29
INTRODUCTION Working with files is a moderately common task in Java.  For most projects hard coding the file names, using parameters in configuration files, or using command-line arguments is sufficient.   However, when your application has vi…
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question