Solved

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Posted on 2009-05-20
16
1,328 Views
Last Modified: 2012-05-07
Hello,

I get the exception in the title in an application I have written.
It seems to be caused by a large StringBuffer. The settings I
use are -Xms1024m -Xmx1512m and
-XX:+AggressiveOpts -XX:+UseParallelGC.

The machine is running Kubuntu 9.04 64bit and the physical
memory is 2g.

Any ideas on how I could overcome the problem?

Regards,

Nick
0
Comment
Question by:ntzanos
  • 5
  • 4
  • 4
  • +1
16 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 24430778
>>Any ideas on how I could overcome the problem?

Probably by using the StringBuffer more carefully - they have big gotchas. Please post code
0
 
LVL 92

Expert Comment

by:objects
ID: 24430798
try running it in a 32 bit jvm

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24430867
though there are issues in frequent StringBuffer creation and setting size, mostly the issue will be with String copies created out of the buffer and memory leaks. first try running profiler and see if there are leaks. On the other hand, also check if it is really required to have that big string in memory. check the possibility of reading partial string to memory through stream rather loading the entire string into memory. also consider increasing the heap!
0
 

Author Comment

by:ntzanos
ID: 24430989
The following bit of code is the problematic one.
The error is thrown for the line where it says sb.append(line)

Could you suggest some more elaborate solution on an alternative? I am a bit lost at the moment

Many thanks

Nick
String sFile = utils.loadFileToStringWithNewlines(sInputFile);

		ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 

		for (String sLine : alLines) {

            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");

            if (saTokens.length < 4)

                continue;

            String sSense = saTokens[0];
 

            StringBuffer sbDef = new StringBuffer();

            int iCnt = 4;

            while (iCnt < saTokens.length)

                sbDef.append(saTokens[iCnt++]);

            String sDefinition = sbDef.toString();
 

            String sPolarity = saTokens[2].trim();

            int iPolarity = POLARITY_NONPOLAR;

            if (sPolarity.length() > 0)

                iPolarity = POLARITY_POSITIVE;

            else

                if (saTokens[3].trim().length() > 0)

                    iPolarity = POLARITY_NEGATIVE;
 

			hSenseToDefinition.put(sSense, sDefinition);

            hSenseToPolarity.put(sSense, iPolarity);
 

		// Start of code that prepares for 10-fold cross validation
 

		int giSize = alLines.size();

		int tenth = giSize / 10;

		LineNumberReader lnr = null;

		try {

			lnr = new LineNumberReader(new FileReader(sInputFile));

		} catch (FileNotFoundException e) {

			System.err.println("File not found..");

		}
 

		StringBuffer sb = new StringBuffer();

		HashMap<Integer,String> giChunks = new HashMap<Integer,String>();

		int counter = 1;

		String line = "";

		try {

			while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below

			//while((String line = lnr.readLine()) != null) {

				line = lnr.readLine();

				if (((lnr.getLineNumber() + 1) % tenth) == 0) {

					giChunks.put(counter++, sb.toString()); // Also increment the counter.

					sb = new StringBuffer();

				}

				sb.append(line);

			}

        } catch (IOException e) {

			// put the final tenth of the file into the HashMap

			giChunks.put(counter, sb.toString());

            try {

                lnr.close();

            } catch (IOException ex) {

                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);

            }

		}

Open in new window

0
 
LVL 92

Expert Comment

by:objects
ID: 24431011
>                                         sb = new StringBuffer();

you don't need to create a new one each time, just clear it

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431047
>>sb = new StringBuffer();

replace with

sb.delete( 0, sb.length() ) ;
0
 

Author Comment

by:ntzanos
ID: 24431237
I changed it (sb.delete(0,sb.length()) and it still crushes, but it crushes in 1min and 40 sec instead of the 40 secs that it took initially

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431282
That will help but a much worse use of memory is the following:

>>
        String sFile = utils.loadFileToStringWithNewlines(sInputFile);
        ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
>>

You create a massive String then proceed to split that String with a regex in order to put it into a List, when you could have simply read it into the List in the first place
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 26

Expert Comment

by:ksivananth
ID: 24431316
can you try this,

String sFile = utils.loadFileToStringWithNewlines(sInputFile);
            ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 StringBuffer sb = new StringBuffer();
StringBuffer sbDef = new StringBuffer();

            for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
if( sbDef.length() > 0 ) sbDef.delete( 0, sbDef.length() ) ;

            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
                  hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
            // Start of code that prepares for 10-fold cross validation
 
            int giSize = alLines.size();
            int tenth = giSize / 10;
            LineNumberReader lnr = null;
            try {
                  lnr = new LineNumberReader(new FileReader(sInputFile));
            } catch (FileNotFoundException e) {
                  System.err.println("File not found..");
            }
 
            
                                if( sb.length() > 0 ) sb.delete( 0, sb.length() ) ;
            HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
            int counter = 1;
            String line = "";
            try {
                  while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
                  //while((String line = lnr.readLine()) != null) {
                        line = lnr.readLine();
                        if (((lnr.getLineNumber() + 1) % tenth) == 0) {
                              giChunks.put(counter++, sb.toString()); // Also increment the counter.
                              sb.delete( 0, sb.length() ) ;
                        }
                        sb.append(line);
                  }
        } catch (IOException e) {
                  // put the final tenth of the file into the HashMap
                  giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
            }
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431337
... in fact the finer details are even worse, but the big picture is bad enough
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24432337
Out of interest - what size is that input file?
0
 
LVL 92

Expert Comment

by:objects
ID: 24437336
use the following to read the lines on the file into a list

http://helpdesk.objects.com.au/java/how-do-i-read-a-text-file-line-by-line-into-a-list

you can adapt it to filter your lines as required

0
 

Author Comment

by:ntzanos
ID: 24443763
The size of the file is about 1mb.
I will try again and let you all know. Maybe I need to reimplement the code because it was a mixture of my own code, with code from another person, so it would be better if I start it from scratch.

I will keep you informed.

Nick
0
 
LVL 92

Accepted Solution

by:
objects earned 500 total points
ID: 24446786
yes it could definititely be improved.  Have a look at the link I posted above which will allow you to directly read the file into a list and should remove the need for split() and regexp

0
 

Author Closing Comment

by:ntzanos
ID: 31583464
Thanks for all the help. I finally decided that it would be best to reimplement the code. And now it goes much faster (using advice from all people here). Thanks a lot
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24467785
ntzanos, can you tell us why you accepted that answer please? Future searchers need to know
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Are you developing a Java application and want to create Excel Spreadsheets? You have come to the right place, this article will describe how you can create Excel Spreadsheets from a Java Application. For the purposes of this article, I will be u…
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now