Solved

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Posted on 2009-05-20
16
1,331 Views
Last Modified: 2012-05-07
Hello,

I get the exception in the title in an application I have written.
It seems to be caused by a large StringBuffer. The settings I
use are -Xms1024m -Xmx1512m and
-XX:+AggressiveOpts -XX:+UseParallelGC.

The machine is running Kubuntu 9.04 64bit and the physical
memory is 2g.

Any ideas on how I could overcome the problem?

Regards,

Nick
0
Comment
Question by:ntzanos
  • 5
  • 4
  • 4
  • +1
16 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 24430778
>>Any ideas on how I could overcome the problem?

Probably by using the StringBuffer more carefully - they have big gotchas. Please post code
0
 
LVL 92

Expert Comment

by:objects
ID: 24430798
try running it in a 32 bit jvm

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24430867
though there are issues in frequent StringBuffer creation and setting size, mostly the issue will be with String copies created out of the buffer and memory leaks. first try running profiler and see if there are leaks. On the other hand, also check if it is really required to have that big string in memory. check the possibility of reading partial string to memory through stream rather loading the entire string into memory. also consider increasing the heap!
0
 

Author Comment

by:ntzanos
ID: 24430989
The following bit of code is the problematic one.
The error is thrown for the line where it says sb.append(line)

Could you suggest some more elaborate solution on an alternative? I am a bit lost at the moment

Many thanks

Nick
String sFile = utils.loadFileToStringWithNewlines(sInputFile);

		ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 

		for (String sLine : alLines) {

            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");

            if (saTokens.length < 4)

                continue;

            String sSense = saTokens[0];
 

            StringBuffer sbDef = new StringBuffer();

            int iCnt = 4;

            while (iCnt < saTokens.length)

                sbDef.append(saTokens[iCnt++]);

            String sDefinition = sbDef.toString();
 

            String sPolarity = saTokens[2].trim();

            int iPolarity = POLARITY_NONPOLAR;

            if (sPolarity.length() > 0)

                iPolarity = POLARITY_POSITIVE;

            else

                if (saTokens[3].trim().length() > 0)

                    iPolarity = POLARITY_NEGATIVE;
 

			hSenseToDefinition.put(sSense, sDefinition);

            hSenseToPolarity.put(sSense, iPolarity);
 

		// Start of code that prepares for 10-fold cross validation
 

		int giSize = alLines.size();

		int tenth = giSize / 10;

		LineNumberReader lnr = null;

		try {

			lnr = new LineNumberReader(new FileReader(sInputFile));

		} catch (FileNotFoundException e) {

			System.err.println("File not found..");

		}
 

		StringBuffer sb = new StringBuffer();

		HashMap<Integer,String> giChunks = new HashMap<Integer,String>();

		int counter = 1;

		String line = "";

		try {

			while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below

			//while((String line = lnr.readLine()) != null) {

				line = lnr.readLine();

				if (((lnr.getLineNumber() + 1) % tenth) == 0) {

					giChunks.put(counter++, sb.toString()); // Also increment the counter.

					sb = new StringBuffer();

				}

				sb.append(line);

			}

        } catch (IOException e) {

			// put the final tenth of the file into the HashMap

			giChunks.put(counter, sb.toString());

            try {

                lnr.close();

            } catch (IOException ex) {

                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);

            }

		}

Open in new window

0
 
LVL 92

Expert Comment

by:objects
ID: 24431011
>                                         sb = new StringBuffer();

you don't need to create a new one each time, just clear it

0
 
LVL 26

Expert Comment

by:ksivananth
ID: 24431047
>>sb = new StringBuffer();

replace with

sb.delete( 0, sb.length() ) ;
0
 

Author Comment

by:ntzanos
ID: 24431237
I changed it (sb.delete(0,sb.length()) and it still crushes, but it crushes in 1min and 40 sec instead of the 40 secs that it took initially

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431282
That will help but a much worse use of memory is the following:

>>
        String sFile = utils.loadFileToStringWithNewlines(sInputFile);
        ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
>>

You create a massive String then proceed to split that String with a regex in order to put it into a List, when you could have simply read it into the List in the first place
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 26

Expert Comment

by:ksivananth
ID: 24431316
can you try this,

String sFile = utils.loadFileToStringWithNewlines(sInputFile);
            ArrayList<String> alLines = new ArrayList<String>(Arrays.asList(sFile.split("\n")));
 StringBuffer sb = new StringBuffer();
StringBuffer sbDef = new StringBuffer();

            for (String sLine : alLines) {
            String[] saTokens = sLine.replaceAll(",,", ", ,").split("\"*,\"*");
            if (saTokens.length < 4)
                continue;
            String sSense = saTokens[0];
 
if( sbDef.length() > 0 ) sbDef.delete( 0, sbDef.length() ) ;

            int iCnt = 4;
            while (iCnt < saTokens.length)
                sbDef.append(saTokens[iCnt++]);
            String sDefinition = sbDef.toString();
 
            String sPolarity = saTokens[2].trim();
            int iPolarity = POLARITY_NONPOLAR;
            if (sPolarity.length() > 0)
                iPolarity = POLARITY_POSITIVE;
            else
                if (saTokens[3].trim().length() > 0)
                    iPolarity = POLARITY_NEGATIVE;
 
                  hSenseToDefinition.put(sSense, sDefinition);
            hSenseToPolarity.put(sSense, iPolarity);
 
            // Start of code that prepares for 10-fold cross validation
 
            int giSize = alLines.size();
            int tenth = giSize / 10;
            LineNumberReader lnr = null;
            try {
                  lnr = new LineNumberReader(new FileReader(sInputFile));
            } catch (FileNotFoundException e) {
                  System.err.println("File not found..");
            }
 
            
                                if( sb.length() > 0 ) sb.delete( 0, sb.length() ) ;
            HashMap<Integer,String> giChunks = new HashMap<Integer,String>();
            int counter = 1;
            String line = "";
            try {
                  while(true) { // The reason this is done is so that I can get the final chunk (which will throw the EOFException and be caught below
                  //while((String line = lnr.readLine()) != null) {
                        line = lnr.readLine();
                        if (((lnr.getLineNumber() + 1) % tenth) == 0) {
                              giChunks.put(counter++, sb.toString()); // Also increment the counter.
                              sb.delete( 0, sb.length() ) ;
                        }
                        sb.append(line);
                  }
        } catch (IOException e) {
                  // put the final tenth of the file into the HashMap
                  giChunks.put(counter, sb.toString());
            try {
                lnr.close();
            } catch (IOException ex) {
                Logger.getLogger(GraphTenFold.class.getName()).log(Level.SEVERE, null, ex);
            }
            }
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24431337
... in fact the finer details are even worse, but the big picture is bad enough
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24432337
Out of interest - what size is that input file?
0
 
LVL 92

Expert Comment

by:objects
ID: 24437336
use the following to read the lines on the file into a list

http://helpdesk.objects.com.au/java/how-do-i-read-a-text-file-line-by-line-into-a-list

you can adapt it to filter your lines as required

0
 

Author Comment

by:ntzanos
ID: 24443763
The size of the file is about 1mb.
I will try again and let you all know. Maybe I need to reimplement the code because it was a mixture of my own code, with code from another person, so it would be better if I start it from scratch.

I will keep you informed.

Nick
0
 
LVL 92

Accepted Solution

by:
objects earned 500 total points
ID: 24446786
yes it could definititely be improved.  Have a look at the link I posted above which will allow you to directly read the file into a list and should remove the need for split() and regexp

0
 

Author Closing Comment

by:ntzanos
ID: 31583464
Thanks for all the help. I finally decided that it would be best to reimplement the code. And now it goes much faster (using advice from all people here). Thanks a lot
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24467785
ntzanos, can you tell us why you accepted that answer please? Future searchers need to know
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Java had always been an easily readable and understandable language.  Some relatively recent changes in the language seem to be changing this pretty fast, and anyone that had not seen any Java code for the last 5 years will possibly have issues unde…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now