Solved

PrintWriter memory consumption

Posted on 2004-04-06
27
1,290 Views
Last Modified: 2012-05-04
Hi,
I have a java program with 20 threads.  Each tread uses a different PrintWriter that is already open to write to a seperate text file.  The problem is that I get a java.lang.OutOfMemory error after a certain period of time.  I was wondering if anyone could tell me if PrintWriter needs to use more memory as the size of the text file increases.  If so is there anyway to perform these write operations without incurring an out of memory error, or does the PrintWriter strain on memory remain static regardless of the size of the text file.
         Cheers,
         everton690  
0
Comment
Question by:everton690
  • 8
  • 7
  • 5
  • +3
27 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764215
Your problem is more likely to be caused by not closing Writers or other resources after use.
0
 

Author Comment

by:everton690
ID: 10764229
The thing is that the Writers must remain open to continually allow data to be written to the text files.
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764234
Seems like all your contents are in memory.  It is not going to the file ... i suppose.

If it is in memory, when the text size grows, obviously, there would be out of memory exception.


Regards,
Muruga
0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764241
Can you post your code
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764244
Tell us more about what's happening. So far assuming 20 PW's and one text file. How big is the text file?
0
 

Author Comment

by:everton690
ID: 10764277
There are 20 PW's,20 text files and 20 threads.  Each thread calls a function that uses a PW that is already open to write to the text file.  The program then calls another function that reads from this text file.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764287
Sounds to me like you should be reading from and writing to buffers at this point - not files (at least until all processing is finished)
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764293
Let me tell you one thing, if all the contents are going into the file, then there would be very less chance of outofmemory exception.

That portion of file would help us finding the exact cause.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764403
The JVM does not use all the available memory. You can specify how much memory you want it to use, by the -Xmx option. Like:

-Xmx200m

- tells the JVM to use 200 MB
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764419
I would still agree with CEHJ that you should use buffers instead of writing directly to files. You can try BufferedWriter.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764428
Ah, I thought CEHJ told you to use buffers. I guess he meant that you were *using* buffers. Well, which side are you?
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764433
>>I would still agree with CEHJ ...

Difficult to tell exactly though without knowing more
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764446
Generally, I prefer using buffers. Disk I/O is slower. But well, if it gives OutOfMemoryException, I would eliminate the buffers ;-)
0
 

Author Comment

by:everton690
ID: 10764449
Here is an example of a thread and the read and write functions.  Hopefully this will help.  


 Thread thread = new Thread(new Runnable() {
      public void run() {
              
    writesomething(input param,nameo PrintWriter);
   
         
    while (program not finished)
    {
   
     
      Writer.flush();
     
      readfromfile(input param,input param);
      
               
     }
   
do some computation

     writetofileagain(input param, input param);
 
     
}
    });
    thread.start();
          

//function to write to file

writesomething(input param, nameofPrintWriter){

nameofPrintWriter.println(something);

}

//function to read from file

readfromfile(int inputparam,nameoffile){
      
FileReader fr=null;
String s=null;

String patt =" "+inputparam+" :";

     String file =
      System.getProperty("user.dir") +
      System.getProperty("file.separator") +nameoffile+".txt";
String str=null;


try {
        fr = new FileReader(file);
        LineNumberReader lr =
          new LineNumberReader(fr);
       
        while ((str = lr.readLine()) != null) {
            
          if (str.indexOf(patt) != -1) {

         s=str.substring(11);
      
       // lr.close();
        }
    }
  fr.close();
}
         
    catch (ArrayIndexOutOfBoundsException e) {
        System.err.println("Caught ArrayIndexOutOfBoundsException: " +e.getMessage());
                                       }

    catch (IOException e) {
        System.err.println("Caught IOException: for readsomething " + e.getMessage());
      
                          }
 
return s;
}


     
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764482
What's the average file size?
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764487
Looks like you might have to try the -Xmx option.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764511
Check the amount of free memory that you have. I guess that the JVM is not using all of it. Suppose its 200 MB, then use:

java -Xmx200m

You can also increase the maximum stack-size with -Xms
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764513
Frankly i'm surprised that is working, (excepting the OutOfMemory) as if the Writers are not being closed, you might get an old copy of the file instead of the data from the most recent write. *Was* it working?

Anyway, those (almost certainly unnecessarily) tight loops will cause a massive allocation of resources that will eventually produce your OOM
0
 

Author Comment

by:everton690
ID: 10764544
I tried using the -Xmx option but it only delays the java.lang.OutOfMemory error.  The file sizes will continually grow until each one contains roughly 700,000 lines of data.  The reason the data is being sent to the text files and not stored in memory  is to reduce the strain on memory.  Do you think this method of using Writers and Readers is increasing the strain on memory as the file size increases or is the strain irrelevant of the file size?
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764569
My suspect is at LineNumberReader.  It will read the entire content.  

I am not seeing any use of LineNumberReader.  Replace that with BufferedReader.  That might help.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764574
If it delays the OutOfMemory error, then it means that it is working to some extent - you are still using lesser memory. You need even more. I suggest that you check how much available memory you have on your system, and then give the entire as parameter to -Xmx. Try increasing the stack-size as well.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764579
I think OOM, given the code, would probably happen with much smaller files but the fact they're large or at least becoming large is certainly going to make things worse.

You should probably be doing this with your own buffers, but at the moment i can't think of a good way to implement it.
0
 
LVL 24

Accepted Solution

by:
sciuriware earned 250 total points
ID: 10764659
I am using some 70 Printwriters, with buffering.

Search in another direction, it's definitely NOT the IO that causes out-of-memory!


;JOOP!
0
 
LVL 16

Expert Comment

by:gnoon
ID: 10772127
>while (program not finished) {
so, when the program got finish?

Maybe, the infinite loop cause the OOM occur. Trying to print something in the loop and run only 1 thread to test it.

0
 

Author Comment

by:everton690
ID: 10959924

The cause of the memory leak was nothing to do with any of the IO.  I gave the points to sciuriware as he was right but to provide a solution would have required a lot more information

Cheers,
everton690  
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10971011
Were you able to figure out that what was the actual cause?
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Java had always been an easily readable and understandable language.  Some relatively recent changes in the language seem to be changing this pretty fast, and anyone that had not seen any Java code for the last 5 years will possibly have issues unde…
Introduction This article is the first of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article explains our test automation goals. Then rationale is given for the tools we use to a…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question