Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

PrintWriter memory consumption

Posted on 2004-04-06
27
Medium Priority
?
1,361 Views
Last Modified: 2012-05-04
Hi,
I have a java program with 20 threads.  Each tread uses a different PrintWriter that is already open to write to a seperate text file.  The problem is that I get a java.lang.OutOfMemory error after a certain period of time.  I was wondering if anyone could tell me if PrintWriter needs to use more memory as the size of the text file increases.  If so is there anyway to perform these write operations without incurring an out of memory error, or does the PrintWriter strain on memory remain static regardless of the size of the text file.
         Cheers,
         everton690  
0
Comment
Question by:everton690
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 7
  • 5
  • +3
27 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764215
Your problem is more likely to be caused by not closing Writers or other resources after use.
0
 

Author Comment

by:everton690
ID: 10764229
The thing is that the Writers must remain open to continually allow data to be written to the text files.
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764234
Seems like all your contents are in memory.  It is not going to the file ... i suppose.

If it is in memory, when the text size grows, obviously, there would be out of memory exception.


Regards,
Muruga
0
Build and deliver software with DevOps

A digital transformation requires faster time to market, shorter software development lifecycles, and the ability to adapt rapidly to changing customer demands. DevOps provides the solution.

 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764241
Can you post your code
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764244
Tell us more about what's happening. So far assuming 20 PW's and one text file. How big is the text file?
0
 

Author Comment

by:everton690
ID: 10764277
There are 20 PW's,20 text files and 20 threads.  Each thread calls a function that uses a PW that is already open to write to the text file.  The program then calls another function that reads from this text file.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764287
Sounds to me like you should be reading from and writing to buffers at this point - not files (at least until all processing is finished)
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764293
Let me tell you one thing, if all the contents are going into the file, then there would be very less chance of outofmemory exception.

That portion of file would help us finding the exact cause.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764403
The JVM does not use all the available memory. You can specify how much memory you want it to use, by the -Xmx option. Like:

-Xmx200m

- tells the JVM to use 200 MB
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764419
I would still agree with CEHJ that you should use buffers instead of writing directly to files. You can try BufferedWriter.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764428
Ah, I thought CEHJ told you to use buffers. I guess he meant that you were *using* buffers. Well, which side are you?
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764433
>>I would still agree with CEHJ ...

Difficult to tell exactly though without knowing more
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764446
Generally, I prefer using buffers. Disk I/O is slower. But well, if it gives OutOfMemoryException, I would eliminate the buffers ;-)
0
 

Author Comment

by:everton690
ID: 10764449
Here is an example of a thread and the read and write functions.  Hopefully this will help.  


 Thread thread = new Thread(new Runnable() {
      public void run() {
              
    writesomething(input param,nameo PrintWriter);
   
         
    while (program not finished)
    {
   
     
      Writer.flush();
     
      readfromfile(input param,input param);
      
               
     }
   
do some computation

     writetofileagain(input param, input param);
 
     
}
    });
    thread.start();
          

//function to write to file

writesomething(input param, nameofPrintWriter){

nameofPrintWriter.println(something);

}

//function to read from file

readfromfile(int inputparam,nameoffile){
      
FileReader fr=null;
String s=null;

String patt =" "+inputparam+" :";

     String file =
      System.getProperty("user.dir") +
      System.getProperty("file.separator") +nameoffile+".txt";
String str=null;


try {
        fr = new FileReader(file);
        LineNumberReader lr =
          new LineNumberReader(fr);
       
        while ((str = lr.readLine()) != null) {
            
          if (str.indexOf(patt) != -1) {

         s=str.substring(11);
      
       // lr.close();
        }
    }
  fr.close();
}
         
    catch (ArrayIndexOutOfBoundsException e) {
        System.err.println("Caught ArrayIndexOutOfBoundsException: " +e.getMessage());
                                       }

    catch (IOException e) {
        System.err.println("Caught IOException: for readsomething " + e.getMessage());
      
                          }
 
return s;
}


     
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764482
What's the average file size?
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764487
Looks like you might have to try the -Xmx option.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764511
Check the amount of free memory that you have. I guess that the JVM is not using all of it. Suppose its 200 MB, then use:

java -Xmx200m

You can also increase the maximum stack-size with -Xms
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764513
Frankly i'm surprised that is working, (excepting the OutOfMemory) as if the Writers are not being closed, you might get an old copy of the file instead of the data from the most recent write. *Was* it working?

Anyway, those (almost certainly unnecessarily) tight loops will cause a massive allocation of resources that will eventually produce your OOM
0
 

Author Comment

by:everton690
ID: 10764544
I tried using the -Xmx option but it only delays the java.lang.OutOfMemory error.  The file sizes will continually grow until each one contains roughly 700,000 lines of data.  The reason the data is being sent to the text files and not stored in memory  is to reduce the strain on memory.  Do you think this method of using Writers and Readers is increasing the strain on memory as the file size increases or is the strain irrelevant of the file size?
0
 
LVL 9

Expert Comment

by:mmuruganandam
ID: 10764569
My suspect is at LineNumberReader.  It will read the entire content.  

I am not seeing any use of LineNumberReader.  Replace that with BufferedReader.  That might help.
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10764574
If it delays the OutOfMemory error, then it means that it is working to some extent - you are still using lesser memory. You need even more. I suggest that you check how much available memory you have on your system, and then give the entire as parameter to -Xmx. Try increasing the stack-size as well.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 10764579
I think OOM, given the code, would probably happen with much smaller files but the fact they're large or at least becoming large is certainly going to make things worse.

You should probably be doing this with your own buffers, but at the moment i can't think of a good way to implement it.
0
 
LVL 24

Accepted Solution

by:
sciuriware earned 1000 total points
ID: 10764659
I am using some 70 Printwriters, with buffering.

Search in another direction, it's definitely NOT the IO that causes out-of-memory!


;JOOP!
0
 
LVL 16

Expert Comment

by:gnoon
ID: 10772127
>while (program not finished) {
so, when the program got finish?

Maybe, the infinite loop cause the OOM occur. Trying to print something in the loop and run only 1 thread to test it.

0
 

Author Comment

by:everton690
ID: 10959924

The cause of the memory leak was nothing to do with any of the IO.  I gave the points to sciuriware as he was right but to provide a solution would have required a lot more information

Cheers,
everton690  
0
 
LVL 30

Expert Comment

by:Mayank S
ID: 10971011
Were you able to figure out that what was the actual cause?
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question