Solved

Improving performance of io in java

Posted on 2004-10-01
26
1,977 Views
Last Modified: 2013-12-29
I am currently writing data to a file using BufferedWriter object, this is so to write data in large chunks eg.

bufferedWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));

I am generating the data to write in a loop and at the end of each iteration I call
bufferedWriter.write(data)
bufferedWriter.newLine().

I wish to improve the performance, I think IO is the bottleneck.

Any ideas?
0
Comment
Question by:inzaghi
  • 6
  • 5
  • 5
  • +7
26 Comments
 
LVL 35

Accepted Solution

by:
TimYates earned 66 total points
ID: 12198641
bufferedWriter = new BufferedWriter( new FileWriter( file ) );

May be quicker...
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12198645
Can you put some timings round the code, and see how long it actually takes?

And can you show us the actual code?  It may be how you are building up "data" that is slow, not necessarially the write operation
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12198646
long then = System.currentTimeMillis() ;

...  your code ...

System.out.println( "Took " + ( System.currentTimeMillis() - then ) + "ms" ) ;
0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 86

Expert Comment

by:CEHJ
ID: 12198648
>>I think IO is the bottleneck.

Why and what are the symptoms of a bottleneck?
0
 
LVL 18

Expert Comment

by:armoghan
ID: 12199021
/**
*Try using this
*/
try {
        BufferedReader in = new BufferedReader(new FileReader("infilename"));
        String str;
        while ((str = in.readLine()) != null) {
           ///////read here
        }
        in.close();
    } catch (IOException e) {
    }
0
 
LVL 21

Assisted Solution

by:MogalManic
MogalManic earned 62 total points
ID: 12199045
If IO is the bottleneck, it might not be possible to speed things up.  You could try increasing the buffersize.  The default size is 8K.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12199060
>>If IO is the bottleneck, it might not be possible to speed things up

But don't worry - because it probably isn't ;-)
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12199076
> *Try using this

He's is writing, not reading...
0
 
LVL 18

Expert Comment

by:armoghan
ID: 12199103
ohh sorry didnt see that
thanks TimYates :)
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12199129
:-)
0
 
LVL 1

Assisted Solution

by:helloexpert
helloexpert earned 62 total points
ID: 12199419
Unless you are using a 10 year old hard disk, your i/o into a file cannot be that slow.  If you have tried out Tim's suggestion of calculating the time taken in i/o, you would have figured out by now. Otherwise your system might be very slow, with lots of processes accessing your poor hard-disk.  Then it is a system problem which you should try to solve...

If neither of the above is the case, It must be something wrong in the code and not java's i/o problem.

two humble suggestions

1. If you are writing really tiny chunks of data in  a loop, then you could probably collect data over several loops and write after collecting a decent amount of data. This would reduce the number of writes, and could give you performance improvements.

3. using java.nio.channels (works in java1.4 and above) will give you good performance improvements. Channels use a lot of native i/o features of the OS and are generally faster than stream based i/o.


hope this helps.


0
 
LVL 7

Assisted Solution

by:bvanderveen
bvanderveen earned 62 total points
ID: 12199563
Are you threading?  You may have some issues with "blocking".  The new IO classes (nio) are a lot more thread-friendly, and won't block other threads.  I also doubt that IO is the root of your issues.  You might also reduce calls by using writeLine(String) instead of write() and newline().
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 62 total points
ID: 12200377
The ultimate writing speed in JAVA is attained by writing an IO stream with a buffer equal to the filesystem
cluster size. Then you can compose your output as bytes i.s.o. characters which is CPU consuming.
You can not do all of this easily but every little step helps.
I did some research and the result was a bit disappointing.
On a fast CPU the char to byte conversion is swift w.r.t. the physical IO.
The buffering into the physical device counts but only when no other IO on the
same disk distracts the heads of the disk from a track I desire.
I ended up with a 9% gain in speed, which was about what I could get in C.
So, don't expect too much of it.

Review your code and if that's fine .... accept it.
;JOOP!
0
 
LVL 92

Assisted Solution

by:objects
objects earned 62 total points
ID: 12204995
0
 
LVL 1

Assisted Solution

by:achilka
achilka earned 62 total points
ID: 12206377
Three things to do:

1. Wrap your bufferedWriter like this....
PrintWriter out
   = new PrintWriter(new BufferedWriter(new FileWriter("foo.out")));

2. Flush the stream after you have written 1000 lines of data

3.Check your string variable declaration if any. If possible declare once and use several times in a loop while writing string to writer.
It would have been better if you posted your complete code

Cheers!!!
0
 
LVL 6

Assisted Solution

by:mightyone
mightyone earned 62 total points
ID: 12206999
I do not agree, having tested a lot with IO I found out that generally all the buffered writers and readers are not the most performat way to read and write.
I am now using a simple stream witch is filled with a static syncronized buffer. But haven´t compared this to the "new" channel methods...

I might do this, but till now the above solution never caused me problems....

to look at the above statments, of course if you hav emuch IO u might have to think of uprgrading ur hardware. think best way to do this would be using a RAID 0 to increase write performance....
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12399017
:-D
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399040
(:<)                                    ~~OK
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399209
Why have i been left out of the points split please? Is it because i have offered no anecdotes about how to speed up IO? If so, that's because i'd lay money on the fact that the performance problems have nothing to do with IO, and in that sense i'd be the only one to whom points *should* be awarded ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399402
... I missed you!
But there is something quite important in CEHJ's comment:

Many people come to EE asking us to solve a problem, but they
simply don't want to hear (read) certain answers, however accurate those might be.
Indeed, like many above noticed, there is little to improve when you follow basic
guidelines about JAVA I/O programming, but people don't want to get that answer.
In some other threads you can read f.i. a question about improving the Math.sqrt(),
about mysterious memory leaks, about components not displaying and all the time
those are ignoring simple, short, but correct answers.
I vote for inclusion of CEHJ in the list because of his sober but to-the-point remarks.

;JOOP!
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399434
LOL - nice to know this particular shot messenger has a champion ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399459
Explain:
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399483
>>
, but they
simply don't want to hear (read) certain answers,
>>

produces the 'shoot the messenger' syndrome. I am the 'shot messenger', you are supporting me, and therefore my champion ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399517
:))
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Are you developing a Java application and want to create Excel Spreadsheets? You have come to the right place, this article will describe how you can create Excel Spreadsheets from a Java Application. For the purposes of this article, I will be u…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question