Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2028
  • Last Modified:

Improving performance of io in java

I am currently writing data to a file using BufferedWriter object, this is so to write data in large chunks eg.

bufferedWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));

I am generating the data to write in a loop and at the end of each iteration I call
bufferedWriter.write(data)
bufferedWriter.newLine().

I wish to improve the performance, I think IO is the bottleneck.

Any ideas?
0
inzaghi
Asked:
inzaghi
  • 6
  • 5
  • 5
  • +7
8 Solutions
 
TimYatesCommented:
bufferedWriter = new BufferedWriter( new FileWriter( file ) );

May be quicker...
0
 
TimYatesCommented:
Can you put some timings round the code, and see how long it actually takes?

And can you show us the actual code?  It may be how you are building up "data" that is slow, not necessarially the write operation
0
 
TimYatesCommented:
long then = System.currentTimeMillis() ;

...  your code ...

System.out.println( "Took " + ( System.currentTimeMillis() - then ) + "ms" ) ;
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
CEHJCommented:
>>I think IO is the bottleneck.

Why and what are the symptoms of a bottleneck?
0
 
armoghanCommented:
/**
*Try using this
*/
try {
        BufferedReader in = new BufferedReader(new FileReader("infilename"));
        String str;
        while ((str = in.readLine()) != null) {
           ///////read here
        }
        in.close();
    } catch (IOException e) {
    }
0
 
MogalManicCommented:
If IO is the bottleneck, it might not be possible to speed things up.  You could try increasing the buffersize.  The default size is 8K.
0
 
CEHJCommented:
>>If IO is the bottleneck, it might not be possible to speed things up

But don't worry - because it probably isn't ;-)
0
 
TimYatesCommented:
> *Try using this

He's is writing, not reading...
0
 
armoghanCommented:
ohh sorry didnt see that
thanks TimYates :)
0
 
TimYatesCommented:
:-)
0
 
helloexpertCommented:
Unless you are using a 10 year old hard disk, your i/o into a file cannot be that slow.  If you have tried out Tim's suggestion of calculating the time taken in i/o, you would have figured out by now. Otherwise your system might be very slow, with lots of processes accessing your poor hard-disk.  Then it is a system problem which you should try to solve...

If neither of the above is the case, It must be something wrong in the code and not java's i/o problem.

two humble suggestions

1. If you are writing really tiny chunks of data in  a loop, then you could probably collect data over several loops and write after collecting a decent amount of data. This would reduce the number of writes, and could give you performance improvements.

3. using java.nio.channels (works in java1.4 and above) will give you good performance improvements. Channels use a lot of native i/o features of the OS and are generally faster than stream based i/o.


hope this helps.


0
 
bvanderveenCommented:
Are you threading?  You may have some issues with "blocking".  The new IO classes (nio) are a lot more thread-friendly, and won't block other threads.  I also doubt that IO is the root of your issues.  You might also reduce calls by using writeLine(String) instead of write() and newline().
0
 
sciuriwareCommented:
The ultimate writing speed in JAVA is attained by writing an IO stream with a buffer equal to the filesystem
cluster size. Then you can compose your output as bytes i.s.o. characters which is CPU consuming.
You can not do all of this easily but every little step helps.
I did some research and the result was a bit disappointing.
On a fast CPU the char to byte conversion is swift w.r.t. the physical IO.
The buffering into the physical device counts but only when no other IO on the
same disk distracts the heads of the disk from a track I desire.
I ended up with a 9% gain in speed, which was about what I could get in C.
So, don't expect too much of it.

Review your code and if that's fine .... accept it.
;JOOP!
0
 
achilkaCommented:
Three things to do:

1. Wrap your bufferedWriter like this....
PrintWriter out
   = new PrintWriter(new BufferedWriter(new FileWriter("foo.out")));

2. Flush the stream after you have written 1000 lines of data

3.Check your string variable declaration if any. If possible declare once and use several times in a loop while writing string to writer.
It would have been better if you posted your complete code

Cheers!!!
0
 
mightyoneCommented:
I do not agree, having tested a lot with IO I found out that generally all the buffered writers and readers are not the most performat way to read and write.
I am now using a simple stream witch is filled with a static syncronized buffer. But havenĀ“t compared this to the "new" channel methods...

I might do this, but till now the above solution never caused me problems....

to look at the above statments, of course if you hav emuch IO u might have to think of uprgrading ur hardware. think best way to do this would be using a RAID 0 to increase write performance....
0
 
TimYatesCommented:
:-D
0
 
sciuriwareCommented:
(:<)                                    ~~OK
0
 
CEHJCommented:
Why have i been left out of the points split please? Is it because i have offered no anecdotes about how to speed up IO? If so, that's because i'd lay money on the fact that the performance problems have nothing to do with IO, and in that sense i'd be the only one to whom points *should* be awarded ;-)
0
 
sciuriwareCommented:
... I missed you!
But there is something quite important in CEHJ's comment:

Many people come to EE asking us to solve a problem, but they
simply don't want to hear (read) certain answers, however accurate those might be.
Indeed, like many above noticed, there is little to improve when you follow basic
guidelines about JAVA I/O programming, but people don't want to get that answer.
In some other threads you can read f.i. a question about improving the Math.sqrt(),
about mysterious memory leaks, about components not displaying and all the time
those are ignoring simple, short, but correct answers.
I vote for inclusion of CEHJ in the list because of his sober but to-the-point remarks.

;JOOP!
0
 
CEHJCommented:
LOL - nice to know this particular shot messenger has a champion ;-)
0
 
sciuriwareCommented:
Explain:
0
 
CEHJCommented:
>>
, but they
simply don't want to hear (read) certain answers,
>>

produces the 'shoot the messenger' syndrome. I am the 'shot messenger', you are supporting me, and therefore my champion ;-)
0
 
sciuriwareCommented:
:))
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 6
  • 5
  • 5
  • +7
Tackle projects and never again get stuck behind a technical roadblock.
Join Now