Solved

Improving performance of io in java

Posted on 2004-10-01
26
1,955 Views
Last Modified: 2013-12-29
I am currently writing data to a file using BufferedWriter object, this is so to write data in large chunks eg.

bufferedWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));

I am generating the data to write in a loop and at the end of each iteration I call
bufferedWriter.write(data)
bufferedWriter.newLine().

I wish to improve the performance, I think IO is the bottleneck.

Any ideas?
0
Comment
Question by:inzaghi
  • 6
  • 5
  • 5
  • +7
26 Comments
 
LVL 35

Accepted Solution

by:
TimYates earned 66 total points
ID: 12198641
bufferedWriter = new BufferedWriter( new FileWriter( file ) );

May be quicker...
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12198645
Can you put some timings round the code, and see how long it actually takes?

And can you show us the actual code?  It may be how you are building up "data" that is slow, not necessarially the write operation
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12198646
long then = System.currentTimeMillis() ;

...  your code ...

System.out.println( "Took " + ( System.currentTimeMillis() - then ) + "ms" ) ;
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12198648
>>I think IO is the bottleneck.

Why and what are the symptoms of a bottleneck?
0
 
LVL 18

Expert Comment

by:armoghan
ID: 12199021
/**
*Try using this
*/
try {
        BufferedReader in = new BufferedReader(new FileReader("infilename"));
        String str;
        while ((str = in.readLine()) != null) {
           ///////read here
        }
        in.close();
    } catch (IOException e) {
    }
0
 
LVL 21

Assisted Solution

by:MogalManic
MogalManic earned 62 total points
ID: 12199045
If IO is the bottleneck, it might not be possible to speed things up.  You could try increasing the buffersize.  The default size is 8K.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12199060
>>If IO is the bottleneck, it might not be possible to speed things up

But don't worry - because it probably isn't ;-)
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12199076
> *Try using this

He's is writing, not reading...
0
 
LVL 18

Expert Comment

by:armoghan
ID: 12199103
ohh sorry didnt see that
thanks TimYates :)
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12199129
:-)
0
 
LVL 1

Assisted Solution

by:helloexpert
helloexpert earned 62 total points
ID: 12199419
Unless you are using a 10 year old hard disk, your i/o into a file cannot be that slow.  If you have tried out Tim's suggestion of calculating the time taken in i/o, you would have figured out by now. Otherwise your system might be very slow, with lots of processes accessing your poor hard-disk.  Then it is a system problem which you should try to solve...

If neither of the above is the case, It must be something wrong in the code and not java's i/o problem.

two humble suggestions

1. If you are writing really tiny chunks of data in  a loop, then you could probably collect data over several loops and write after collecting a decent amount of data. This would reduce the number of writes, and could give you performance improvements.

3. using java.nio.channels (works in java1.4 and above) will give you good performance improvements. Channels use a lot of native i/o features of the OS and are generally faster than stream based i/o.


hope this helps.


0
 
LVL 7

Assisted Solution

by:bvanderveen
bvanderveen earned 62 total points
ID: 12199563
Are you threading?  You may have some issues with "blocking".  The new IO classes (nio) are a lot more thread-friendly, and won't block other threads.  I also doubt that IO is the root of your issues.  You might also reduce calls by using writeLine(String) instead of write() and newline().
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 62 total points
ID: 12200377
The ultimate writing speed in JAVA is attained by writing an IO stream with a buffer equal to the filesystem
cluster size. Then you can compose your output as bytes i.s.o. characters which is CPU consuming.
You can not do all of this easily but every little step helps.
I did some research and the result was a bit disappointing.
On a fast CPU the char to byte conversion is swift w.r.t. the physical IO.
The buffering into the physical device counts but only when no other IO on the
same disk distracts the heads of the disk from a track I desire.
I ended up with a 9% gain in speed, which was about what I could get in C.
So, don't expect too much of it.

Review your code and if that's fine .... accept it.
;JOOP!
0
 
LVL 92

Assisted Solution

by:objects
objects earned 62 total points
ID: 12204995
0
 
LVL 1

Assisted Solution

by:achilka
achilka earned 62 total points
ID: 12206377
Three things to do:

1. Wrap your bufferedWriter like this....
PrintWriter out
   = new PrintWriter(new BufferedWriter(new FileWriter("foo.out")));

2. Flush the stream after you have written 1000 lines of data

3.Check your string variable declaration if any. If possible declare once and use several times in a loop while writing string to writer.
It would have been better if you posted your complete code

Cheers!!!
0
 
LVL 6

Assisted Solution

by:mightyone
mightyone earned 62 total points
ID: 12206999
I do not agree, having tested a lot with IO I found out that generally all the buffered writers and readers are not the most performat way to read and write.
I am now using a simple stream witch is filled with a static syncronized buffer. But haven´t compared this to the "new" channel methods...

I might do this, but till now the above solution never caused me problems....

to look at the above statments, of course if you hav emuch IO u might have to think of uprgrading ur hardware. think best way to do this would be using a RAID 0 to increase write performance....
0
 
LVL 35

Expert Comment

by:TimYates
ID: 12399017
:-D
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399040
(:<)                                    ~~OK
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399209
Why have i been left out of the points split please? Is it because i have offered no anecdotes about how to speed up IO? If so, that's because i'd lay money on the fact that the performance problems have nothing to do with IO, and in that sense i'd be the only one to whom points *should* be awarded ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399402
... I missed you!
But there is something quite important in CEHJ's comment:

Many people come to EE asking us to solve a problem, but they
simply don't want to hear (read) certain answers, however accurate those might be.
Indeed, like many above noticed, there is little to improve when you follow basic
guidelines about JAVA I/O programming, but people don't want to get that answer.
In some other threads you can read f.i. a question about improving the Math.sqrt(),
about mysterious memory leaks, about components not displaying and all the time
those are ignoring simple, short, but correct answers.
I vote for inclusion of CEHJ in the list because of his sober but to-the-point remarks.

;JOOP!
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399434
LOL - nice to know this particular shot messenger has a champion ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399459
Explain:
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12399483
>>
, but they
simply don't want to hear (read) certain answers,
>>

produces the 'shoot the messenger' syndrome. I am the 'shot messenger', you are supporting me, and therefore my champion ;-)
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 12399517
:))
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

709 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now