• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 243
  • Last Modified:

How to improve I/O efficiency?

Java
FileInputStream is = new FileInputStream(new File(fileName));

Open in new window


I am currently using FileInputStream to read in bytes from a file. (FileOutputStream for output)
I am processing the data by 4 bytes per block.

Is there anyway / use of any other buffered class or anything to imrpove it's I/O efficiency?
I tested on a 6mb file (reading in and writing out) and it took close to 2minutes.

Note: Need them to be in bytes stream because of some processing in bytes
0
humansg
Asked:
humansg
  • 9
  • 7
1 Solution
 
CEHJCommented:
IO is a trade-off. Certainly you want to be using a buffer size massively larger than 4 bytes. If you have large files, try the following
final int BUFFER_SIZE = 1 << 20; // 1MiB buffer

InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName), BUFFER_SIZE);

Open in new window

0
 
CEHJCommented:
Sorry
InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName)), BUFFER_SIZE);

Open in new window

0
 
humansgAuthor Commented:
what will be compromised if i set a higher buffer size?

how do i set higher?
final int BUFFER_SIZE = 2 << 20; //  for 2MiB buffer???
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
CEHJCommented:
That's probably large enough already. What gets compromised is memory. I got quite distracted posting before. that should have been
final int BUFFER_SIZE = 1 << 20; // 1MiB buffer (1<<21) for 2MiB - probably won't make significant difference
byte[] buffer = new byte[BUFFER_SIZE];
InputStream in = new BufferedInputStream(new FileInputStream(fileName), buffer);

Open in new window

0
 
humansgAuthor Commented:
InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName)), BUFFER_SIZE);

Open in new window


should be the correct one.
0
 
CEHJCommented:
Yes, you're right - i'm a bit wooly today ;)

You needn't create a File object though
0
 
humansgAuthor Commented:
ok i got you :)
0
 
CEHJCommented:
You might want to experiment *without* the BufferedInputStream too (same buffer size). If you do, i'd be interested to know your findings.
0
 
humansgAuthor Commented:
CEHJ,

we need to pass in the File object.

What do you mean by without the BufferedInputStream? the buffersize is only allowed with BufferedInputStream's constructor.
0
 
CEHJCommented:
>>we need to pass in the File object.

As used *exactly* as in the code above, the File object is redundant

>>What do you mean by without the BufferedInputStream? the buffersize is only allowed with BufferedInputStream's constructor.

You'd just read the FileInputStream directly into a buffer of that size
0
 
humansgAuthor Commented:
My previous implementation was using this (without buffer)
FileInputStream is = new FileInputStream(new File(fileName));
reading in 4 bytes at a time and processing it

it took about 2minutes for a 6mb file.
and now with BufferedInputStream, it took just 1 second!

totally awesome!

is this what you mean?
0
 
CEHJCommented:
>>reading in 4 bytes at a time and processing it

Try reading it 1 or 2 MiB at a time
0
 
humansgAuthor Commented:
Oic.

I think there will be more coding to be done as what my application actually does is to read in a byte stream and encoding it 4 bytes at a time.

encode(4 byte input) and its output written to file.

If I were to create a buffer to do it, I will need to constantly check if buffer is empty and loading new stream into this buffer.
0
 
CEHJCommented:
This is a different issue
0
 
humansgAuthor Commented:
Are you asking me to process the whole data as a 1 mb chunk instead of 4bytes?

I can't concurrently process this whole 1mb chunk because they are not independent.
ie: first 4byte output is used as input parameter for second 4byte. and second output for third input...and so on...
0
 
CEHJCommented:
Reading it and processing it are two separate issues
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 9
  • 7
Tackle projects and never again get stuck behind a technical roadblock.
Join Now