How to improve I/O efficiency?

Java
FileInputStream is = new FileInputStream(new File(fileName));

Open in new window


I am currently using FileInputStream to read in bytes from a file. (FileOutputStream for output)
I am processing the data by 4 bytes per block.

Is there anyway / use of any other buffered class or anything to imrpove it's I/O efficiency?
I tested on a 6mb file (reading in and writing out) and it took close to 2minutes.

Note: Need them to be in bytes stream because of some processing in bytes
humansgAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CEHJCommented:
IO is a trade-off. Certainly you want to be using a buffer size massively larger than 4 bytes. If you have large files, try the following
final int BUFFER_SIZE = 1 << 20; // 1MiB buffer

InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName), BUFFER_SIZE);

Open in new window

0
CEHJCommented:
Sorry
InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName)), BUFFER_SIZE);

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
humansgAuthor Commented:
what will be compromised if i set a higher buffer size?

how do i set higher?
final int BUFFER_SIZE = 2 << 20; //  for 2MiB buffer???
0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

CEHJCommented:
That's probably large enough already. What gets compromised is memory. I got quite distracted posting before. that should have been
final int BUFFER_SIZE = 1 << 20; // 1MiB buffer (1<<21) for 2MiB - probably won't make significant difference
byte[] buffer = new byte[BUFFER_SIZE];
InputStream in = new BufferedInputStream(new FileInputStream(fileName), buffer);

Open in new window

0
humansgAuthor Commented:
InputStream in = new BufferedInputStream(new FileInputStream(new File(fileName)), BUFFER_SIZE);

Open in new window


should be the correct one.
0
CEHJCommented:
Yes, you're right - i'm a bit wooly today ;)

You needn't create a File object though
0
humansgAuthor Commented:
ok i got you :)
0
CEHJCommented:
You might want to experiment *without* the BufferedInputStream too (same buffer size). If you do, i'd be interested to know your findings.
0
humansgAuthor Commented:
CEHJ,

we need to pass in the File object.

What do you mean by without the BufferedInputStream? the buffersize is only allowed with BufferedInputStream's constructor.
0
CEHJCommented:
>>we need to pass in the File object.

As used *exactly* as in the code above, the File object is redundant

>>What do you mean by without the BufferedInputStream? the buffersize is only allowed with BufferedInputStream's constructor.

You'd just read the FileInputStream directly into a buffer of that size
0
humansgAuthor Commented:
My previous implementation was using this (without buffer)
FileInputStream is = new FileInputStream(new File(fileName));
reading in 4 bytes at a time and processing it

it took about 2minutes for a 6mb file.
and now with BufferedInputStream, it took just 1 second!

totally awesome!

is this what you mean?
0
CEHJCommented:
>>reading in 4 bytes at a time and processing it

Try reading it 1 or 2 MiB at a time
0
humansgAuthor Commented:
Oic.

I think there will be more coding to be done as what my application actually does is to read in a byte stream and encoding it 4 bytes at a time.

encode(4 byte input) and its output written to file.

If I were to create a buffer to do it, I will need to constantly check if buffer is empty and loading new stream into this buffer.
0
CEHJCommented:
This is a different issue
0
humansgAuthor Commented:
Are you asking me to process the whole data as a 1 mb chunk instead of 4bytes?

I can't concurrently process this whole 1mb chunk because they are not independent.
ie: first 4byte output is used as input parameter for second 4byte. and second output for third input...and so on...
0
CEHJCommented:
Reading it and processing it are two separate issues
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.