Solved

What is the best way to write large amount of data to a file?

Posted on 2004-08-02
25
215 Views
Last Modified: 2010-03-31
Hi,

I have a Java program that does queries on a table for records meeting specific criteria and put the data in a file.  I store the data that will go to the file in a byte array.  Problem is I may have a large set of results that I may need to allocate a large enough byte array that maybe too big for the computer's memory.

I know there is FileOutputStream out = new FileOutputStream(String filename, boolean append) and write to the file over and over getting some results from the database at a time.   But I was told the pointer to a place in the file in the FileOutputStream is unreliable.

It is very IMPORTANT that the data coming from the database is on the right positions in the file because the file will go to a mainframe.

What very reliable methods do you all suggest I do to write to a file several times when I am fetching large amounts of data from a database?  Thanks.
0
Comment
Question by:Mero
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 8
  • 5
  • +1
25 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 11694101
Use a RandomAccessFile and use its seek method
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11694115
>>I store the data that will go to the file in a byte array

Can you show us how?
0
 

Author Comment

by:Mero
ID: 11694610
The code is long so I'll show you snippets of it.  This is the code currently gets all the data in the byte array and puts in the file one time.

for (loop=0; loop<numberofrecords; loop++)
{
.....
                byte[] buff;  //the byte array for the file
      for ( i = 0; i < len; i++ )
                {
          c = string.charAt(i);
          buff[i] = (byte)c;
      }
.....
//write the end of the line
dataForFile[end] = 10;
end++;
dataForFile[end] = 0;  
}

processFile.writeData(dataForFile, true);
...
public void writeData(byte[] data, boolean append)
{
      try {
            FileOutputStream out = new FileOutputStream(anout, append);
            BufferedOutputStream bufout = new BufferedOutputStream(out);
            int i = 0;

            while ( data[i] != 0 )
            {
                                                ...
                  out.write(data[i]);
                  i++;
                                 }
                 }
}
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 24

Expert Comment

by:sciuriware
ID: 11695350
Do not worry too much; write as much as you can in every write but not too much:

1 byte at a time is tragically slow.
> 1Mb per write is not showing any improvement over some 16 Kb.
Ideally you should write a disk cluster if you only knew what the disk situation was
and if drivers didn't interfere.
So, write 16K - 64Kb at a time and you're fine, even on a mainframe!

;JOOP!
0
 
LVL 92

Expert Comment

by:objects
ID: 11699848
> But I was told the pointer to a place in the file in the FileOutputStream is unreliable.

I'm not aware of any problem with it. Using a FileOutputStream in append mode should be fine.
0
 
LVL 92

Expert Comment

by:objects
ID: 11699955
to save wrting to memory, write your data directly to file:

FileOutputStream out = new FileOutputStream(anout, true);
BufferedOutputStream bufout = new BufferedOutputStream(out);
for (loop=0; loop<numberofrecords; loop++)
{
.....
     bufout.write(string.getBytes());
.....
    //write the end of the line
    bufout.write(10);
}
0
 

Author Comment

by:Mero
ID: 11703873
sciriuware,

I'm not currently writing one byte at a time.  I have it right now so that I allocate about 380K for the byte array.  I put all the data just one time provided the data does not exceed 380K.  That's why I'm trying to find another way where I don't have to allocate much more where there is a danger of running out of memory.

I might try using the RandomAccess method, CEHJ.  Thanks.

I don't think I want to use the append mode if there is some doubt about that.  The positions of the data ARE IMPORTANT on the file that is going to the mainframe.  

I'm just open to some more suggestions.  Thanks.
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 55 total points
ID: 11704167
You won't run out of memory: JAVA starts with a limit of 64Mb

You can use RandomAccessFile or FileOutputStream, both are 100% reliable.
The latter is important if the mainframe "eats" EBCDIC rather than ASCII.
The fears you have about positions or appending are superfluous:
   without programming errors there is no problem whatsoever.
;JOOP!
0
 

Author Comment

by:Mero
ID: 11704364
Believe me, I have.  I have had a out of memory exception trying to allocate 56MB for a byte array.
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 11704407
That's in line with what I said.
I run with a 512Mb limit.
You can, you know?

;JOOP!
0
 
LVL 92

Expert Comment

by:objects
ID: 11710810
> That's why I'm trying to find another way where I don't have to allocate much more where there is a danger of running out of memory.

the code i posted above removes the need for the byte array altogether, by writing directly to file.

> I don't think I want to use the append mode if there is some doubt about that.

there isn't
0
 

Author Comment

by:Mero
ID: 11737497
Can anybody show me an example  of writing to the file using RandomAccessFile and seek() with the I/O buffered?  Thanks.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11737535
Here is an example of using the RAF i suggested. Of course, you can seek() to a different point in the file if you want:

http://javaalmanac.com/egs/java.io/UseRandomAccessFile.html
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 75 total points
ID: 11737667
Your method with RAF would be:

      public void writeData(byte[] data, boolean append)
      {
            RandomAccessFile raf = null;
            try
            {
                  File f = new File("filename");
                  raf = new RandomAccessFile(f, "rw");
                  // Seek to end of file
                  raf.seek(f.length());
                  int i = 0;

                  // Append to the end
                  while ( data[i] != 0 )
                  {
                        raf.write(data[i]);
                        i++;
                  }
            }
            catch (IOException e)
            {
                  e.printStackTrace();
            }
            finally
            {

                  try { raf.close(); } catch(IOException e) { /* ignore */ }
            }
      }
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11737706
(that assumes append == true. If not, you may want to seek elsewhere)
0
 

Author Comment

by:Mero
ID: 11737807
is this write buffered?  If not, can you show me how?  Thanks.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11737823
>>is this write buffered?  

No. Why would you want it to be?
0
 

Author Comment

by:Mero
ID: 11737885
Why?  It is very common in Java to buffer your I/O
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11737904
Sometimes buffering can improve write speeds, but not necessarily. You should try both with and without before deciding.
0
 

Author Comment

by:Mero
ID: 11737968
but how do you do it in this example?
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11738061
objects has already posted an example of that, beginning:

>>to save wrting to memory, write your data directly to file:

although it must be said that the statement quoted above is inaccurate. The reason buffering is used is precisely to write to memory rather than 'directly fo file', which is what can *sometimes* improve performance
0
 
LVL 92

Assisted Solution

by:objects
objects earned 70 total points
ID: 11740449
RAF has no real advanatages in this situation, there are no problems with opening a file in append mode that I am aware of.
Can you post a reference or bug report of any problems?

Your code is already buffered :)
0
 

Author Comment

by:Mero
ID: 11872528
thanks folks!
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 11872612
:-)
0
 
LVL 92

Expert Comment

by:objects
ID: 11875140
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
spring jars download 1 68
Desingning Refactoring existing code 2 46
Glassfish admin console not working 1 88
Java ArrayList and if statement 2 55
INTRODUCTION Working with files is a moderately common task in Java.  For most projects hard coding the file names, using parameters in configuration files, or using command-line arguments is sufficient.   However, when your application has vi…
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Viewers learn about the “while” loop and how to utilize it correctly in Java. Additionally, viewers begin exploring how to include conditional statements within a while loop and avoid an endless loop. Define While Loop: Basic Example: Explanatio…
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
Suggested Courses

739 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question