What is the best way to write large amount of data to a file?

Use a RandomAccessFile and use its seek method

>>I store the data that will go to the file in a byte array

Can you show us how?

The code is long so I'll show you snippets of it.  This is the code currently gets all the data in the byte array and puts in the file one time.

for (loop=0; loop<numberofrecords; loop++)
{
.....
                byte[] buff;  //the byte array for the file
      for ( i = 0; i < len; i++ )
                {
          c = string.charAt(i);
          buff[i] = (byte)c;
      }
.....
//write the end of the line
dataForFile[end] = 10;
end++;
dataForFile[end] = 0;  
}

processFile.writeData(dataForFile, true);
...
public void writeData(byte[] data, boolean append)
{
      try {
            FileOutputStream out = new FileOutputStream(anout, append);
            BufferedOutputStream bufout = new BufferedOutputStream(out);
            int i = 0;

            while ( data[i] != 0 )
            {
                                                ...
                  out.write(data[i]);
                  i++;
                                 }
                 }
}

Do not worry too much; write as much as you can in every write but not too much:

1 byte at a time is tragically slow.
> 1Mb per write is not showing any improvement over some 16 Kb.
Ideally you should write a disk cluster if you only knew what the disk situation was
and if drivers didn't interfere.
So, write 16K - 64Kb at a time and you're fine, even on a mainframe!

;JOOP!

> But I was told the pointer to a place in the file in the FileOutputStream is unreliable.

I'm not aware of any problem with it. Using a FileOutputStream in append mode should be fine.

to save wrting to memory, write your data directly to file:

FileOutputStream out = new FileOutputStream(anout, true);
BufferedOutputStream bufout = new BufferedOutputStream(out);
for (loop=0; loop<numberofrecords; loop++)
{
.....
     bufout.write(string.getBytes());
.....
    //write the end of the line
    bufout.write(10);
}

sciriuware,

I'm not currently writing one byte at a time.  I have it right now so that I allocate about 380K for the byte array.  I put all the data just one time provided the data does not exceed 380K.  That's why I'm trying to find another way where I don't have to allocate much more where there is a danger of running out of memory.

I might try using the RandomAccess method, CEHJ.  Thanks.

I don't think I want to use the append mode if there is some doubt about that.  The positions of the data ARE IMPORTANT on the file that is going to the mainframe.  

I'm just open to some more suggestions.  Thanks.

Believe me, I have.  I have had a out of memory exception trying to allocate 56MB for a byte array.

That's in line with what I said.
I run with a 512Mb limit.
You can, you know?

;JOOP!

> That's why I'm trying to find another way where I don't have to allocate much more where there is a danger of running out of memory.

the code i posted above removes the need for the byte array altogether, by writing directly to file.

> I don't think I want to use the append mode if there is some doubt about that.

there isn't

Can anybody show me an example  of writing to the file using RandomAccessFile and seek() with the I/O buffered?  Thanks.

Here is an example of using the RAF i suggested. Of course, you can seek() to a different point in the file if you want:

http://javaalmanac.com/egs/java.io/UseRandomAccessFile.html

(that assumes append == true. If not, you may want to seek elsewhere)

is this write buffered?  If not, can you show me how?  Thanks.

>>is this write buffered?  

No. Why would you want it to be?

Why?  It is very common in Java to buffer your I/O

Sometimes buffering can improve write speeds, but not necessarily. You should try both with and without before deciding.

but how do you do it in this example?

objects has already posted an example of that, beginning:

>>to save wrting to memory, write your data directly to file:

although it must be said that the statement quoted above is inaccurate. The reason buffering is used is precisely to write to memory rather than 'directly fo file', which is what can *sometimes* improve performance

thanks folks!

:-)

:-)

http://www.objects.com.au/staff/mick