Newbie question: write text file FAST?

Dear Experts,

I'm using JDBC to return a large result set (>1,000,000 records...).  I want to write it out to a simple text file.  Getting the records from a remote server very fast.  But writing them out locally to a simple text file is very slow.

Is there a way in java to write text files FAST?  Like, can you somehow read the next 1000 results, store them in memory, and then write them out at once?

This is the code:

  Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    System.out.println(rs.getString(1));
  }
  con.close();

Thanks, Experts !

--BrianMc1958
BrianMc1958Asked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
sudhakar_koundinyaConnect With a Mentor Commented:
>> System.out.println(sb.toString().getBytes());
should be
 System.out.println(sb);
0
 
sudhakar_koundinyaCommented:
StringBuffer sb=new StringBuffer();
int i=0;
 Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1));
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
  con.close();
fout.close();
0
 
sudhakar_koundinyaCommented:
while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
BrianMc1958Author Commented:
Dear sudhakar:

Thanks, but it's not quite right yet.  My results are much faster, but garbled.

I did not include my full code.  I'm using this to set up the text file:

  File logFile = new File("C:\\JUNK\\BIGFILE.txt");
  try
  {
  PrintStream printStream =
      new PrintStream(new BufferedOutputStream(new FileOutputStream(logFile,true)), true);
  System.setOut(printStream);
  }
  catch (Exception e)
  {
    e.printStackTrace();
  }

I replaced your:

  fout.write(sb.toString().getBytes());

with:

  System.out.println(sb.toString().getBytes());

Is that where I went wrong?  I don't have much experience writing out to text files...  Should I be doing a different kind of write?

Thanks a lot,
BrianMc1958
0
 
sudhakar_koundinyaCommented:
   // All writes to this print stream are copied to two print streams
    public class TeeStream extends PrintStream {
        PrintStream out;
        public TeeStream(PrintStream out1, PrintStream out2) {
            super(out1);
            this.out = out2;
        }
        public void write(byte buf[], int off, int len) {
            try {
                super.write(buf, off, len);
                out.write(buf, off, len);
            } catch (Exception e) {
            }
        }
        public void flush() {
            super.flush();
            out.flush();
        }
    }



_______________________________________________________________


try {
        // Tee standard output

        PrintStream tee = null;
   

   
        // Tee standard error
        PrintStream err = new PrintStream(new FileOutputStream("test.log"));
        tee = new TeeStream(System.err, err);
   
        System.setErr(tee);
    } catch (FileNotFoundException e) {
    }
   


 while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
System.out.println(sb);
 sb=new StringBuffer();
 
}
  }

0
 
sudhakar_koundinyaCommented:
thanks :-)
0
 
sudhakar_koundinyaCommented:
for ur model

replace  sb.append(rs.getString(1)).append("\r\n");
  to
 sb.append(rs.getString(1));
0
All Courses

From novice to tech pro — start learning today.