Solved

Newbie question: write text file FAST?

Posted on 2004-09-29
7
440 Views
Last Modified: 2012-05-05
Dear Experts,

I'm using JDBC to return a large result set (>1,000,000 records...).  I want to write it out to a simple text file.  Getting the records from a remote server very fast.  But writing them out locally to a simple text file is very slow.

Is there a way in java to write text files FAST?  Like, can you somehow read the next 1000 results, store them in memory, and then write them out at once?

This is the code:

  Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    System.out.println(rs.getString(1));
  }
  con.close();

Thanks, Experts !

--BrianMc1958
0
Comment
Question by:BrianMc1958
  • 6
7 Comments
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182006
StringBuffer sb=new StringBuffer();
int i=0;
 Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1));
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
  con.close();
fout.close();
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182052
while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
0
 

Author Comment

by:BrianMc1958
ID: 12182371
Dear sudhakar:

Thanks, but it's not quite right yet.  My results are much faster, but garbled.

I did not include my full code.  I'm using this to set up the text file:

  File logFile = new File("C:\\JUNK\\BIGFILE.txt");
  try
  {
  PrintStream printStream =
      new PrintStream(new BufferedOutputStream(new FileOutputStream(logFile,true)), true);
  System.setOut(printStream);
  }
  catch (Exception e)
  {
    e.printStackTrace();
  }

I replaced your:

  fout.write(sb.toString().getBytes());

with:

  System.out.println(sb.toString().getBytes());

Is that where I went wrong?  I don't have much experience writing out to text files...  Should I be doing a different kind of write?

Thanks a lot,
BrianMc1958
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 14

Accepted Solution

by:
sudhakar_koundinya earned 125 total points
ID: 12182455
>> System.out.println(sb.toString().getBytes());
should be
 System.out.println(sb);
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182523
   // All writes to this print stream are copied to two print streams
    public class TeeStream extends PrintStream {
        PrintStream out;
        public TeeStream(PrintStream out1, PrintStream out2) {
            super(out1);
            this.out = out2;
        }
        public void write(byte buf[], int off, int len) {
            try {
                super.write(buf, off, len);
                out.write(buf, off, len);
            } catch (Exception e) {
            }
        }
        public void flush() {
            super.flush();
            out.flush();
        }
    }



_______________________________________________________________


try {
        // Tee standard output

        PrintStream tee = null;
   

   
        // Tee standard error
        PrintStream err = new PrintStream(new FileOutputStream("test.log"));
        tee = new TeeStream(System.err, err);
   
        System.setErr(tee);
    } catch (FileNotFoundException e) {
    }
   


 while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
System.out.println(sb);
 sb=new StringBuffer();
 
}
  }

0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182532
thanks :-)
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182591
for ur model

replace  sb.append(rs.getString(1)).append("\r\n");
  to
 sb.append(rs.getString(1));
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

After being asked a question last year, I went into one of my moods where I did some research and code just for the fun and learning of it all.  Subsequently, from this journey, I put together this article on "Range Searching Using Visual Basic.NET …
Java had always been an easily readable and understandable language.  Some relatively recent changes in the language seem to be changing this pretty fast, and anyone that had not seen any Java code for the last 5 years will possibly have issues unde…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now