Solved

Newbie question: write text file FAST?

Posted on 2004-09-29
7
458 Views
Last Modified: 2012-05-05
Dear Experts,

I'm using JDBC to return a large result set (>1,000,000 records...).  I want to write it out to a simple text file.  Getting the records from a remote server very fast.  But writing them out locally to a simple text file is very slow.

Is there a way in java to write text files FAST?  Like, can you somehow read the next 1000 results, store them in memory, and then write them out at once?

This is the code:

  Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    System.out.println(rs.getString(1));
  }
  con.close();

Thanks, Experts !

--BrianMc1958
0
Comment
Question by:BrianMc1958
  • 6
7 Comments
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182006
StringBuffer sb=new StringBuffer();
int i=0;
 Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1));
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
  con.close();
fout.close();
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182052
while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
0
 

Author Comment

by:BrianMc1958
ID: 12182371
Dear sudhakar:

Thanks, but it's not quite right yet.  My results are much faster, but garbled.

I did not include my full code.  I'm using this to set up the text file:

  File logFile = new File("C:\\JUNK\\BIGFILE.txt");
  try
  {
  PrintStream printStream =
      new PrintStream(new BufferedOutputStream(new FileOutputStream(logFile,true)), true);
  System.setOut(printStream);
  }
  catch (Exception e)
  {
    e.printStackTrace();
  }

I replaced your:

  fout.write(sb.toString().getBytes());

with:

  System.out.println(sb.toString().getBytes());

Is that where I went wrong?  I don't have much experience writing out to text files...  Should I be doing a different kind of write?

Thanks a lot,
BrianMc1958
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 
LVL 14

Accepted Solution

by:
sudhakar_koundinya earned 125 total points
ID: 12182455
>> System.out.println(sb.toString().getBytes());
should be
 System.out.println(sb);
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182523
   // All writes to this print stream are copied to two print streams
    public class TeeStream extends PrintStream {
        PrintStream out;
        public TeeStream(PrintStream out1, PrintStream out2) {
            super(out1);
            this.out = out2;
        }
        public void write(byte buf[], int off, int len) {
            try {
                super.write(buf, off, len);
                out.write(buf, off, len);
            } catch (Exception e) {
            }
        }
        public void flush() {
            super.flush();
            out.flush();
        }
    }



_______________________________________________________________


try {
        // Tee standard output

        PrintStream tee = null;
   

   
        // Tee standard error
        PrintStream err = new PrintStream(new FileOutputStream("test.log"));
        tee = new TeeStream(System.err, err);
   
        System.setErr(tee);
    } catch (FileNotFoundException e) {
    }
   


 while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
System.out.println(sb);
 sb=new StringBuffer();
 
}
  }

0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182532
thanks :-)
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182591
for ur model

replace  sb.append(rs.getString(1)).append("\r\n");
  to
 sb.append(rs.getString(1));
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How to get all the API from website? 11 84
runtime exception 2 27
jboss wildfly 10.1 10 80
Java Restore security prompts not working 10 7
Are you developing a Java application and want to create Excel Spreadsheets? You have come to the right place, this article will describe how you can create Excel Spreadsheets from a Java Application. For the purposes of this article, I will be u…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Viewers learn about the “while” loop and how to utilize it correctly in Java. Additionally, viewers begin exploring how to include conditional statements within a while loop and avoid an endless loop. Define While Loop: Basic Example: Explanatio…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question