Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Newbie question: write text file FAST?

Posted on 2004-09-29
7
Medium Priority
?
483 Views
Last Modified: 2012-05-05
Dear Experts,

I'm using JDBC to return a large result set (>1,000,000 records...).  I want to write it out to a simple text file.  Getting the records from a remote server very fast.  But writing them out locally to a simple text file is very slow.

Is there a way in java to write text files FAST?  Like, can you somehow read the next 1000 results, store them in memory, and then write them out at once?

This is the code:

  Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    System.out.println(rs.getString(1));
  }
  con.close();

Thanks, Experts !

--BrianMc1958
0
Comment
Question by:BrianMc1958
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
7 Comments
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182006
StringBuffer sb=new StringBuffer();
int i=0;
 Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1));
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
  con.close();
fout.close();
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182052
while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
0
 

Author Comment

by:BrianMc1958
ID: 12182371
Dear sudhakar:

Thanks, but it's not quite right yet.  My results are much faster, but garbled.

I did not include my full code.  I'm using this to set up the text file:

  File logFile = new File("C:\\JUNK\\BIGFILE.txt");
  try
  {
  PrintStream printStream =
      new PrintStream(new BufferedOutputStream(new FileOutputStream(logFile,true)), true);
  System.setOut(printStream);
  }
  catch (Exception e)
  {
    e.printStackTrace();
  }

I replaced your:

  fout.write(sb.toString().getBytes());

with:

  System.out.println(sb.toString().getBytes());

Is that where I went wrong?  I don't have much experience writing out to text files...  Should I be doing a different kind of write?

Thanks a lot,
BrianMc1958
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 14

Accepted Solution

by:
sudhakar_koundinya earned 500 total points
ID: 12182455
>> System.out.println(sb.toString().getBytes());
should be
 System.out.println(sb);
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182523
   // All writes to this print stream are copied to two print streams
    public class TeeStream extends PrintStream {
        PrintStream out;
        public TeeStream(PrintStream out1, PrintStream out2) {
            super(out1);
            this.out = out2;
        }
        public void write(byte buf[], int off, int len) {
            try {
                super.write(buf, off, len);
                out.write(buf, off, len);
            } catch (Exception e) {
            }
        }
        public void flush() {
            super.flush();
            out.flush();
        }
    }



_______________________________________________________________


try {
        // Tee standard output

        PrintStream tee = null;
   

   
        // Tee standard error
        PrintStream err = new PrintStream(new FileOutputStream("test.log"));
        tee = new TeeStream(System.err, err);
   
        System.setErr(tee);
    } catch (FileNotFoundException e) {
    }
   


 while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
System.out.println(sb);
 sb=new StringBuffer();
 
}
  }

0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182532
thanks :-)
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182591
for ur model

replace  sb.append(rs.getString(1)).append("\r\n");
  to
 sb.append(rs.getString(1));
0

Featured Post

Build and deliver software with DevOps

A digital transformation requires faster time to market, shorter software development lifecycles, and the ability to adapt rapidly to changing customer demands. DevOps provides the solution.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn different types of Android Layout and some basics of an Android App.
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.
Suggested Courses

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question