Solved

Newbie question: write text file FAST?

Posted on 2004-09-29
7
475 Views
Last Modified: 2012-05-05
Dear Experts,

I'm using JDBC to return a large result set (>1,000,000 records...).  I want to write it out to a simple text file.  Getting the records from a remote server very fast.  But writing them out locally to a simple text file is very slow.

Is there a way in java to write text files FAST?  Like, can you somehow read the next 1000 results, store them in memory, and then write them out at once?

This is the code:

  Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    System.out.println(rs.getString(1));
  }
  con.close();

Thanks, Experts !

--BrianMc1958
0
Comment
Question by:BrianMc1958
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
7 Comments
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182006
StringBuffer sb=new StringBuffer();
int i=0;
 Connection con = DriverManager.getConnection("jdbc:as400://" + host, user, pwd);
  java.sql.Statement stmt = con.createStatement();
  String sql="SELECT * FROM MYLIB.BIGFILE";
  ResultSet rs = stmt.executeQuery(sql);
  while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1));
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
  con.close();
fout.close();
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182052
while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
 sb=new StringBuffer();
 
}
  }
0
 

Author Comment

by:BrianMc1958
ID: 12182371
Dear sudhakar:

Thanks, but it's not quite right yet.  My results are much faster, but garbled.

I did not include my full code.  I'm using this to set up the text file:

  File logFile = new File("C:\\JUNK\\BIGFILE.txt");
  try
  {
  PrintStream printStream =
      new PrintStream(new BufferedOutputStream(new FileOutputStream(logFile,true)), true);
  System.setOut(printStream);
  }
  catch (Exception e)
  {
    e.printStackTrace();
  }

I replaced your:

  fout.write(sb.toString().getBytes());

with:

  System.out.println(sb.toString().getBytes());

Is that where I went wrong?  I don't have much experience writing out to text files...  Should I be doing a different kind of write?

Thanks a lot,
BrianMc1958
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 14

Accepted Solution

by:
sudhakar_koundinya earned 125 total points
ID: 12182455
>> System.out.println(sb.toString().getBytes());
should be
 System.out.println(sb);
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182523
   // All writes to this print stream are copied to two print streams
    public class TeeStream extends PrintStream {
        PrintStream out;
        public TeeStream(PrintStream out1, PrintStream out2) {
            super(out1);
            this.out = out2;
        }
        public void write(byte buf[], int off, int len) {
            try {
                super.write(buf, off, len);
                out.write(buf, off, len);
            } catch (Exception e) {
            }
        }
        public void flush() {
            super.flush();
            out.flush();
        }
    }



_______________________________________________________________


try {
        // Tee standard output

        PrintStream tee = null;
   

   
        // Tee standard error
        PrintStream err = new PrintStream(new FileOutputStream("test.log"));
        tee = new TeeStream(System.err, err);
   
        System.setErr(tee);
    } catch (FileNotFoundException e) {
    }
   


 while(rs.next())
  {
    // System.out IS the text file.  The row is all one string.
    sb.append(rs.getString(1)).append("\r\n");
i++;
if(i%1000==0)
{
       i=0;
          fout.write(sb.toString().getBytes());
System.out.println(sb);
 sb=new StringBuffer();
 
}
  }

0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182532
thanks :-)
0
 
LVL 14

Expert Comment

by:sudhakar_koundinya
ID: 12182591
for ur model

replace  sb.append(rs.getString(1)).append("\r\n");
  to
 sb.append(rs.getString(1));
0

Featured Post

Salesforce Made Easy to Use

On-screen guidance at the moment of need enables you & your employees to focus on the core, you can now boost your adoption rates swiftly and simply with one easy tool.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
What's wrong with this code? 4 47
difference between sorce folder and folder in eclipise 3 65
spring maven example issues 3 109
netbeans can not run self signed app 8 38
By the end of 1980s, object oriented programming using languages like C++, Simula69 and ObjectPascal gained momentum. It looked like programmers finally found the perfect language. C++ successfully combined the object oriented principles of Simula w…
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
This video teaches viewers about errors in exception handling.
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question