• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 910
  • Last Modified:

Java Runtime.exec() Command Execution Problem

Ok guys, help is appreciated in advance.  I'm developing a backup system in Java for interenal use in our office, part of which backs up a certain set of MySQL.  For this portion, I'm using the Runtime class in Java to execute the command "mysqldump" with the appropriate parameters on the Linux database server.  



The code first builds a list of databases to dump by reading lines from a text file.  Then, it creates a new thread for each database, each which using the Runttime.getRuntime().exec() method to execute the mysqldump.  The output is read, and then stored in a seperate .sql file for each database.



THE PROBLEM:

The code executes as desired, however some of the mysqldumps do not complete before the program exits.  If you open some of the .sql files after the program terminates, they are incomplete, meaning the data did not finish dumping for some reason.  It appears to only affect the large databases, but with a small 5mb database I may end up with more output than a large 20mb database.

The program is setup to log the error stream of the command execution, but nothing is being written to the log, so I assume there are no errors.  Also, I modified the program to print the command it is executing instead of actually executing it, and when I run the printed command by hand, it works fine.



THE CODE:


--- FILE:  MySQLDBBackup.java ---

import java.util.Stack;
import java.io.*;



public class MySQLDBBackup {
   
    public MySQLDBBackup() {
    }
   
    public static void main(String[] args) {
       
        String database = "";
        Stack dbList = new Stack();
        String curdb = "";

        File listfile = new File("databases.txt");
        //read the lines from databases.txt and issue a command for
        try {
            BufferedReader fr = new BufferedReader(new FileReader(listfile));
           
          
            database = fr.readLine();
            while (database != null) {
                dbList.push(database);
            database = fr.readLine();
            }
           
        } catch (FileNotFoundException e) {
            System.out.println("Error while reading databases.txt:" + e.getMessage());
        } catch (IOException ioe) {
            System.out.println("Error while reading databases.txt: " + ioe.getMessage());
        }
       
       
        for (int x = 0; x < dbList.size(); x++) {
            curdb = (String)dbList.pop();
            new Thread(new DBDumper(curdb,new File("/var/dbbackups/"+ curdb + ".sql"))).start();
        }
       
    }
   
}








--- FILE: DBDumper.java ----------


import java.io.*;


public class DBDumper implements Runnable {

      private String dbname = "";
      private File outputfile = null;
      
      public DBDumper(String database, File outputfile) {
            this.dbname = database;
            this.outputfile = outputfile;
      }

      public void run () {
      
            String curdb =  this.dbname;
            try {
                System.out.println("Backing up database "+ curdb);
            String[] command = {"mysqldump", "-uroot", curdb};
            Process p = Runtime.getRuntime().exec(command);
            
            //write the output to a file
            BufferedReader pout= new BufferedReader(new InputStreamReader(p.getInputStream()));
            BufferedReader perr= new BufferedReader(new InputStreamReader(p.getErrorStream()));
            BufferedWriter logwriter = new BufferedWriter(new FileWriter(this.outputfile));
            BufferedWriter errorwriter = new BufferedWriter(new FileWriter("error.log"));
            String output = pout.readLine();
            while (output != null) {
                  logwriter.write(output + "\n");
                  output = pout.readLine();
            }
            
            String error = perr.readLine();
            while (error != null) {
                  errorwriter.write(error + "\n");
                  error = perr.readLine();
            }            
                  
            p.waitFor();
               
            } catch (IOException ioe) {
                System.out.println("\t...Error while executing MySQL Dump command: " + ioe.getMessage());
         } catch (InterruptedException ie) {
               System.out.println("\t...Error: Thread Interrupted!");
         }
      
      }

}
0
AJBrown
Asked:
AJBrown
  • 6
  • 4
  • 3
  • +1
1 Solution
 
Mig-OCommented:
Try reading the inputStream from an executing program always !!!
If the inputBuffer is full, the called external programm will stop working, so leaving you with half dumps!
0
 
CEHJCommented:
Try it with this:

String[] command = {"mysqldump", "-uroot", curdb};

RunAsync.main(command);


SNIP==================================================

import java.io.*;

/**
 *  Description of the Class
 *
 * @author     CEHJ
 * @created    23 February 2004
 */
public class RunAsync {

  /**
   *  Description of the Method
   *
   * @param  args  Description of the Parameter
   */
  public static void main(String args[]) {

    try {

      if (args.length < 1) {
        System.out.println("Usage: java RunAsync <command string>");
        System.exit(-1);
      }
      Process pro = null;
      if (args.length > 1) {
        pro = Runtime.getRuntime().exec(args);
      }
      else {
        pro = Runtime.getRuntime().exec(args[0]);
      }
      InputStream error = pro.getErrorStream();
      InputStream output = pro.getInputStream();
      Thread err = new Thread(new OutErrReader(error));
      Thread out = new Thread(new OutErrReader(output));
      out.start();
      err.start();
      pro.waitFor();
    }
    catch (java.io.IOException e) {
      e.printStackTrace();
    }
    catch (java.lang.InterruptedException e) {
      e.printStackTrace();
    }

  }


  /**
   *  Description of the Class
   *
   * @author     CEHJ
   * @created    23 February 2004
   */
  static class OutErrReader implements Runnable {
    InputStream is;


    /**
     *Constructor for the OutErrReader object
     *
     * @param  is  Description of the Parameter
     */
    public OutErrReader(InputStream is) {
      this.is = is;
    }


    /**
     *  Main processing method for the OutErrReader object
     */
    public void run() {
      try {
        BufferedReader in = new BufferedReader(new InputStreamReader(is));
        String temp = null;
        while ((temp = in.readLine()) != null) {
          System.out.println(temp);
        }
        is.close();
      }
      catch (Exception e) {
        e.printStackTrace();
      }
    }
  }
}

0
 
Mig-OCommented:
In your code, i would try to read stdin and stderr parallel, not one after the other. Do this by checking the .available()-Method the the streams, sleeping a while, then polling again. Or start streamreader threads.
Maybe it helps, if you read the stderrStream before the stdinStream.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
objectsCommented:
Heres some code you can use to handle reading the process output:

http://www.objects.com.au/java/examples/util/ConsoleExec.do
0
 
AJBrownAuthor Commented:
sorry guys, I've been backed up and haven't gotten to these yet.  I will try the solutions and get back to you.
0
 
AJBrownAuthor Commented:
ok guys, I used the suggestions above and I'm still having either the same cut-off problems, or an entirely different problem.  Just to reinterate, the execution returns MOST of the output from mysqldump but not all of it.  The actual ammount of output retrieved varies.

Here is the code as it stands now, and this seems to be working the BEST of what I've tried:


--------------------------------
AUTOREADER Class
--------------------------------
/*
*  Copyright (c) 2001-2004 by Objects Pty Ltd
*/

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.util.ArrayList;
import java.util.Iterator;


public class AutoReader implements Runnable
{
        private BufferedReader In = null;

        private ArrayList Listeners = new ArrayList();

        /**
        *  Constructor
        *  @param in stream to read, line by line
        */

        public AutoReader(InputStream in)
        {
                this(new InputStreamReader(in));
        }

        /**
        *  Constructor
        *  @param in reader to read, line by line
        */

        public AutoReader(Reader in)
        {
                In = new BufferedReader(in);
        }

        /**
        *  Adds listener interested in progress of reading
        *  @param listener listener to add
        */

        public void addListener(Listener listener)
        {
                Listeners.add(listener);
        }

        /**
        *  Removes listener interested in progress of reading
        *  @param listener listener to remove
        */

        public void removeListener(Listener listener)
        {
                Listeners.remove(listener);
        }

        /**
        *  Handles reading from stream until eof, notify registered listeners of progress.
        */

        public void run()
        {
                try
                {
                        String line = null;
                        while (null!=(line = In.readLine()))
                        {
                                fireLineRead(line);
                        }
                }
                catch (IOException ex)
                {
                        fireError(ex);
                }
                finally
                {
                        fireEOF();
                }
        }

        /**
        *  Interface listeners must implement
        */

        public interface Listener
        {
                /**
                *  Invoked after each new line is read from stream
                *  @param reader where line was read from
                *  @param line line read
                */

                public void lineRead(AutoReader reader, String line);

                /**
                *  Invoked if an I/O error occurs during reading
                *  @param reader where error occurred
                *  @param ex exception that was thrown
                */

                public void error(AutoReader reader, IOException ex);

                /**
                *  Invoked after EOF is reached
                *  @param reader where EOF has occurred
                */

                public void eof(AutoReader reader);
        }

        /**
        *  Notifies registered listeners that a line has been read
        */

        private void fireLineRead(String line)
        {
                Iterator i = Listeners.iterator();
                while (i.hasNext())
                {
                        ((Listener)i.next()).lineRead(this, line);
                }
        }

        /**
        *  Notifies registered listeners that an error occurred during reading
        */

        private void fireError(IOException ex)
        {
                Iterator i = Listeners.iterator();
                while (i.hasNext())
                {
                        ((Listener)i.next()).error(this, ex);
                }
        }

        /**
        *  Notifies registered listeners that EOF has been reached
        */

        private void fireEOF()
        {
                Iterator i = Listeners.iterator();
                while (i.hasNext())
                {
                        ((Listener)i.next()).eof(this);
                }
        }

}


------------------------------------------------
 DBDumper Class
-------------------------------------------------
import java.io.*;


public class DBDumper implements AutoReader.Listener {

    private String dbname = "";
    private File outputfile = null;
    private BufferedWriter writer;

    public DBDumper(String database, File outputfile) {
        this.dbname = database;
        this.outputfile = outputfile;
        try {
            this.writer = new BufferedWriter(new FileWriter(this.outputfile));
        } catch (IOException ioe) {
            ioe.printStackTrace();
        }
    }

    public synchronized void dump() {
        try
                {
                        String[] command = {"mysqldump", "-u<username>","-p<password>", this.dbname};

                        Process p = Runtime.getRuntime().exec(command);
                        AutoReader in = new AutoReader(p.getInputStream());
                        in.addListener(this);
                        AutoReader err = new AutoReader(p.getErrorStream());
                        err.addListener(this);
                        new Thread(in).start();
                        new Thread(err).start();
                        p.waitFor();

                }
                catch (Exception ex)
                {
                        ex.printStackTrace();
        }
    }


    public void lineRead(AutoReader reader, String line)
    {
        try {
            this.writer.write(line);
        } catch (IOException ioe) {
            System.out.print("IOException while dumping " + this.dbname +
                             ":" + ioe.getMessage());
        } catch (Exception e) {
            System.out.print("Exception while dumping " + this.dbname +
                             ":" + e.getMessage());
        }
    }

    public void error(AutoReader reader, IOException ex)
    {
            ex.printStackTrace();
    }

    public void eof(AutoReader reader)
    {
        //do nothing
    }


}


-------------------------------------------------
MAIN (MySQLDBBackup) Class
-------------------------------------------------
import java.util.Stack;
import java.io.*;


public class MySQLDBBackup {

    public MySQLDBBackup() {
    }

    public static void main(String[] args) {

        String database = "";
        Stack dbList = new Stack();
        String curdb = "";

        File listfile = new File("databases.txt");
        //read the lines from databases.txt and issue a command for
        try {
            BufferedReader fr = new BufferedReader(new FileReader(listfile));

            database = fr.readLine();
            while (database != null) {
                dbList.push(database);
                database = fr.readLine();
            }

        } catch (FileNotFoundException e) {
            System.out.println("Error while reading databases.txt:" +
                               e.getMessage());
        } catch (IOException ioe) {
            System.out.println("Error while reading databases.txt: " +
                               ioe.getMessage());
        }

        for (int x = 0; x < dbList.size(); x++) {
            curdb = (String)dbList.pop();
            DBDumper dbd =  new DBDumper(curdb,new File("/var/dbbackups/" + curdb + ".sql"));
            dbd.dump();
        }

    }

}

0
 
AJBrownAuthor Commented:
Increased points from 75 to 130
0
 
objectsCommented:
try closing the wrioter on eof:

    public void eof(AutoReader reader)
    {
        writer.close();
    }
0
 
Mig-OCommented:
You have to delay reading from the stream, until a listener registers. Or you pass the initial lsitener with the construktor. Do you see the small gap, where the first events get fired to nomansland because you register the listeners AFTER creating the readers?
0
 
Mig-OCommented:
ups, sorry, you start the thread after registering the listeners... forget my previous comment ;)
0
 
AJBrownAuthor Commented:
objects,

when I close the writer on eof, it gives me an IOException: Stream Closed error repeatedly.
0
 
AJBrownAuthor Commented:
more points!
0
 
objectsCommented:
Thinks thats cause the listener gets notified of eof twice, once for each stream so closes the file twice

try:

    public void eof(AutoReader reader)
    {
        if (writer!=null)
        {
            writer.close();
            writer = null;
        }
    }
0
 
AJBrownAuthor Commented:
objects,

Thanks much for you help!  I added the modified code for closing the writer, which prevented the dump from apppear to "cut off", but I then noticed that all of the data wasn't being retrieved.  In otherwords, all of the data retrieved was full and valid, but it wasn't ALL of the records).  So, I then went and commented out the line which registers the DBDumper object as a listener for the error stream reader thread.  That did the trick -- my program's output now matches the output when running the command manually.

Thank you much for your help!
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 6
  • 4
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now