Link to home
Start Free TrialLog in
Avatar of scottcw
scottcwFlag for United Kingdom of Great Britain and Northern Ireland

asked on

File filter program that writes a log for one directory layer but no more, I have code for doing extra layers although cannot encorpate it

For this program I need to search multiple directory layers and then if the file is in a directory then the file name will have the directory name added to it. This is the code for the directory scanning so you can get an idea of what I mean:
[code]
public void processDirectory(String directoryName)
    {
        try
        {
            File dir = new File(directoryName);

            File[] files = dir.listFiles();

            for (int i = 0; i < files.length; i++)
            {
                if (files[i].isDirectory())
                {
                    processDirectory(directoryName + "/" + files[i].getName());
                }
                  }
        } catch (Exception e)
        {
            System.err.println("Problem with " + directoryName);
            System.exit(0);
        }
    }
[/code]

This is what I have managed to do so far, it includes comments. This is obviously not all of the code - I can post all of the code if you think it will help.

[code]
public void process(String[] argStrings, File files)
{
   
    //lots of code for validation of command line (the directory, copydate, fileextension1 and fileexttesnion 2 are entered on the command line)
   
    File folder = new File(name);
    processDirectory(folder); // This passes the folder (argstrings0 to the processdirectory method)

    MultiFilter multiFilter = new MultiFilter(fileExtensionOne);
      File[] htmlFiles = files.listFiles(multiFilter); //when this was folder.listFiles(multiFilter) it worked for one directory level

      multiFilter.setExtension(fileExtensionTwo);
    File[] phpFiles = files.listFiles(multiFilter);

    File[] filesToLog = mergeFiles(htmlFiles, phpFiles);
    log(filesToLog, archiveLimit, copyLimit);
  }

  public File processDirectory(File folder)
  {
    try
    {
      File[] files = folder.listFiles();

      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        { // once this method has found directories in the directories it recurses,
          processDirectory(directoryName + "\\" + files[i].getName()); // Add the directories to the filename so that they appear in the log

          return files[i];// here i am trying to get all the files in the directory (s) back to the method above so that they can be filtered
                          // i.e. either php or html as eneterd on the command line
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
  }
 
  // Also I use this to run the application, this might be wrong also:
 
  import java.io.*;
 
  public class CrawlerApp
  {
    public static void main(String[] argStrings, File files)
    {
        Crawler crawler = new Crawler();
      crawler.process(argStrings, files);
    }
}
 
//At the moment my program does not compile and I get these errors:

//C:\Documents and Settings\User\Desktop\Crawler.java:209: processDirectory(java.io.File) in Crawler cannot be applied to (java.lang.String)
//processDirectory(folder + "\\" + files[i].getName());

[/code]
Thank you
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image

>>processDirectory(folder + "\\" + files[i].getName());

should be

processDirectory(new File(folder,files[i].getName()));
Avatar of scottcw

ASKER

Ok this gets rid of that compile error, cheers.
Although now I get this compile error:

C:\Crawler.java:220: missing return statement
  }

This is at the end of the processDirectory method.

Thanks
Follow

>>System.err.println("Problem with directory " + folder);

with

return null;
Avatar of scottcw

ASKER

Yes this is what I tried a while ago, and it does compile although does not run now, I get an error like this at command line:

java.lang.nosuchmethoderror: main

When the method looks like this:

 public File processDirectory(File folder)
  {
    try
    {
      File[] files = folder.listFiles();

      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        {
          processDirectory(new File(folder,files[i].getName())); // Add the directories to the filename so that they appear in the log

          return files[i];
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
    return null;
  }

Cheers
>>public static void main(String[] argStrings, File files)

should be

public static void main(String[] argStrings)
Avatar of scottcw

ASKER

Making progress.

I get one compile error:

C:\Documents and Settings\User\Desktop\CrawlerApp.java:6: process(java.lang.String[],java.io.File) in Crawler cannot be applied to (java.lang.String[])
    crawler.process(argStrings);

When the App file looks like this:

public class CrawlerApp
{
  public static void main(String[] argStrings)
  {
      Crawler crawler = new Crawler();
    crawler.process(argStrings);
  }
}

Thanks
No, you need to pass the directory.Somehting like:

 Crawler crawler = new Crawler();
 crawler.process(argStrings, new File("."));
Avatar of scottcw

ASKER

I tried what you said and i get a compile error:

public class CrawlerApp
{
  public static void main(String[] argStrings)
  {
      Crawler crawler = new Crawler();
    crawler.process(argStrings, new File("."));
  }
}

Gives:

C:\Documents and Settings\User\Desktop\CrawlerApp.java:6: cannot find symbol
symbol  : class File
location: class CrawlerApp
    crawler.process(argStrings, new File("."));
                                    ^
1 error

Tool completed with exit code 1

I have not done much work on App files before, I just use the same format for each program therefore I dont know what this does.
Cheers
Avatar of scottcw

ASKER

I did not import
sorry
Avatar of scottcw

ASKER

Well it compiles, it runs when I input the directory, copydate and extensions althrough it says it has finished and then I read the logs and they are empty for some reason.

Cheers
Try starting at a different directory than the current directory
Avatar of scottcw

ASKER

The program should work when the directory is in the same directory as the program (this worked when it only did one level).

I tried somedifferent file extensions and i got some output in the log, but these were files on my desktop where the Crawler and CrawlerApp are.

So I think at the moment it is doing the files on the same level as the directory specified and not directories inside the directory.

Cheers
I'm not sure what you're trying to do with this code
Avatar of scottcw

ASKER

This is some of the javadoc in my program:

/**
 * The Crawler class executes six methods that identify files which could potentially
 * be archived and copied in a directory specified on the command line. A file is to
 * be archived if the last modified date is more than 100 days ago. A file is to be copied
 * if it has been modified since a given time. Only nominated extensions are to be
 * considered (php and html).
 * <p>
 * The output from this class is two files (called archive.txt and copy.txt)
 * containing the names of files to be archived, and those which may be copied. The
 * names will follow the convention specified in the create application.
 * <p>
 * $ java CrawlerApp [directory name] [copy date] [extension one] [extension two]
 *
 * @author        John

Do you want me to post the whole code to you, or email it maybe, then you can compile it yourself to get a better understanding?
Cheers
John
Avatar of scottcw

ASKER

I am recursing over all the directories, but only reporting on the top level.

I am not using the argument "files" to process().

I went to the Java forums and they suggested this, I have been trying for a while to implement this but with no luck. Please could you have a go. Once this is done I think it will work:

1) start recursing (do processDirectory())
2) at each directory, call process() to process that directory.

Don't call processDirectory() from process(). It should be the other way around.

This is the code that needs to be changed:

public void process(String[] argStrings, File files)
  {
// validation code for the command line
    File folder = new File(name);
    processDirectory(folder);
 
    MultiFilter multiFilter = new MultiFilter(fileExtensionOne);
      File[] htmlFiles = files.listFiles(multiFilter);
 
      multiFilter.setExtension(fileExtensionTwo);
    File[] phpFiles = files.listFiles(multiFilter);
 
    File[] filesToLog = mergeFiles(htmlFiles, phpFiles);
    log(filesToLog, archiveLimit, copyLimit);
  }
 
  public File processDirectory(File folder)
  {
    try
    {
      File[] files = folder.listFiles();
 
      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        {
          processDirectory(new File(folder,files[i].getName())); // Add the directories to the filename so that they appear in the log
 
          return files[i];
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
    return null;
  }


Thanks
Your code looks more or less alright, but the sole functionality of processDirectory, IMO is to recurse subdirectories. If so,  all it needs to be is

public void processDirectory(File folder)
  {
    try
    {

        if (folder.isDirectory())
        {
          process(folder); // Do your processing
        }

      File[] files = folder.listFiles();
 
      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        {
          processDirectory(files[i]); // Add the directories to the filename so that they appear in the log
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
  }
Avatar of scottcw

ASKER

I changed to what you said, and I only get one compile error now which is good. Is this something to do with not passing enough values back to process?

java:184: process(java.lang.String[],java.io.File) in Crawler cannot be applied to (java.io.File)
        process(folder); // Do your processing
        ^
1 error

Tool completed with exit code 1

Cheers
Yes. What's the String parameter meant to be
Avatar of scottcw

ASKER

The process method takes arguements from the command line e.g. directory = argstrings[0], copydate = argstrings[1], extension 1 and 2 = argstrings[2] and [3]:

public void process(String[] argStrings, File files)
{
Thanks
>>Don't call processDirectory() from process(). It should be the other way around.

If you want the above to be the case, you will have to forward that String[] *through* processDirectory *to* process
Avatar of scottcw

ASKER

Something like this?

public void process(String[] argStrings, File files)
  {
//code not shown
  }

 public void processDirectory(File folder, String[] argStrings)
  {
    try
    {
      if (folder.isDirectory())
      {
        process(argStrings, folder); // Do your processing
      }

      File[] files = folder.listFiles();

      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        {
          processDirectory(files[i], argStrings);
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
  }

It compiles, but still has the same problem, it only does one level.
Cheers
Avatar of scottcw

ASKER

Oh and I forgot to say that this has been removed from process:
processDirectory(folder);
Put some debug code in before

>>processDirectory(files[i], argStrings);

such as

System.out.println("Calling processDirectory for directory " + files[i]);

and tell me what it does
Avatar of scottcw

ASKER

It does nothing.
Avatar of scottcw

ASKER

This must mean that it is still only doing oe level as indicated by the logs.
cheers
Oh - please put the same sort of thing before

>>process(argStrings, folder); // Do your processing
Avatar of scottcw

ASKER

I get these errors with this code:

C:\Documents and Settings\User\Desktop\Computing\Crawler.java:184: cannot find symbol
symbol  : variable files
location: class Crawler
            System.out.println("Calling processDirectory for directory " + files[i]);
                                                                               ^
C:\Documents and Settings\User\Desktop\Computing\Crawler.java:184: cannot find symbol
symbol  : variable i
location: class Crawler
            System.out.println("Calling processDirectory for directory " + files[i]);
                                                                                     ^
2 errors
Code:
public void process(String[] argStrings, File files)
  {
//more code
    File folder = new File(directory);
      process(argStrings, folder);

    MultiFilter multiFilter = new MultiFilter(fileExtensionOne);
      File[] extensionOne = files.listFiles(multiFilter);

      multiFilter.setExtension(fileExtensionTwo);
    File[] extensionTwo = files.listFiles(multiFilter);

    File[] filesToLog = mergeFiles(extensionOne, extensionTwo);
    log(filesToLog, archiveLimit, copyLimit);
  }

  public void processDirectory(File folder, String[] argStrings)
  {
    try
    {
      if (folder.isDirectory())
      {
            System.out.println("Calling processDirectory for directory " + files[i]);
        process(argStrings, folder); // Do your processing
      }

      File[] files = folder.listFiles();

      for (int i = 0; i < files.length; i++)
      {
        if (files[i].isDirectory())
        {
          processDirectory(files[i], argStrings);
        }
      }
    }
    catch (Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
  }

Cheers
Avatar of scottcw

ASKER

I tried this also:

System.out.println("Calling processDirectory for directory " + folder);

And still nothing.
Cheers
>>I tried this also:

That's the one. Are you saying you are getting *no output at all*?
Avatar of scottcw

ASKER

On the command line nothing prints i.e. the ""Calling processDirectory for directory " + folder" does not show.

Get one level of directories in the logs though - still the same.
Then that method is not being called at all. Got to go for a while
Avatar of scottcw

ASKER

Ok cheers for the help
>>Then that method is not being called at all.

(Unless of course it's not being called in the first place with a directory - which would be quite usesless)
Avatar of scottcw

ASKER

Right now I have got it to scan through every directory and file (each is displayed on the command line).

Some of the logs are being written properly also (deeper directories).

This is my latest code:

App file:

import java.io.*;

public class CrawlerApp
{
  public static void main(String[] argStrings)
  {
      Crawler crawler = new Crawler();
    crawler.processDirectory(new File("."), argStrings);
    //crawler.process(argStrings, new File("."));
  }
}

This is process and processDirectory:
public void process(String[] argStrings, File files)
  {
    File folder = new File(directory);

    MultiFilter multiFilter = new MultiFilter(fileExtensionOne);
      File[] extensionOne = files.listFiles(multiFilter);

      multiFilter.setExtension(fileExtensionTwo);
    File[] extensionTwo = files.listFiles(multiFilter);

    File[] filesToLog = mergeFiles(extensionOne, extensionTwo);
    log(filesToLog, archiveLimit, copyLimit);

      processDirectory(folder, argStrings);
  }

  public File processDirectory(File folder, String[] argStrings)
  {
    try
    {
      File[] files = folder.listFiles();

      for(int i = 0; i < files.length; i++)
      {
            if(files[i].isFile())
        {
              System.out.println("Files " + files[i]);
            }
        else if(files[i].isDirectory())
        {
          processDirectory(files[i], argStrings);
          System.out.println("Directories " + files[i]);
          process(argStrings, files[i]);
          }
      }
    }
    catch(Exception e)
    {
      System.err.println("Problem with directory " + folder);
      System.exit(0);
    }
    return null;
  }
Cheers
Well it looks like you need to implement 'author John's' functionality

e.g. 'argStrings' is currently completely unused ...
Avatar of scottcw

ASKER

Sorry its taken a while, but I have actually managed to get the program working perfectly with HTML formatted output.

I think if you were to give me some info on functional and structural testing then it would be appropriate to give the 200 points.

So far for functional testing I have tried different parameters on the command line (the program takes input from the command line) e.g. java CrawlerApp [directory name] [copy date] [extension one] [extension two] would be input or the actual input would be java CrawlerApp knuth 2005-12-28 php html.

The program then searches all files in that directroy even if they are in lower levels of directories and outputs the php and html files that have a last modified date of 2005-12-28.

I was wondering if there are any more functional tests that i could do for a program like this?

I am very new to structural testing - would I test try and catch blocks and for for loops would i just show that it printed every iteration to the command line for instance? Any tips on this would be appreciated.

Thank you
ASKER CERTIFIED SOLUTION
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
:-)