[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 292
  • Last Modified:

Split the files

Hi Chris (Wint),

If you remember you helped me few months ago with the following code:
Now I got a problem whenever the number of the files to be read and spilt is more than for example 150 or the file has so many rows (more than a million lines) it's going to almost freeze the system by using memory and system will hang. Would you please help me to fix this problem?

Thank you so much,
Mark

using System;
using System.Collections;
using System.Collections.Specialized;
using System.IO;
using System.Threading;
using System.Xml;

namespace MultiThreadFileReader
{
      /////////////////////////////////////////////////////
      // Program Class: Launches the program.
      /////////////////////////////////////////////////////
      internal class Program
      {
            private static void Main(string[] args)
            {
                  try
                  {
                        new MultiThreadFileReader(args);
                  }
                  catch (System.Exception e)
                  {
                        Console.WriteLine("An exception was caught whilst running the MultiThreadFileReader:"
                              + Environment.NewLine + e.ToString());
                  }
                  ////---Console.WriteLine("Press ENTER to exit.");
                  ///----Console.ReadLine();
            }
      }


      /////////////////////////////////////////////////////
      // MultiThreadFileReader Class: Runs the reading etc.
      /////////////////////////////////////////////////////
      internal class MultiThreadFileReader
      {
            /// <summary>Run the MultiThreadFileReader.</summary>
            /// <param name="args">The directory name to read files from.</param>
            internal MultiThreadFileReader(string[] args)
            {
                  DateTime startTime = DateTime.Now;
                  string dirToRead = null;
                  string settingsFile = null;
                  int numThreads = -1;

                  #region Initial Checks
                  //Usage:
                  //  MyProg.exe -d <dirToRead> -s <settingsFile> [-t ##]
                  //We also need to check that the number of args is correct, we *need* a settings file and *need* a dir to read.
                  //-t / -threads is the only arg we don't need (as we can default it).

                  //NEW!!

                  if (args.Length < 4)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  for (int i = 0; i < args.Length; i++)
                  {
                        switch (args[i])
                        {
                                    //The user can use either -t or -threads to set the number of threads to use.
                              case "-t":
                              case "-threads":
                                    i++;
                                    numThreads = Convert.ToInt32(args[i]);
                                    break;
                                    //The user can use -d or -dir to indicate the directory to read in from.
                              case "-d":
                              case "-dir":
                                    i++;
                                    dirToRead = args[i];
                                    break;
                                    //The user can use -s or -settings to indicate the location of the settings file to read in.
                              case "-s":
                              case "-settings":
                                    i++;
                                    settingsFile = args[i];
                                    break;
                        }
                  }

                  //Have the settings file and directory to read from been set?
                  if (settingsFile == null || settingsFile.Length == 0 || dirToRead == null || dirToRead.Length == 0)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  //Does the settings file exist?
                  if (!File.Exists(settingsFile))
                  {
                        Console.WriteLine("The settings file given: " + settingsFile + ", doesn't exist, I can't continue!");
                        return;
                  }

                  //Does the directory to read exist?
                  if (!Directory.Exists(dirToRead))
                  {
                        Console.WriteLine("This Path (" + dirToRead + ") does not exists.");
                        return;
                  }

                  //Get the files.
                  string[] fileEntries = Directory.GetFiles(dirToRead);
                  if (fileEntries == null || fileEntries.Length == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files, there is nothing for me to read!!");
                        return;
                  }

                  //Create a new StringCollection (an ArrayList, but specialized to handle only strings),
                  //this means we can avoid casting and use it type-safely.
                  StringCollection filesToRead = new StringCollection();

                  //If the filename has '.log' anywhere in it, it will be read.
                  //i.e. my.LOG.txt, my.LOG, my.LOGarithm, etc. Not just the extension!!
                  //This can be changed by using 'FileInfo' classes instead of just strings.
                  foreach (string fileName in fileEntries)
                  {
                        if (fileName.IndexOf(".log") != -1)
                              filesToRead.Add(fileName);
                  }

                  if (filesToRead.Count == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files I can read (log files), there is nothing I can do!!");
                        return;
                  }
                  #endregion Initial Checks


                  #region Read in the Settings file and create a collection of paths.
                  //Maybe make this typesafe by making extending collectionbase to make a PathCollection.
                  //Or in C# 2.0 return a List<Path> collection...
                  ArrayList myPaths = ExtractPathsFromSettingsFile(settingsFile);
                  if (myPaths == null || myPaths.Count == 0)
                  {
                        Console.WriteLine("I have no 'Paths' to run from, perhaps the settings file was malformed?");
                        return;
                  }
                  #endregion Read in the Settings file and create a collection of paths.

                  #region Create New Threads & Do Reading
                  //Create a new thread reader for each of the files. NB, we aren't going to be starting them until later!
                  FileOpenerThread[] openFiles = new FileOpenerThread[filesToRead.Count];
                  for (int i = 0; i < filesToRead.Count; i++)
                        if (filesToRead[i] != null && filesToRead[i].Length != 0)
                              openFiles[i] = new FileOpenerThread(filesToRead[i], myPaths);

                  //This is where we set any negative value for the number of threads to be the number of files to read.
                  //i.e. if the user passes in 0, or any negative value, they will get a new thread for each file.
                  if (numThreads <= 0 || numThreads > filesToRead.Count)
                        numThreads = filesToRead.Count;

                  int started = 0, runningThreads = 0, end = started + numThreads;

                  //Here we want to start numThreads, so we go through the threads we have, starting them until we reach the
                  //numThreads limit, OR we go past the number of files in the directory!
                  for (int i = started; i < end; i++)
                  {
                        if (i >= openFiles.Length) //Are we attempting to open more threads than files?
                              break;
                        openFiles[i].Start();
                        started++;
                        runningThreads++;
                  }

                  //A local variable to make the rest of this program wait until all the files are written.
                  bool finished = false;
                  while (!finished)
                  {
                        finished = true;
                        //Sleep to prevent 100% CPU usage.
                        Thread.Sleep(300);
                        int nonFinishedThreads = 0;
                        //Check if the thread has finished...
                        foreach (FileOpenerThread thread in openFiles)
                        {
                              if (!thread.Finished)
                              {
                                    nonFinishedThreads++;
                                    finished = false;
                                    if (nonFinishedThreads == numThreads)
                                          break;
                              }
                              else if (thread.Finished && !thread.Checked)
                              {
                                    //If the variable 'start' (which is defined up about 30 lines) is >= openFiles.Length
                                    //then we do nothing, else we need to start another thread, as we know the previous one
                                    //has finished.
                                    runningThreads--;
                                    thread.BeenChecked();
                                    thread.Stop(); //Just incase...
                                    if (!(started >= openFiles.Length))
                                    {
                                          runningThreads++;
                                          openFiles[started].Start();
                                          started++;
                                    }
                              }
                        }
                  }
                  #endregion Create New Threads & Do Reading

                  #region Finalise
                  /* All the threads have finished, so now we can commit all the 'Paths'.
                   * We do it this way so that the creation of a StreamWriter occurs only once for each
                   * output file.
                   * */
                  foreach (Path path in myPaths)
                        path.Commit();
                  TimeSpan timeTaken = DateTime.Now - startTime;
                  Console.Out.WriteLine("NumThreads: " + numThreads + ", Time taken (ms): " + timeTaken.TotalMilliseconds);
                  Console.Out.WriteLine("=====================================================================");
                  Console.Out.WriteLine("Finished reading files");
                        ////, press [ENTER] to exit.");
                  #endregion Finalise
            }

            //NEW!!
            /// <summary>Extract the 'Paths' from the settings file given.</summary>
            /// <param name="settingsFilename">The filename of the settings file.</param>
            /// <returns>A collection of paths, or null if the file doesn't exist.</returns>
            private ArrayList ExtractPathsFromSettingsFile(string settingsFilename)
            {
                  ArrayList myPaths = new ArrayList();

                  if (!File.Exists(settingsFilename))
                        return null;

                  string filename = "";
                  string search1 = "";
                  string search2 = "";
                  long fileSize = 0;

                  XmlTextReader sReader = new XmlTextReader(settingsFilename);
                  while (sReader.Read())
                  {
                        switch (sReader.NodeType)
                        {
                              case XmlNodeType.Element:
                                    if (sReader.Name.Equals("path"))
                                    {
                                          if (sReader.AttributeCount != 4)
                                                break;

                                          sReader.MoveToAttribute("output");
                                          filename = sReader.Value;

                                          sReader.MoveToAttribute("search1");
                                          search1 = sReader.Value;

                                          sReader.MoveToAttribute("search2");
                                          search2 = sReader.Value;

                                          sReader.MoveToAttribute("filesize");
                                          fileSize = Convert.ToInt64(sReader.Value);
                                          myPaths.Add(new Path(filename, search1, search2, fileSize));
                                    }
                                    //reset all vals:
                                    filename = search1 = search2 = "";
                                    fileSize = 0;
                                    break;
                        }
                  }
                  return myPaths;
            }

            //NEW!!
            /// <summary>Just display a the usage message.</summary>
            private void DisplayUsageMessage()
            {
                  Console.Out.WriteLine(
                        "Usage:\r\n------------------------------------\r\n" +
                        "My.Exe -d <dir> -s <settings> [-t ##]\r\n" +
                        "\t -d <dir> = the directory you want to read the '.log' files from." +
                        "\t -s <settings> = The settings the file to read in." +
                        "\t -t ## = optional, the number of threads to read the log files on.");
            }
      }

      /////////////////////////////////////////////////////
      // Path Class: Writes the file and contains search info.
      /////////////////////////////////////////////////////
      /// <summary>A class to store the search strings, filenames etc for use in 'Arrays' method of the 'FileOpenerThread' class.</summary>
      internal class Path
      {
            #region Member Variables & Properties
            /// <summary>The size that the file is allowed to get (in bytes) before a new file is opened.</summary>
            private long _FileSize;

            /// <summary>The output FileInfo instance that is written to.</summary>
            private FileInfo _OutputFileInfo;

            /// <summary>All the lines read in previously.</summary>
            private StringCollection _Lines = new StringCollection();

            /// <summary>The first string that must be found in the line to be checked.</summary>
            private string _SearchString1;

            /// <summary>The second string that must be found in the line to be checked.</summary>
            private string _SearchString2;
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new Path instance.</summary>
            /// <param name="filename">The filename to output results to.</param>
            /// <param name="search1">The first search string.</param>
            /// <param name="search2">The second search string.</param>
            /// <param name="fileSize">The size in bytes the file can be before it needs to be closed and a new file opened.</param>
            public Path(string filename, string search1, string search2, long fileSize)
            {
                  _OutputFileInfo = new FileInfo(filename);
                  _SearchString1 = search1;
                  _SearchString2 = search2;
                  if (fileSize > 0)
                        _FileSize = fileSize;
                  else
                        _FileSize = long.MaxValue;
            }
            #endregion Constructor

            #region Checking a Line
            /// <summary>Check if a given line contains both of the SearchStrings.</summary>
            /// <param name="line">The line to check.</param>
            public void CheckLine(string line)
            {
                  if (line == null || _SearchString1 == null || _SearchString2 == null)
                        return;
                  if (line.IndexOf(_SearchString1) != -1 && line.IndexOf(_SearchString2) != -1)
                        AppendText(line);
            }

            /// <summary>Append the line to the private Collection of lines.</summary>
            /// <param name="line">The line to append.</param>
            private void AppendText(string line)
            {
                  lock (_Lines)
                  {
                        if (_Lines == null)
                              _Lines = new StringCollection();
                        _Lines.Add(line);
                  }
            }

            /// <summary>Commit *all* the lines read in to the files.</summary>
            public void Commit()
            {
                  if (_Lines == null || _Lines.Count <= 0 || _OutputFileInfo == null)
                        return;

                  _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));
                  Console.Out.WriteLine("Writing " + _Lines.Count + " lines to: " + _OutputFileInfo.FullName);
                  foreach (string line in _Lines)
                        this.WriteLine(line);
            }

            //NEW!!
            /// <summary>Generate a new filename (if required).</summary>
            /// <param name="filenameToCheck">The filename to check.</param>
            /// <returns>The new filename if needed.</returns>
            private string GenerateFileName(string filenameToCheck)
            {
                  //File doesn't exist, so can't be bigger than limit.
                  if (!File.Exists(filenameToCheck))
                        return filenameToCheck;

                  //File does exist, check size:
                  FileInfo fi = new FileInfo(filenameToCheck);

                  //Size ok, so return the name.
                  if (!(fi.Length > _FileSize))
                        return filenameToCheck;

                  //Size isn't ok, so we need a new filename:
                  //Check if this is a file with an _# on it...
                  string name = "";
                  int fileNumber = 0;

                  int indexOfUnderscore = fi.FullName.LastIndexOf("_");
                  if (indexOfUnderscore == -1)
                        name = fi.FullName.Substring(0, fi.FullName.Length - fi.Extension.Length);
                  else
                  {
                        name = fi.FullName.Substring(0, indexOfUnderscore);
                        fileNumber = Convert.ToInt32(fi.FullName.Substring(indexOfUnderscore + 1, fi.FullName.Length - (indexOfUnderscore + 1) - fi.Extension.Length));
                  }

                  fileNumber++;
                  return GenerateFileName(name + "_" + fileNumber + fi.Extension);
            }

            //NEW!!
            /// <summary>Write a line to the file.
            /// This checks the file size and compares it to the file length (which is in bytes).</summary>
            /// <remarks>If the file is bigger than allowed the _FileWriter will be closed, and a new one will be created
            /// of the new 'advanced' name (givenName_#.extension).</remarks>
            /// <param name="line">The line to write.</param>
            private void WriteLine(string line)
            {
                  _OutputFileInfo.Refresh();
                  if (_OutputFileInfo.Exists)
                  {
                        if (_OutputFileInfo.Length > _FileSize)
                        {
                              Console.WriteLine("File changing to: " + _OutputFileInfo.FullName + " as size (" + _OutputFileInfo.Length + ") was > " + _FileSize + "bytes");
                              _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));                    
                        }
                  }
                  StreamWriter _FileWriter = _OutputFileInfo.AppendText();
                  _FileWriter.WriteLine(line);
                  _FileWriter.Close();
            }
            #endregion Checking a Line
      }

      /////////////////////////////////////////////////////
      // FileOpenerThread Class: Opens a file on a new thread.
      /////////////////////////////////////////////////////
      /// <summary>Will open a file on a new thread and retrieve the entire contents.</summary>
      internal class FileOpenerThread
      {
            #region Member Variables & Properties
            /// <summary>The filename to read, set in the constructor.</summary>
            private string mFilename;

            /// <summary>Has the thread finished reading the file?</summary>
            private bool mFinished = false;

            /// <summary>Has this thread been checked by the calling method?</summary>
            private bool mChecked = false;

            /// <summary>The current contents of the file.</summary>
            private string mContentsOfFile;

            /// <summary>The thread that will do the reading.</summary>
            private Thread mReaderThread;

            /// <summary>Used for disposing and stopping the threads.</summary>
            private bool mAlive = true;

            /// <summary>The collection of "Path's" that contain the search information etc.</summary>
            private ArrayList mPaths;

            /// <summary>Gets the contents of the file.</summary>
            /// <remarks>Will return an error message if an attempt is made to access the contents before the file is read.</remarks>
            public string ContentsOfFile
            {
                  get
                  {
                        if (Finished)
                              return mContentsOfFile;
                        return "File reading (" + Filename + ") either hasn't begun, or hasn't finished.";
                  }
            }

            /// <summary>Gets the filename the thread is running on.</summary>
            public string Filename
            {
                  get { return mFilename; }
            }

            /// <summary>Gets whether the file has been read or not.</summary>
            public bool Finished
            {
                  get { return mFinished; }
            }

            /// <summary>Gets whether this thread has been checked by the calling method.</summary>
            public bool Checked
            {
                  get { return mChecked; }
            }
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new instance of the FileOpenerThread passing in the file to open.</summary>
            /// <param name="filename">The file to open.</param>
            /// <param name="paths">The paths to write to.</param>
            public FileOpenerThread(string filename, ArrayList paths)
            {
                  mFilename = filename;
                  mReaderThread = new Thread(new ThreadStart(ReadFile));
                  mPaths = paths;
            }
            #endregion Constructor

            #region Start / Stop / Checked
            /// <summary>Start the thread to begin reading.</summary>
            public void Start()
            {
                  mFinished = false;
                  mReaderThread.Start();
            }

            /// <summary>Stop the thread, the thread will read one more line and then die.</summary>
            public void Stop()
            {
                  mAlive = false;
            }

            /// <summary>The file thread has been checked.</summary>
            public void BeenChecked()
            {
                  mChecked = true;
            }
            #endregion Start / Stop / Checked

            #region Parse The File
            /// <summary>Read the file.</summary>
            private void ReadFile()
            {
                  if (Filename.Length == 0 || !File.Exists(Filename))
                  {
                        mContentsOfFile = "Filename: " + Filename + " doesn't exist.";
                        mFinished = true;
                        return;
                  }
                  StreamReader myFile = new StreamReader(Filename);
                  string line = myFile.ReadLine();
                  while (line != null)
                  {
                        if (!mAlive)
                              break;
                        Arrays(line);
                        line = myFile.ReadLine();
                  }
                  myFile.Close();
                  mFinished = true;
            }

            /// <summary>Check each line read in against the paths given.</summary>
            /// <param name="lines">The line to check.</param>
            public void Arrays(string lines)
            {
                  if (lines == null || mPaths == null)
                  {
                        Console.Out.WriteLine("ERROR: Null line/Null Paths read in, can't deal with this, as such am exiting 'Arrays'.");
                        return;
                  }
                  foreach (Path path in mPaths)
                        path.CheckLine(lines);
            }
            #endregion Parse The File
      }
}
0
Mark1616
Asked:
Mark1616
  • 13
  • 11
  • 2
1 Solution
 
htang_usCommented:
Code snippet from your program:

                  #region Create New Threads & Do Reading
                  //Create a new thread reader for each of the files. NB, we aren't going to be starting them until later!
                  FileOpenerThread[] openFiles = new FileOpenerThread[filesToRead.Count];
                  for (int i = 0; i < filesToRead.Count; i++)
                        if (filesToRead[i] != null && filesToRead[i].Length != 0)
                              openFiles[i] = new FileOpenerThread(filesToRead[i], myPaths);

Here you creates as many threads as the number of log files. If you have thousands of log files, you will end up with thousands of threads. That will almost definitely kill most systems.

However, you can handle this easily with a thread pool.

Create a thread pool with, say, 20 threads. So at any moment, you handle only 20 log files. This will greatly reduce the system resources your program will use.

hope this will help.
0
 
existenz2Commented:
I agree with htang_us, using a threadpool will definatly solve your problem:

More information at ThreadPools: http://www.codeproject.com/csharp/threadtests.asp
0
 
Mark1616Author Commented:
Hi htangs,

How I can do that would you please add what you say the existing Code?

Thanks.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
Mark1616Author Commented:
Hi htangs,

I I say   MyProg.exe -d <dirToRead> -s <settingsFile> [-t ##]
and pass the -t for example 3 stiil it kil most systems.

Thanks
Mark
0
 
htang_usCommented:
From what I saw, the -t parameter is not used to control the number of threads.

You create the same number of threads as the number of files. This will definitely crash your system when the number of files is big.

this is hte code snippet from your program:
                  #region Create New Threads & Do Reading
                  //Create a new thread reader for each of the files. NB, we aren't going to be starting them until later!
                  FileOpenerThread[] openFiles = new FileOpenerThread[filesToRead.Count];
                  for (int i = 0; i < filesToRead.Count; i++)
                        if (filesToRead[i] != null && filesToRead[i].Length != 0)
                              openFiles[i] = new FileOpenerThread(filesToRead[i], myPaths);

                  //This is where we set any negative value for the number of threads to be the number of files to read.
                  //i.e. if the user passes in 0, or any negative value, they will get a new thread for each file.
                  if (numThreads <= 0 || numThreads > filesToRead.Count)
                        numThreads = filesToRead.Count;

This is the part you can use thread pool.

You can use -t to pass in the number of threads to be run in a thread pool. For example, -t 20

To use thread pool, you need to do this:
1. put each file name in a queue structure, you need to synchronize this structure for the pool to work.
2. Create predefined number of threads, let say 20
3. Inside the thread, try to read a file name from queue, then work on this file until it's done.
4. when thread finishes one file, it will go back to read queue for another file if queue is not empty.
5. This process will be over when queue is empty (let thread exit when queue is emtpy).
0
 
Mark1616Author Commented:
Hi htang,

Would you please add the  neccesry code to the exising code to work?

Thanks
Mark
0
 
htang_usCommented:
i finished the thread pool part and file handling. But there is an unfinished part. (Check "#region unfinished" in class MultiThreadFileReader )

The code passes compilation.

=====================================================

using System;
using System.Collections;
using System.Collections.Specialized;
using System.IO;
using System.Threading;
using System.Xml;

namespace MultiThreadFileReader
{
      /////////////////////////////////////////////////////
      // Program Class: Launches the program.
      /////////////////////////////////////////////////////
      ///
      //*
      internal class Program
      {
            private static void Main(string[] args)
            {
                  try
                  {
                        new MultiThreadFileReader(args);
                  }
                  catch (System.Exception e)
                  {
                        Console.WriteLine("An exception was caught whilst running the MultiThreadFileReader:"
                              + Environment.NewLine + e.ToString());
                  }
                  ////---Console.WriteLine("Press ENTER to exit.");
                  ///----Console.ReadLine();
            }
      }
      //*/


      /////////////////////////////////////////////////////
      // MultiThreadFileReader Class: Runs the reading etc.
      /////////////////////////////////////////////////////
      internal class MultiThreadFileReader
      {
            /// <summary>Run the MultiThreadFileReader.</summary>
            /// <param name="args">The directory name to read files from.</param>
            internal MultiThreadFileReader(string[] args)
            {
                  DateTime startTime = DateTime.Now;
                  string dirToRead = null;
                  string settingsFile = null;
                  int numThreads = -1;

                  #region Initial Checks
                  //Usage:
                  //  MyProg.exe -d <dirToRead> -s <settingsFile> [-t ##]
                  //We also need to check that the number of args is correct, we *need* a settings file and *need* a dir to read.
                  //-t / -threads is the only arg we don't need (as we can default it).

                  //NEW!!

                  if (args.Length < 4)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  for (int i = 0; i < args.Length; i++)
                  {
                        switch (args[i])
                        {
                                    //The user can use either -t or -threads to set the number of threads to use.
                              case "-t":
                              case "-threads":
                                    i++;
                                    numThreads = Convert.ToInt32(args[i]);
                                    break;
                                    //The user can use -d or -dir to indicate the directory to read in from.
                              case "-d":
                              case "-dir":
                                    i++;
                                    dirToRead = args[i];
                                    break;
                                    //The user can use -s or -settings to indicate the location of the settings file to read in.
                              case "-s":
                              case "-settings":
                                    i++;
                                    settingsFile = args[i];
                                    break;
                        }
                  }

                  //Have the settings file and directory to read from been set?
                  if (settingsFile == null || settingsFile.Length == 0 || dirToRead == null || dirToRead.Length == 0)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  //Does the settings file exist?
                  if (!File.Exists(settingsFile))
                  {
                        Console.WriteLine("The settings file given: " + settingsFile + ", doesn't exist, I can't continue!");
                        return;
                  }

                  //Does the directory to read exist?
                  if (!Directory.Exists(dirToRead))
                  {
                        Console.WriteLine("This Path (" + dirToRead + ") does not exists.");
                        return;
                  }

                  //Get the files.
                  string[] fileEntries = Directory.GetFiles(dirToRead);
                  if (fileEntries == null || fileEntries.Length == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files, there is nothing for me to read!!");
                        return;
                  }

                  //Create a new StringCollection (an ArrayList, but specialized to handle only strings),
                  //this means we can avoid casting and use it type-safely.
                  filesToRead = new StringCollection();

                  //If the filename has '.log' anywhere in it, it will be read.
                  //i.e. my.LOG.txt, my.LOG, my.LOGarithm, etc. Not just the extension!!
                  //This can be changed by using 'FileInfo' classes instead of just strings.
                  foreach (string fileName in fileEntries)
                  {
                        if (fileName.IndexOf(".log") != -1)
                              filesToRead.Add(fileName);
                  }

                  if (filesToRead.Count == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files I can read (log files), there is nothing I can do!!");
                        return;
                  }
                  #endregion Initial Checks


                  #region Read in the Settings file and create a collection of paths.
                  //Maybe make this typesafe by making extending collectionbase to make a PathCollection.
                  //Or in C# 2.0 return a List<Path> collection...
                  ArrayList myPaths = ExtractPathsFromSettingsFile(settingsFile);
                  if (myPaths == null || myPaths.Count == 0)
                  {
                        Console.WriteLine("I have no 'Paths' to run from, perhaps the settings file was malformed?");
                        return;
                  }
                  #endregion Read in the Settings file and create a collection of paths.

                  #region Create New Threads & Do Reading
                  //here you specify the number of threads to be created
                  if (numThreads <= 0 || numThreads > 20 ) // you can change the maximal number of threads
                        numThreads = 20;

                  if ( numThreads > filesToRead.Count )
                        numThreads = filesToRead.Count;

                  //Create a new thread reader for each of the files. NB, we aren't going to be starting them until later!
                  FileOpenerThread[] openFiles = new FileOpenerThread[numThreads];
                  for (int i = 0; i < numThreads; i++)
                  {
                        openFiles[i] = new FileOpenerThread(this, myPaths);
                        openFiles[i].Start();
                  }
                  #endregion Create New Threads & Do Reading

                  #region unfinished
                  // here we should wait for all the threads to return.
                  // code is not done.
                  #endregion

                  #region Finalise
                  /* All the threads have finished, so now we can commit all the 'Paths'.
                        * We do it this way so that the creation of a StreamWriter occurs only once for each
                        * output file.
                        * */
                  foreach (Path path in myPaths)
                        path.Commit();
                  TimeSpan timeTaken = DateTime.Now - startTime;
                  Console.Out.WriteLine("NumThreads: " + numThreads + ", Time taken (ms): " + timeTaken.TotalMilliseconds);
                  Console.Out.WriteLine("=====================================================================");
                  Console.Out.WriteLine("Finished reading files");
                  ////, press [ENTER] to exit.");
                  #endregion Finalise
            }

            //NEW!!
            /// <summary>Extract the 'Paths' from the settings file given.</summary>
            /// <param name="settingsFilename">The filename of the settings file.</param>
            /// <returns>A collection of paths, or null if the file doesn't exist.</returns>
            private ArrayList ExtractPathsFromSettingsFile(string settingsFilename)
            {
                  ArrayList myPaths = new ArrayList();

                  if (!File.Exists(settingsFilename))
                        return null;

                  string filename = "";
                  string search1 = "";
                  string search2 = "";
                  long fileSize = 0;

                  XmlTextReader sReader = new XmlTextReader(settingsFilename);
                  while (sReader.Read())
                  {
                        switch (sReader.NodeType)
                        {
                              case XmlNodeType.Element:
                                    if (sReader.Name.Equals("path"))
                                    {
                                          if (sReader.AttributeCount != 4)
                                                break;

                                          sReader.MoveToAttribute("output");
                                          filename = sReader.Value;

                                          sReader.MoveToAttribute("search1");
                                          search1 = sReader.Value;

                                          sReader.MoveToAttribute("search2");
                                          search2 = sReader.Value;

                                          sReader.MoveToAttribute("filesize");
                                          fileSize = Convert.ToInt64(sReader.Value);
                                          myPaths.Add(new Path(filename, search1, search2, fileSize));
                                    }
                                    //reset all vals:
                                    filename = search1 = search2 = "";
                                    fileSize = 0;
                                    break;
                        }
                  }
                  return myPaths;
            }

            //NEW!!
            /// <summary>Just display a the usage message.</summary>
            private void DisplayUsageMessage()
            {
                  Console.Out.WriteLine(
                        "Usage:\r\n------------------------------------\r\n" +
                        "My.Exe -d <dir> -s <settings> [-t ##]\r\n" +
                        "\t -d <dir> = the directory you want to read the '.log' files from." +
                        "\t -s <settings> = The settings the file to read in." +
                        "\t -t ## = optional, the number of threads to read the log files on.");
            }

            public string GetNextLogFile()
            {
                  lock(this);

                  if ( filesToRead.Count == 0 )
                        return null;

                  string strRet = filesToRead[0];
                  filesToRead.RemoveAt ( 0 );
                  return strRet;
            }

            #region private fields
            StringCollection filesToRead = null;
            #endregion private fields
      }

      /////////////////////////////////////////////////////
      // Path Class: Writes the file and contains search info.
      /////////////////////////////////////////////////////
      /// <summary>A class to store the search strings, filenames etc for use in 'Arrays' method of the 'FileOpenerThread' class.</summary>
      internal class Path
      {
            #region Member Variables & Properties
            /// <summary>The size that the file is allowed to get (in bytes) before a new file is opened.</summary>
            private long _FileSize;

            /// <summary>The output FileInfo instance that is written to.</summary>
            private FileInfo _OutputFileInfo;

            /// <summary>All the lines read in previously.</summary>
            private StringCollection _Lines = new StringCollection();

            /// <summary>The first string that must be found in the line to be checked.</summary>
            private string _SearchString1;

            /// <summary>The second string that must be found in the line to be checked.</summary>
            private string _SearchString2;
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new Path instance.</summary>
            /// <param name="filename">The filename to output results to.</param>
            /// <param name="search1">The first search string.</param>
            /// <param name="search2">The second search string.</param>
            /// <param name="fileSize">The size in bytes the file can be before it needs to be closed and a new file opened.</param>
            public Path(string filename, string search1, string search2, long fileSize)
            {
                  _OutputFileInfo = new FileInfo(filename);
                  _SearchString1 = search1;
                  _SearchString2 = search2;
                  if (fileSize > 0)
                        _FileSize = fileSize;
                  else
                        _FileSize = long.MaxValue;
            }
            #endregion Constructor

            #region Checking a Line
            /// <summary>Check if a given line contains both of the SearchStrings.</summary>
            /// <param name="line">The line to check.</param>
            public void CheckLine(string line)
            {
                  if (line == null || _SearchString1 == null || _SearchString2 == null)
                        return;
                  if (line.IndexOf(_SearchString1) != -1 && line.IndexOf(_SearchString2) != -1)
                        AppendText(line);
            }

            /// <summary>Append the line to the private Collection of lines.</summary>
            /// <param name="line">The line to append.</param>
            private void AppendText(string line)
            {
                  lock (_Lines)
                  {
                        if (_Lines == null)
                              _Lines = new StringCollection();
                        _Lines.Add(line);
                  }
            }

            /// <summary>Commit *all* the lines read in to the files.</summary>
            public void Commit()
            {
                  if (_Lines == null || _Lines.Count <= 0 || _OutputFileInfo == null)
                        return;

                  _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));
                  Console.Out.WriteLine("Writing " + _Lines.Count + " lines to: " + _OutputFileInfo.FullName);
                  foreach (string line in _Lines)
                        this.WriteLine(line);
            }

            //NEW!!
            /// <summary>Generate a new filename (if required).</summary>
            /// <param name="filenameToCheck">The filename to check.</param>
            /// <returns>The new filename if needed.</returns>
            private string GenerateFileName(string filenameToCheck)
            {
                  //File doesn't exist, so can't be bigger than limit.
                  if (!File.Exists(filenameToCheck))
                        return filenameToCheck;

                  //File does exist, check size:
                  FileInfo fi = new FileInfo(filenameToCheck);

                  //Size ok, so return the name.
                  if (!(fi.Length > _FileSize))
                        return filenameToCheck;

                  //Size isn't ok, so we need a new filename:
                  //Check if this is a file with an _# on it...
                  string name = "";
                  int fileNumber = 0;

                  int indexOfUnderscore = fi.FullName.LastIndexOf("_");
                  if (indexOfUnderscore == -1)
                        name = fi.FullName.Substring(0, fi.FullName.Length - fi.Extension.Length);
                  else
                  {
                        name = fi.FullName.Substring(0, indexOfUnderscore);
                        fileNumber = Convert.ToInt32(fi.FullName.Substring(indexOfUnderscore + 1, fi.FullName.Length - (indexOfUnderscore + 1) - fi.Extension.Length));
                  }

                  fileNumber++;
                  return GenerateFileName(name + "_" + fileNumber + fi.Extension);
            }

            //NEW!!
            /// <summary>Write a line to the file.
            /// This checks the file size and compares it to the file length (which is in bytes).</summary>
            /// <remarks>If the file is bigger than allowed the _FileWriter will be closed, and a new one will be created
            /// of the new 'advanced' name (givenName_#.extension).</remarks>
            /// <param name="line">The line to write.</param>
            private void WriteLine(string line)
            {
                  _OutputFileInfo.Refresh();
                  if (_OutputFileInfo.Exists)
                  {
                        if (_OutputFileInfo.Length > _FileSize)
                        {
                              Console.WriteLine("File changing to: " + _OutputFileInfo.FullName + " as size (" + _OutputFileInfo.Length + ") was > " + _FileSize + "bytes");
                              _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));                    
                        }
                  }
                  StreamWriter _FileWriter = _OutputFileInfo.AppendText();
                  _FileWriter.WriteLine(line);
                  _FileWriter.Close();
            }
            #endregion Checking a Line
      }

      /////////////////////////////////////////////////////
      // FileOpenerThread Class: Opens a file on a new thread.
      /////////////////////////////////////////////////////
      /// <summary>Will open a file on a new thread and retrieve the entire contents.</summary>
      internal class FileOpenerThread
      {
            #region Member Variables & Properties
            /// <summary>Has this thread been checked by the calling method?</summary>
            private bool mChecked = false;

            /// <summary>The thread that will do the reading.</summary>
            private Thread mReaderThread;

            /// <summary>The collection of "Path's" that contain the search information etc.</summary>
            private ArrayList mPaths;

            private MultiThreadFileReader m_DataSource;

            /// <summary>Gets whether this thread has been checked by the calling method.</summary>
            public bool Checked
            {
                  get { return mChecked; }
            }
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new instance of the FileOpenerThread passing in the file to open.</summary>
            /// <param name="filename">The file to open.</param>
            /// <param name="paths">The paths to write to.</param>
            public FileOpenerThread(MultiThreadFileReader dataSource, ArrayList paths)
            {
                  m_DataSource = dataSource;
                  mReaderThread = new Thread(new ThreadStart(ReadFile));
                  mPaths = paths;
            }
            #endregion Constructor

            #region Start / Stop / Checked
            /// <summary>Start the thread to begin reading.</summary>
            public void Start()
            {
                  mReaderThread.Start();
            }

            #endregion Start / Stop / Checked

            #region Parse The File
            /// <summary>Read the file.</summary>
            private void ReadFile()
            {
                  while ( true )
                  {
                        string strFile = m_DataSource.GetNextLogFile();
                        if ( strFile == null )
                              break;

                        if ( strFile.Length == 0 || ! File.Exists ( strFile ) )
                              continue;

                        StreamReader myFile = new StreamReader(strFile);
                        string line = myFile.ReadLine();
                        while (line != null)
                        {
                              Arrays(line);
                              line = myFile.ReadLine();
                        }
                        myFile.Close();

                  }
            }

            /// <summary>Check each line read in against the paths given.</summary>
            /// <param name="lines">The line to check.</param>
            public void Arrays(string lines)
            {
                  if (lines == null || mPaths == null)
                  {
                        Console.Out.WriteLine("ERROR: Null line/Null Paths read in, can't deal with this, as such am exiting 'Arrays'.");
                        return;
                  }
                  foreach (Path path in mPaths)
                        path.CheckLine(lines);
            }
            #endregion Parse The File
      }
}
0
 
Mark1616Author Commented:
Hi htang,

Thank you so much.
Would you please add the necessary code for unfinished part?
I really appreciate it.

Thanks,
 Mark
0
 
Mark1616Author Commented:
Hi htang,

Would you please add the necessary code for unfinished part?
I really appreciate it.

Thanks,
 Mark
0
 
Mark1616Author Commented:
Hi htang,

Would you please add the necessary code for unfinished part?
I really need it done, so please add the code for  unfinished part.
I really appreciate it.

Thanks,
 Mark
0
 
existenz2Commented:
I suggest you give it a try yourself instead of repeatingly asking for him to complete the code, because of the following reasons:
- You are going to use the code, YOU should understand the code! Finishing it yourself will help understanding it much better and faster
- Do not expect to do htang_us or any contributer work for you on which you commercially depend on. We all freely answer questions in our (spare) time for several different reasons. So it can happen that somebody isn't able to finish the samples for any reason.
0
 
htang_usCommented:
Hi, Mark,

Sorry for not answering your questions.

I had been out of town.

I don't think I will have time this week. If I have time, I will work on this.

The unfinished part is about synchronization between parent thread and child threads. This shouldn't be very difficult for you to figure it out.

-htang
0
 
Mark1616Author Commented:
Hi htang,
Thank you som much.
I try but  I hope you have some free time, I reallty appricate you.
Mark
0
 
htang_usCommented:
I squeezed some time and wrote you a quick solution. It passed compilation, but I don't have time to test it.

==========================================

using System;
using System.Collections;
using System.Collections.Specialized;
using System.IO;
using System.Threading;
using System.Xml;

namespace MultiThreadFileReader
{
      /////////////////////////////////////////////////////
      // Program Class: Launches the program.
      /////////////////////////////////////////////////////
      ///
      //*
      internal class Program
      {
            private static void Main(string[] args)
            {
                  try
                  {
                        new MultiThreadFileReader(args);
                  }
                  catch (System.Exception e)
                  {
                        Console.WriteLine("An exception was caught whilst running the MultiThreadFileReader:"
                              + Environment.NewLine + e.ToString());
                  }
                  ////---Console.WriteLine("Press ENTER to exit.");
                  ///----Console.ReadLine();
            }
      }
      //*/


      /////////////////////////////////////////////////////
      // MultiThreadFileReader Class: Runs the reading etc.
      /////////////////////////////////////////////////////
      internal class MultiThreadFileReader
      {
            /// <summary>Run the MultiThreadFileReader.</summary>
            /// <param name="args">The directory name to read files from.</param>
            internal MultiThreadFileReader(string[] args)
            {
                  DateTime startTime = DateTime.Now;
                  string dirToRead = null;
                  string settingsFile = null;
                  int numThreads = -1;

                  #region Initial Checks
                  //Usage:
                  //  MyProg.exe -d <dirToRead> -s <settingsFile> [-t ##]
                  //We also need to check that the number of args is correct, we *need* a settings file and *need* a dir to read.
                  //-t / -threads is the only arg we don't need (as we can default it).

                  //NEW!!

                  if (args.Length < 4)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  for (int i = 0; i < args.Length; i++)
                  {
                        switch (args[i])
                        {
                                    //The user can use either -t or -threads to set the number of threads to use.
                              case "-t":
                              case "-threads":
                                    i++;
                                    numThreads = Convert.ToInt32(args[i]);
                                    break;
                                    //The user can use -d or -dir to indicate the directory to read in from.
                              case "-d":
                              case "-dir":
                                    i++;
                                    dirToRead = args[i];
                                    break;
                                    //The user can use -s or -settings to indicate the location of the settings file to read in.
                              case "-s":
                              case "-settings":
                                    i++;
                                    settingsFile = args[i];
                                    break;
                        }
                  }

                  //Have the settings file and directory to read from been set?
                  if (settingsFile == null || settingsFile.Length == 0 || dirToRead == null || dirToRead.Length == 0)
                  {
                        DisplayUsageMessage();
                        return;
                  }

                  //Does the settings file exist?
                  if (!File.Exists(settingsFile))
                  {
                        Console.WriteLine("The settings file given: " + settingsFile + ", doesn't exist, I can't continue!");
                        return;
                  }

                  //Does the directory to read exist?
                  if (!Directory.Exists(dirToRead))
                  {
                        Console.WriteLine("This Path (" + dirToRead + ") does not exists.");
                        return;
                  }

                  //Get the files.
                  string[] fileEntries = Directory.GetFiles(dirToRead);
                  if (fileEntries == null || fileEntries.Length == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files, there is nothing for me to read!!");
                        return;
                  }

                  //Create a new StringCollection (an ArrayList, but specialized to handle only strings),
                  //this means we can avoid casting and use it type-safely.
                  filesToRead = new StringCollection();

                  //If the filename has '.log' anywhere in it, it will be read.
                  //i.e. my.LOG.txt, my.LOG, my.LOGarithm, etc. Not just the extension!!
                  //This can be changed by using 'FileInfo' classes instead of just strings.
                  foreach (string fileName in fileEntries)
                  {
                        if (fileName.IndexOf(".log") != -1)
                              filesToRead.Add(fileName);
                  }

                  if (filesToRead.Count == 0)
                  {
                        Console.Out.WriteLine("The path '" + dirToRead + "' contains no files I can read (log files), there is nothing I can do!!");
                        return;
                  }
                  #endregion Initial Checks


                  #region Read in the Settings file and create a collection of paths.
                  //Maybe make this typesafe by making extending collectionbase to make a PathCollection.
                  //Or in C# 2.0 return a List<Path> collection...
                  ArrayList myPaths = ExtractPathsFromSettingsFile(settingsFile);
                  if (myPaths == null || myPaths.Count == 0)
                  {
                        Console.WriteLine("I have no 'Paths' to run from, perhaps the settings file was malformed?");
                        return;
                  }
                  #endregion Read in the Settings file and create a collection of paths.

                  #region Create New Threads & Do Reading
                  //here you specify the number of threads to be created
                  if (numThreads <= 0 || numThreads > 20 ) // you can change the maximal number of threads
                        numThreads = 20;

                  if ( numThreads > filesToRead.Count )
                        numThreads = filesToRead.Count;

                  System.Threading.ManualResetEvent[] mres = new ManualResetEvent[numThreads];

                  //Create a new thread reader for each of the files. NB, we aren't going to be starting them until later!
                  FileOpenerThread[] openFiles = new FileOpenerThread[numThreads];
                  for (int i = 0; i < numThreads; i++)
                  {
                        mres[i] = new ManualResetEvent ( false );

                        openFiles[i] = new FileOpenerThread(this, myPaths, mres[i]);
                        openFiles[i].Start();
                  }
                  #endregion Create New Threads & Do Reading

                  #region Wait for all child threads to finish
                  
                  WaitHandle.WaitAll ( mres );

                  #endregion

                  #region Finalise
                  /* All the threads have finished, so now we can commit all the 'Paths'.
                        * We do it this way so that the creation of a StreamWriter occurs only once for each
                        * output file.
                        * */
                  foreach (Path path in myPaths)
                        path.Commit();
                  TimeSpan timeTaken = DateTime.Now - startTime;
                  Console.Out.WriteLine("NumThreads: " + numThreads + ", Time taken (ms): " + timeTaken.TotalMilliseconds);
                  Console.Out.WriteLine("=====================================================================");
                  Console.Out.WriteLine("Finished reading files");
                  ////, press [ENTER] to exit.");
                  #endregion Finalise
            }

            //NEW!!
            /// <summary>Extract the 'Paths' from the settings file given.</summary>
            /// <param name="settingsFilename">The filename of the settings file.</param>
            /// <returns>A collection of paths, or null if the file doesn't exist.</returns>
            private ArrayList ExtractPathsFromSettingsFile(string settingsFilename)
            {
                  ArrayList myPaths = new ArrayList();

                  if (!File.Exists(settingsFilename))
                        return null;

                  string filename = "";
                  string search1 = "";
                  string search2 = "";
                  long fileSize = 0;

                  XmlTextReader sReader = new XmlTextReader(settingsFilename);
                  while (sReader.Read())
                  {
                        switch (sReader.NodeType)
                        {
                              case XmlNodeType.Element:
                                    if (sReader.Name.Equals("path"))
                                    {
                                          if (sReader.AttributeCount != 4)
                                                break;

                                          sReader.MoveToAttribute("output");
                                          filename = sReader.Value;

                                          sReader.MoveToAttribute("search1");
                                          search1 = sReader.Value;

                                          sReader.MoveToAttribute("search2");
                                          search2 = sReader.Value;

                                          sReader.MoveToAttribute("filesize");
                                          fileSize = Convert.ToInt64(sReader.Value);
                                          myPaths.Add(new Path(filename, search1, search2, fileSize));
                                    }
                                    //reset all vals:
                                    filename = search1 = search2 = "";
                                    fileSize = 0;
                                    break;
                        }
                  }
                  return myPaths;
            }

            //NEW!!
            /// <summary>Just display a the usage message.</summary>
            private void DisplayUsageMessage()
            {
                  Console.Out.WriteLine(
                        "Usage:\r\n------------------------------------\r\n" +
                        "My.Exe -d <dir> -s <settings> [-t ##]\r\n" +
                        "\t -d <dir> = the directory you want to read the '.log' files from." +
                        "\t -s <settings> = The settings the file to read in." +
                        "\t -t ## = optional, the number of threads to read the log files on.");
            }

            public string GetNextLogFile()
            {
                  lock(this);

                  if ( filesToRead.Count == 0 )
                        return null;

                  string strRet = filesToRead[0];
                  filesToRead.RemoveAt ( 0 );
                  return strRet;
            }

            #region private fields
            StringCollection filesToRead = null;
            #endregion private fields
      }

      /////////////////////////////////////////////////////
      // Path Class: Writes the file and contains search info.
      /////////////////////////////////////////////////////
      /// <summary>A class to store the search strings, filenames etc for use in 'Arrays' method of the 'FileOpenerThread' class.</summary>
      internal class Path
      {
            #region Member Variables & Properties
            /// <summary>The size that the file is allowed to get (in bytes) before a new file is opened.</summary>
            private long _FileSize;

            /// <summary>The output FileInfo instance that is written to.</summary>
            private FileInfo _OutputFileInfo;

            /// <summary>All the lines read in previously.</summary>
            private StringCollection _Lines = new StringCollection();

            /// <summary>The first string that must be found in the line to be checked.</summary>
            private string _SearchString1;

            /// <summary>The second string that must be found in the line to be checked.</summary>
            private string _SearchString2;
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new Path instance.</summary>
            /// <param name="filename">The filename to output results to.</param>
            /// <param name="search1">The first search string.</param>
            /// <param name="search2">The second search string.</param>
            /// <param name="fileSize">The size in bytes the file can be before it needs to be closed and a new file opened.</param>
            public Path(string filename, string search1, string search2, long fileSize)
            {
                  _OutputFileInfo = new FileInfo(filename);
                  _SearchString1 = search1;
                  _SearchString2 = search2;
                  if (fileSize > 0)
                        _FileSize = fileSize;
                  else
                        _FileSize = long.MaxValue;
            }
            #endregion Constructor

            #region Checking a Line
            /// <summary>Check if a given line contains both of the SearchStrings.</summary>
            /// <param name="line">The line to check.</param>
            public void CheckLine(string line)
            {
                  if (line == null || _SearchString1 == null || _SearchString2 == null)
                        return;
                  if (line.IndexOf(_SearchString1) != -1 && line.IndexOf(_SearchString2) != -1)
                        AppendText(line);
            }

            /// <summary>Append the line to the private Collection of lines.</summary>
            /// <param name="line">The line to append.</param>
            private void AppendText(string line)
            {
                  lock (_Lines)
                  {
                        if (_Lines == null)
                              _Lines = new StringCollection();
                        _Lines.Add(line);
                  }
            }

            /// <summary>Commit *all* the lines read in to the files.</summary>
            public void Commit()
            {
                  if (_Lines == null || _Lines.Count <= 0 || _OutputFileInfo == null)
                        return;

                  _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));
                  Console.Out.WriteLine("Writing " + _Lines.Count + " lines to: " + _OutputFileInfo.FullName);
                  foreach (string line in _Lines)
                        this.WriteLine(line);
            }

            //NEW!!
            /// <summary>Generate a new filename (if required).</summary>
            /// <param name="filenameToCheck">The filename to check.</param>
            /// <returns>The new filename if needed.</returns>
            private string GenerateFileName(string filenameToCheck)
            {
                  //File doesn't exist, so can't be bigger than limit.
                  if (!File.Exists(filenameToCheck))
                        return filenameToCheck;

                  //File does exist, check size:
                  FileInfo fi = new FileInfo(filenameToCheck);

                  //Size ok, so return the name.
                  if (!(fi.Length > _FileSize))
                        return filenameToCheck;

                  //Size isn't ok, so we need a new filename:
                  //Check if this is a file with an _# on it...
                  string name = "";
                  int fileNumber = 0;

                  int indexOfUnderscore = fi.FullName.LastIndexOf("_");
                  if (indexOfUnderscore == -1)
                        name = fi.FullName.Substring(0, fi.FullName.Length - fi.Extension.Length);
                  else
                  {
                        name = fi.FullName.Substring(0, indexOfUnderscore);
                        fileNumber = Convert.ToInt32(fi.FullName.Substring(indexOfUnderscore + 1, fi.FullName.Length - (indexOfUnderscore + 1) - fi.Extension.Length));
                  }

                  fileNumber++;
                  return GenerateFileName(name + "_" + fileNumber + fi.Extension);
            }

            //NEW!!
            /// <summary>Write a line to the file.
            /// This checks the file size and compares it to the file length (which is in bytes).</summary>
            /// <remarks>If the file is bigger than allowed the _FileWriter will be closed, and a new one will be created
            /// of the new 'advanced' name (givenName_#.extension).</remarks>
            /// <param name="line">The line to write.</param>
            private void WriteLine(string line)
            {
                  _OutputFileInfo.Refresh();
                  if (_OutputFileInfo.Exists)
                  {
                        if (_OutputFileInfo.Length > _FileSize)
                        {
                              Console.WriteLine("File changing to: " + _OutputFileInfo.FullName + " as size (" + _OutputFileInfo.Length + ") was > " + _FileSize + "bytes");
                              _OutputFileInfo = new FileInfo(GenerateFileName(_OutputFileInfo.FullName));                    
                        }
                  }
                  StreamWriter _FileWriter = _OutputFileInfo.AppendText();
                  _FileWriter.WriteLine(line);
                  _FileWriter.Close();
            }
            #endregion Checking a Line
      }

      /////////////////////////////////////////////////////
      // FileOpenerThread Class: Opens a file on a new thread.
      /////////////////////////////////////////////////////
      /// <summary>Will open a file on a new thread and retrieve the entire contents.</summary>
      internal class FileOpenerThread
      {
            #region Member Variables & Properties
            /// <summary>Has this thread been checked by the calling method?</summary>
            private bool mChecked = false;

            /// <summary>The thread that will do the reading.</summary>
            private Thread mReaderThread;

            /// <summary>The collection of "Path's" that contain the search information etc.</summary>
            private ArrayList mPaths;

            private MultiThreadFileReader m_DataSource;

            private ManualResetEvent m_ManualResetEvent;

            /// <summary>Gets whether this thread has been checked by the calling method.</summary>
            public bool Checked
            {
                  get { return mChecked; }
            }
            #endregion Member Variables & Properties

            #region Constructor
            /// <summary>Create a new instance of the FileOpenerThread passing in the file to open.</summary>
            /// <param name="filename">The file to open.</param>
            /// <param name="paths">The paths to write to.</param>
            public FileOpenerThread(MultiThreadFileReader dataSource, ArrayList paths, ManualResetEvent mre)
            {
                  m_DataSource = dataSource;
                  mReaderThread = new Thread(new ThreadStart(ReadFile));
                  mPaths = paths;
                  m_ManualResetEvent = mre;
            }
            #endregion Constructor

            #region Start / Stop / Checked
            /// <summary>Start the thread to begin reading.</summary>
            public void Start()
            {
                  mReaderThread.Start();
            }

            #endregion Start / Stop / Checked

            #region Parse The File
            /// <summary>Read the file.</summary>
            private void ReadFile()
            {
                  while ( true )
                  {
                        string strFile = m_DataSource.GetNextLogFile();
                        if ( strFile == null )
                              break;

                        if ( strFile.Length == 0 || ! File.Exists ( strFile ) )
                              continue;

                        StreamReader myFile = new StreamReader(strFile);
                        string line = myFile.ReadLine();
                        while (line != null)
                        {
                              Arrays(line);
                              line = myFile.ReadLine();
                        }
                        myFile.Close();
                  }
                  m_ManualResetEvent.Set();
            }

            /// <summary>Check each line read in against the paths given.</summary>
            /// <param name="lines">The line to check.</param>
            public void Arrays(string lines)
            {
                  if (lines == null || mPaths == null)
                  {
                        Console.Out.WriteLine("ERROR: Null line/Null Paths read in, can't deal with this, as such am exiting 'Arrays'.");
                        return;
                  }
                  foreach (Path path in mPaths)
                        path.CheckLine(lines);
            }
            #endregion Parse The File
      }
}
0
 
Mark1616Author Commented:
Hi htang,
Thank you som much for your time.
I tested it but it causes the syetem to run so slow almost I can  not do any thhing.
Mark,
0
 
htang_usCommented:
1. Open "Task Manager" and go to "Performance"
2. Test with 1 thread, 10 files
3. Test with 5 threads, 10 files,
4. Test with 5 threads, 20 files,
5. Test with 5 threads, 50 files,
6. Test with 20 threads, 50 files,

When you do tests, check the CPU usage and memory usage.
0
 
Mark1616Author Commented:
Hi htang,

I run it when the number of files are over 140 the system will be so slow that I can not do anything.

Thank you so much.

Mark


0
 
htang_usCommented:
This sounds like your old problem.

1. What's your hardware configuration and OS? CPU? Memory?
2. Is there any services running on the same machine?
3. Post your setting file (you need to modify it if it's too big)
4. Post a sample log file (only contains the first 100 lines)

0
 
Mark1616Author Commented:
Hi htang,

Sorry I run the old version.
But when I run your version. the is no out put created:

Thank you so much for all your help.

Mark
Here is the result:
C:\AuditLogNew\AuditLogs>AuditlogTransact3.bat

C:\AuditLogNew\AuditLogs>AuditLogs -d "C:\Transact3" -s "TConfig.xml" -t 1
NumThreads: 1, Time taken (ms): 15.6256
=====================================================================
Finished reading files

The Tconfig file is like :

<?xml version="1.0" encoding="utf-8" ?>
<multiThreadFileReaderSettings>
  <paths>
    <path output="TransactData.log" search1="someting" search2="somethingelse" filesize="10485760"/>
  </paths>
</multiThreadFileReaderSettings>
0
 
htang_usCommented:
Hi, Mark,

I run the program with one thread and one log file.
Everything works fine. The output file contains one line. That's correct.

your serach1 is "someting", not "something", you might miss one letter "h",
this might be the cause you got nothing.

0
 
Mark1616Author Commented:
Hi htang,
When the number of files are <150 it works even its slow ,
the size of each file is almost 2 M.
Here is the performance
PF Usage 1.44
Physical Memory 1048044
Avialable 2744

Thanks,
Mark
0
 
htang_usCommented:
Hi, Mark,

I created 201 files, each of which has 141858 lines and the size is 2,411,633 bytes. The number of threads is 20.

In each file except one, there are two lines that match the standard. So the expected output should be 400 lines.

My machine has 512Mb memory. When I ran the program, there are about 10 windows open, including FireFox, VS.NET, Emacs, Word, Excel, Acrobat Reader, and, of course, Task Manager.

The PF Usage is about 600Mb, CPU fluctuated  between 40% to 70%. The available memory fluctuated between about 1M and 200M.

When Program is running, I can do browser, a little slower than normal, but very acceptable.
0
 
htang_usCommented:
Hi, Mark,

If your question gets resolved, coupld you please close the question?
0
 
Mark1616Author Commented:
Hi htang,
Still I have the same problem,
Would you mind copy the code that you run and works fine for you, may be I missied something in the code.

Thank yo so much.
Mark
0
 
htang_usCommented:
Hi, Mark,

I use the code I posted 5/15. So you should have the code I am using.

You can try a debug run with one thread to see if everything works correctly.
0
 
Mark1616Author Commented:
Hi htang,
Thank you so much for all your help.

Mark
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 13
  • 11
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now