• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 395
  • Last Modified:

Monitor multiple directories and execute process if files are added.

I have been primarily an applications developer but am now migrating into more systems level stuff, so most of this is very new to me. Given the complexity of this question, I wish I could assign it a point value of about 2000 points, but I have confidence the experts will come through, nonetheless.

I need to design a program that will monitor directories and perform actions. Basically, I have an XML file that contains multiple nodes that define the source directory and filename "mask" (e.g. <SOURCEPATH>c:\myDirectory</SOURCEPATH> <SOURCEFILE>abc*</SOURCEFILE>)

For each node, I need to set up a file watcher to raise an event if a new file matching this criteria is added.

Once done, I will process the file, which includes opening it, reading some data, closing it, then copying it to another directory (also defined in the XML). I will then add the file info to a list control on a form.

I have a method that parses the XML file and reads the nodes. My intention is to create a class object (cImportFile) for each node and load the properties with the data from the child nodes in the XML file.

The kicker is: how do I tie each of these to a FileSystemWatcher. I assume I will need one for each node in the XML file (currently there are 8). And then communicate back to the form which one fired and get the actual file properties back to the main form?

So process flow is:
 Starting project reads XML file
  Process each node, creating a cImportFile object for each one and setting its properties from the XML child nodes data
  Assign a FileSystemWatcher to each cImportFile
  FSW monitors directory
    Have FSW kick off process(es) in the cImportFile object to read data, process it, move the file, communicate back to the form so the list control can be updated.
  Continue monitoring and repeat as necessary

Initially, I would also like the process to act as if an event was raised by the FSW so it would process any files that got moved into the monitored directory while the program was not running.

If you need any clarification, please let me know. Dev environment is VB 2005.
  • 8
  • 6
1 Solution
dbbishopAuthor Commented:
Okay, but to be honest, I am still learning .net and some of this stuff is still a little above my head. I have been a VB 6 programmer for over 10-years, got into a little .net stuff at my last job (still mostly app development).

I could use some hard examples other than the example given in the help files in VS 2005 (which I got working as a console app just to see how it worked). That is just a single FSW watching a single directory for a specific file type. I'll need multiple FSW objects watching several directories for different file patterns.

As I indicated, I would have form code that processes the XML file. I anticipate creating a cImportFile object for each node in the XML file, setting it's properties based on child node values.

Would I instantiate the FSW within each cImportFile class that I create (like in Sub New() ) and set it's properties through the class' properties, and create each cImportFile from my form with and event handler so they can communicate back  to the form? I do not necessarily need to know which instance of cImportFile object raised the event, since all it will be passing back to the form is the path and filename of the file it processed, and possibly a status code.

Also, how would I safely shut this application down. I would have a [Quit] button on the form, but I don't want to risk setting one of the cImportFile objects to Nothing if it is in the middle of moving a file or writing data to my database.

I would envision setting an IsProcessing property in cImportFile to True when the FSW event fired, then setting it to False when I was done with all processing, and set each cImportile object = Nothing only if the IsProcessing property is False. I would override Finalize and set the FSW to nothing in the cImportFile object. Would this be the best way to handle a safe shutdown?
dbbishopAuthor Commented:
One additional question...
Some of the files I will be monitoring will be 10-12MB in size. The processing I am doing is:
  - open a filestream and read the file, determining the number of records in it.
  - write the filename, path and other information to a table in SQL Server.
  - move the file to another directory.

The large files will show up in the directory immediately, but if I do an F5 from within Explorer (refresh) I can see the filesize increasing as the file is being written. It can take 2-3 minutes or longer to completely write the file.

At which point with FSW create an event when a new file is added to the directory? If it is before the file is completely written, is there some way I can determine when the file has been completely written to and closed by the process that is creating it (it is actually being FTP'd from a mainframe onto a server).
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

for your request i'd be more of the designer as opposed to the programmer, but let me see if i have this all correct as far as what you are wanting and i may be able to put some of it together.

this is how i would organize what you want:

1) a watchdog program that will look at a specific directory to see if any new files exists
2) move the file to a new directory and copy the name into a variable.
3) the watchdog program will then compare the file extension to an XML file. (it will read the xml file only after it has located a new file in the directory, this way you can make dynamic changes without having to restart the watchdog)
depending on the results it will run a series of scripts or exes or functions(depending on how you want to do that part)
4) then release the variable and wait for another file to enter the directory.

the watchdog program could be setup as a service that runs all the other programs or script or whatnot independently.

* Sample XML

<?xml version="1.0" encoding="utf-8" ?>
  <hero type="superhero">Superman</hero>

* Sample script to read XML

  Private Sub Page_Load(ByVal sender As System.Object,
    ByVal e As System.EventArgs) Handles MyBase.Load
    Dim xmlRoot As New SNANET.XMLParser.XMLRoot(
    If xmlRoot.isLoaded Then
    For Each item As SNANET.XMLParser.XMLNode
     In xmlRoot.GetNodesByType("type","superhero")
        Response.Write(item.Name & "=" & item.value)
  End If
  End Sub

* results
superhero = superman

* monitor file sub pulled from http://msdn.microsoft.com/msdnmag/issues/01/07/vbnet/figures.asp
Private Class MonitorFiles
    Private Shared BITraceLevel As New TraceSwitch(myName, _
        "Batch Importer Trace Level")
    Const dirpath = "C:\temp\"

    Public Sub StartMonitor()
        Dim fw As New FileSystemWatcher()
        Dim result As WaitForChangedResult
        fw.Path = dirpath
        fw.Target = IO.WatcherTarget.File
        fw.IncludeSubdirectories = False
        fw.Filter = "*.xml"            
        AddHandler fw.Created, New FileSystemEventHandler( _
            AddressOf Me.OnFileCreated)
        fw.Enabled = True
              result = fw.WaitForChanged( _
        Catch e As Exception
            'stop monitoring the directory
            fw.Enabled = False
        End Try
    End Sub
End Class
dbbishopAuthor Commented:
The 'problem' I see with this is that I need to process only certain files, defined with a filter, in more than one directory. My thoughts were that for each directory and 'filter' defined in the XML file, I could spawn a class that contained a FSW with the appropriate directory path and file filter to watch for.

Each parent node in the XML file defines the source directory and filename filter to watch, the destination directory the file will be copied to, and some other parameters associated with that particular node.
it seems as if you already have the answer to your own questions.

you should be able to setup the xml file so that you can parse through the seperate levels depending on what you want then run a series of  "for each x in y" loop that calls the watchdog sub for each extention and folderpath.
dbbishopAuthor Commented:
Well, I think I know the processing I want to accomplish. I am very new to ,net framework and as stated more of an application developer than a system developer and this is more on the lines of system level development. Getting a bit more complex than what I've done in the past. Knowing the process and knowing how to accomplish that process are two different things. There is definately more complexity to .net than VB6.
Sometimes just having a sounding-board to bounce ideas off helps. Thanks.
dbbishopAuthor Commented:
weellio - I am going to accept your answer. Although it did not necessarily solve the problem, like I said, just brainsorming it back and forth can sometimes help you think a little clearer. Except for a minor issue, I do have it up and running.

I am certain I am going to have to learn and use thread pooling with this eventually to avoid collisions. I've got 8 watchers and each one is eventually firing an event back to the form, so a collision is just waiting to happen (if I knew what that meant :-))

I do know that the files are not received on the server simultaneously, but are rather created one after another, can take several minutes to create, and the processing I do only takes a couple of seconds, if that long, so hopefully, until I get a better handle on using threads, I should be safe.

Thanks for being my sounding-board.
we can still hash out the details of what you are looking for,. keep replying and so will i.

i'm assuming you will have the xmls set up something like this.

+ <Ext>
        <script>c:\scripts\programs.exe txt</script>
  + <Ext>
        <script>c:\scripts\programs.exe jpg</script>
  + <Ext>
        <script>c:\scripts\programs.exe mdb</script>
here is a simple instance of thread pooling
dbbishopAuthor Commented:
Thanks - I'll provide a bit mor detail tomorrow.
dbbishopAuthor Commented:
<?xml version="1.0" encoding="utf-8" ?>

In the form code, for each node (<COPYJOB> I create an instance of cImportFile add and event handler to the form and add the class to an array.

The cImportFile has a FileSystemWatcher that is set up using properties passed to it from the form that come from the XML file.

 When the FSW fires, it reads the input file to get some file statistics, copies the file from the source directory to the destination directory, and writes information to a SQL Server database. After this code has been executed in the class, it raises an event passing some information in EventArgs. The form captures the event and adds an item to a listview control. The only controls on the form are the listview and a command button to shut down the application.

My only concern is that the FSW fires as soon as the file is created. However, the filestream cannot be opened until the file is done being written by the host process. If the attempt to open the file throws an error, I wait for 15 seconds, then tries again. The host process that creates these files currently creates 8 files, one after another. The total process of creating these files takes about 10-12 minutes.

Assume the following:

- A file is written to a source directory.
- The FSW fires.
- The file cannot be opened, so I sleep for 15 seconds. I go through this TRY/CATCH loop 5 times.
- One second after my last attempt, the file is completely copied to the source directory and the host process closes the file.
- The host process then immediately starts writing the next file.
- The FSW in another class fires, tries to open the file, fails and is now sleeping for 15 seconds.

I am not sure this would even cause a problem. My main 'concern' is that I am using System.Threading.Thread.Sleep(15000) to wait within the loop that tries to open the file. Not using thread pooling and putting each instance of cImportFile into a separate thread, I suspect that is putting my whol application to sleep. And if that is the case, and a new file comes across while it is sleeping, the FSW will not fire.

I don't know anything about using threads (yet). However, I do know that sooner or later I will need to delve more into thread pooling.
take a look here
Encapsulates operating systemspecific objects that wait for exclusive access to shared resources.
you can utilize the example subs on the page in your code to implement a waithandle to waitfor the file to finish copying, thus not having to rely on the try/catch loops
dbbishopAuthor Commented:
weellio: I'm not too certain (for some reason, a lot of the thread-pooling/multithreading is going over my head), but unless I am missing something, I do not think the waithandle will work for me. My process has nothing to do with, and has absolutely no communication with, the process that initiates the initial file copy. It is an FTP process initiated from a mainfram computer. All I have access to is the event fired by the FSW when it detects a new file is being created.

I am still studying the codeguru stuff you sent me along with getting Invoke to work properly in my form (per a question I've posted at http://www.experts-exchange.com/Programming/Languages/.NET/Visual_Basic.NET/Q_22738709.html).

This is all still really mind-boggling to me.

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 8
  • 6
Tackle projects and never again get stuck behind a technical roadblock.
Join Now