Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

How to do separate threads, when using Filesystem Watcher

Posted on 2011-10-01
5
Medium Priority
?
361 Views
Last Modified: 2012-05-12
I'm using filesystem watcher in a windows service to watch a folder.  The folder will get a comma delimited text file that it will have to parse and write to a sql database.  I have all that working, but I'm getting system hangs when the oncreated event is fired.  

The filesystem watch is ran in a new thread, correct?  Can I create a new thread or a new backgound worker task for my LINQ statements and sql calls?  I'm not sure how to go about this?

Here is a snipper of my code, when the oncreated is fired, it writes the text file to a main linq result list, then I go through that master list 16 times for each of my product types.  I only show two here in the snippet.

   
public partial class ImportServer : ServiceBase
		{                 
			private System.Diagnostics.EventLog eventLog1;         
			private FileSystemWatcher watcher;

			public ImportServer()         
			{
				this.ServiceName = "UnityImport";              
				this.CanHandlePowerEvent = true;             
				this.CanHandleSessionChangeEvent = true;             
				this.CanPauseAndContinue = true;             
				this.CanShutdown = true;             
				this.CanStop = true;             
				this.AutoLog = true;                         
				
				InitializeComponent();                             
				if (!System.Diagnostics.EventLog.SourceExists("UnityImportLogSource"))
					System.Diagnostics.EventLog.CreateEventSource("UnityImportLogSource", "UnityImportLog");

				eventLog1.Source = "UnityImportLogSource";
				eventLog1.Log = "UnityImportLog";         
			}          
			
			protected override void OnStart(string[] args)
			{             
				//base.OnStart(args); 
				eventLog1.WriteEntry("service started");              
				watcher = new FileSystemWatcher();
                watcher.Path = Properties.Settings.Default.watcherPath.ToString();
				watcher.Filter = "";  
				watcher.EnableRaisingEvents = true;
				watcher.Created += new FileSystemEventHandler(OnCreated);
			}          
			
			public void OnCreated(object sender, FileSystemEventArgs e) 
			{
                    eventLog1.WriteEntry("OnCreatedHit");

                    // To Read Use:
                    if (e.Name != null)
                    {
                        string thisFile = Properties.Settings.Default.watcherPath.ToString() + e.Name;
                        try
                        {
                            string[] lines = System.IO.File.ReadAllLines(thisFile);
                            string pattern = ",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))";
                            System.Text.RegularExpressions.Regex r = new System.Text.RegularExpressions.Regex(pattern);
                           
                            var mainResults = from line in lines.AsEnumerable()
                                              let data = r.Split(line.Trim(','))  
                                               where line.Length > 0                                               
                                               select data;
                                                        
                            arraytoDataTable at = new arraytoDataTable();
                            fillNIRDataTable fillDataTable = new fillNIRDataTable();

                                // BMR CORN SILAGE
                                var bmrCornResult = from bmrCorn in mainResults.AsEnumerable()
                                                    where bmrCorn[4].ToString().Contains("BMR Corn")
                                                    select bmrCorn;
                                if (bmrCornResult.Count() > 0)
                                {
                                    DataTable dtbmrCorn = at.makeDatatable(bmrCornResult.ToArray());
                                    DataAccess bmrCornWrite = new DataAccess();
                                    bmrCornWrite.UpsertUnityRaw("[dbo].[sp_updateCornSilageBMR]", dtbmrCorn, "@tableVariable");
                                    // for insert into advantage
                                    //fillNIRDataTable fillBMR = new fillNIRDataTable();
                                    AdsAccess aBMR = new AdsAccess();
                                    aBMR.UpdateNirData(fillDataTable.fillBMRCornSilageDT(dtbmrCorn));
                                }

                                // CORN SILAGE
                                var cornSResult = from cornS in mainResults.AsEnumerable()
                                                  where cornS[4].ToString().Contains("Corn Silage")
                                                  select cornS;
                                if (cornSResult.Count() > 0)
                                {
                                    DataTable dt = at.makeDatatable(cornSResult.ToArray());
                                    DataAccess cornSWrite = new DataAccess();
                                    cornSWrite.UpsertUnityRaw("[dbo].[sp_updateCornSilage]", dt, "@tableVariable");
                                    // for insert into advantage
                                    //fillNIRDataTable fillCorn = new fillNIRDataTable();
                                    AdsAccess aCorn = new AdsAccess();
                                    aCorn.UpdateNirData(fillDataTable.fillCornSilageDT(dt));
                                }

Open in new window

0
Comment
Question by:anderdw2
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 31

Accepted Solution

by:
MlandaT earned 2000 total points
ID: 36899768
There is sample code here showing how you can do the actaul file processing of the detected files on another thread. http://csharp-codesamples.com/2009/02/file-system-watcher-and-large-file-volumes/. The code samples are really nice, clean and to the point.
0
 
LVL 8

Expert Comment

by:Volox
ID: 36904895
You generally don't want to be doing that much work inside of the event from the FileSystemWatcher.

I would also point out that you may want to add some code to your OnCreated event to check that the file is done being written (by whatever is placing it into that folder) and wait for it to finish if it still has the file open.  I've run into cases with large files where FileSystemWatcher will fire the event before the file is done writing and so if you do your reads at that point you only get a portion of the data.  This seems to be especially true if the process placing the file into the folder is FTPing it.
0
 

Author Comment

by:anderdw2
ID: 36905080
Thanks, I'll take a look.  And perhaps this isn't a thread issue.  What I'm noticing, if i place a file in the watcher folder, it seems to go through the process and even moves it.  However, it doesn't update any of my sql statements.  If I place the file in again and debug through the code, it works fine or if I place it in 2x without debugging. Almost as if processes are over-stepping each other.
0
 

Author Comment

by:anderdw2
ID: 36905089
This is a drag and drop scenario, so far.  Basically, we have a comma delimited file from an instrument that we drag to this folder, it processes it and places it into a database.   I'd like to eventually FTP.

So far, the files are like 2k for our testing and they should never be more than 100k each.  However, I could still see FTP beling an issue.
0
 
LVL 31

Expert Comment

by:MlandaT
ID: 36905371
@Volox... I agree with you. I've also had some issues with large files, and incidentally, these files were being dumped into my watch folder via FTP. I came up with a scheme where when the file was created and my watcher first became aware of it, I kept a structure with the FileSize and LastModified time. I only then processed the files only after their size didnt change in a couple of seconds. I used a FileSystemWatcher to detect new files and changes etc, but also ran a timer (on a seperate thread) that polled the files every few seconds - wasnt a problem for me because the files were not dumped too fast and I could pull this kind of stunt. The timer would then queue files that hadnt changed for further processing. I'll dig around and see if I still have that code... it was literally more than 5 years ago.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: Ivo
C# And Nullable Types Since 2.0 C# has Nullable(T) Generic Structure. The idea behind is to allow value type objects to have null values just like reference types have. This concerns scenarios where not all data sources have values (like a databa…
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question