?
Solved

Multi-threading and windows service application help needed

Posted on 2011-05-10
9
Medium Priority
?
537 Views
Last Modified: 2012-08-13
Hi

I have a requirement for a messaging application which can loop through any number of databases and use the data to assemble messages and process. The databases are independent of each other but share the same application logic. I'm writing in C# and  using the .net framework v3.5.

I would like to create a windows service application which does the following:

1. At timed intervals, say 30 seconds, loops through and checks each database to see if any messages need processing
2. Create a new messaging object for each database and run the process on its own thread until the work is finished

Each new object may take from anything between a few milliseconds to a minute to complete.

I'm new to threading so I'm not sure about the best way to handle this and which technologies are best to use in a  windows service. I have already created the application which works with one database at a time and already made sure that static fields are marked with the ThreadStaticAttribute.

Can someone help get me started with a simple example or suggest any good tutorials for such an exercise?

Many thanks.
Andrew
0
Comment
Question by:j055
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 10

Expert Comment

by:oxyoo
ID: 35735216
Threading is a vast topic and I would recommend that you do some reading first. There is a 3-part video series [1] at TekPub (commercial) that would get you started.

There is also a nice introductory free e-book [2] by Joseph Albahari

 [1]: http://tekpub.com/view/threading/1
 [2]: http://www.albahari.com/threading


Good Luck!
0
 
LVL 11

Expert Comment

by:Sudhakar Pulivarthi
ID: 35736000
Hi j055,

use ThreadPool which is maintained by CLR and provides a simple mechanism to insert work items into queue and rest creation/destruction of threads are all handled by CLR itself.

Here is an example how to use Thread Pool
http://msdn.microsoft.com/en-us/library/3dasc8as(v=vs.80).aspx
0
 

Author Comment

by:j055
ID: 35736603
Hi there

Thanks for the advice. The Joseph Albahari article has helped so far. As I'm new to threading I do want to keep things as simple as possible. I think the ThreadPool class is something I can work with.

I think the code snippet I've attached below almost does what I need except for one missing requirement. I want to somehow put a lock on each method where it is already running with the same connection string.

In my example you will see that connectionString 'conn2' starts again before it has finished it's first method call. Can you show me a way to ensure this does not happen in my code snippet. Also please feel free to point out any flaws with my approach.

Thanks again
Andrew
using System;
using System.Threading;

namespace ConsoleApp
{
    internal class Program
    {
        private static void Main()
        {
            var connections = new[] {"conn1", "conn2", "conn3"};

            // use System.Timers.Timer to keep foreach loop running continuously in production
            for (var i = 0; i < 2; i++)
            {
                foreach (var connectionString in connections)
                {
                    ThreadPool.QueueUserWorkItem(Go, connectionString);
                    Thread.Sleep(1000);
                }
            }

            Console.ReadLine();
        }

        private static void Go(object data) // data will be null with the first call.
        {
            Console.WriteLine("Started: {0}", data);

            var connectionString = (string) data;

            if (connectionString == "conn2")
                Thread.Sleep(5000);

            Console.WriteLine("Finished: {0}", data);
        }
    }
}

Open in new window

0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 11

Expert Comment

by:Sudhakar Pulivarthi
ID: 35737082
Hi,
Please test with the code ur scenario is taken care.. only after conn2 is finished then only it starts next time.
 
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Collections;

namespace TestApp
{
    class Program
    {
        static ArrayList lockers = new ArrayList();

        private static void Main()
        {
            string[] connections = {"conn1", "conn2", "conn3"};

            // Set the lockers to lock for each connection string
            lockers.AddRange(connections);

            // use System.Timers.Timer to keep foreach loop running continuously in production
            for (int i = 0; i < 2; i++)
            {
                foreach (string connectionString in connections)
                {
                    ThreadPool.QueueUserWorkItem(Go, connectionString);
                    Thread.Sleep(1000);
                }
            }

            Console.ReadLine();
        }

        //private static void Go(object data) // data will be null with the first call.
        //{
        //    string connectionString = (string)data;
            
        //        Console.WriteLine("Started: {0}", data);

        //        if (connectionString == "conn2")
        //            Thread.Sleep(5000);

        //        Console.WriteLine("Finished: {0}", data);            
        //}


        private static void Go(object data) // data will be null with the first call.
        {
            string connectionString = (string)data;

            lock (lockers[lockers.IndexOf(data)])
            {
                Console.WriteLine("Started: {0}", data);

                if (connectionString == "conn2")
                    Thread.Sleep(5000);

                Console.WriteLine("Finished: {0}", data);
            }
        }

    }
}

Open in new window

0
 
LVL 11

Expert Comment

by:Sudhakar Pulivarthi
ID: 35737120
Locks can be applied only on objects, Since string is a built in type which behaves as value type. So we have put the data into array list as objects and we locked it before processing in the 'GO' method.
0
 

Author Comment

by:j055
ID: 35747190
Hi

This certainly looks promising. I've realized that I have to do something with my shared fields as using [ThreadStatic] attributes doesn't help with the ThreadPool class.

Thanks for your help.
Andrew
0
 
LVL 8

Expert Comment

by:Volox
ID: 35747462
I'll provide you a couple of notes on this...

First, if you only want a connection used by one thread at a time, then why not just assign one worker thread per connection?  If you only have one thread for a connection then you don't have to worry about locking the connection.

Second, you may want to consider creating and / or opening your connections in your main method and then providing those open connections to the threads (it kind of depends on what you are doing).  Consider this... if you have a thread run every 30 seconds that opens it's own connection and then closes it, you are opening and closing a connection every 30 seconds.  On the other hand, if you leave it open, you are consuming an open connection.  So you need to consider the amount of time the connection will be idle between being used versus the amount of overhead required to open and close it each time.

Next comment is in regard to error handling...  Your "Go"  method needs a try catch around the entire content of the method that does something useful with any potential unhandled errors.  If an exception is thrown within your code and you don't have this it will just crash the thread and you will have no idea what happened.

I'm not sure how many target databases you are aiming at, but you should know that the thread pool is (normally) 20 threads.  So if you have more databases than are in the pool, the work for the [pool limit] + 1 database won't happen until the first thread completes.  If that works for you, then great, but if you truely wanted to do the work against EVERY database on 30 seconds on the dot, then that wouldn't be what you would be getting.

The other thing about timing that I'd mention is that if you do your work and then sleep for 30 seconds, you are going to get variable intervals between your starts.  Again, this may be ok for what you want, but if you desire start times that are consistently 30 seconds apart regardless of the length of time the work took, then you need to track your start time and do a little math about how long to sleep.  And if you need super accurate timing you should know that Sleep won't give that to you.

And while we're on the subject of sleep.. Thee are those that will say Sleep isn't really recommended for usage in production level code.  They generally recommend having the Thread wait on itself with a timeout value instead.  I'm sure a little google-ing will get you a number of pages that describe the details behind this.

Not sure what the "i < 2" limit is for in your loop, but both code examples will essentially run each connection twice and then end.  If you really want to loop indefinitely, then you want a static variable that signals to stop and code at the end of the Go method to re-queue a new thread for the same connection.  You might consider putting the re-queue in the finally block of that try catch so that a new thread gets scheduled even if the prior run bombed out.

Last but not least, you may want your application to wait for all threads to stop before ending.  That is the part that usually gets a bit more complicated and I'm not going to cover all the details right now but to point you in the right direction you essentially need to use a thread-safe manner of signaling between the running threads and the main application code - look up information on WaitHandles.

I've modified your sample code to incorporate some of these ideas...
using System;
using System.Threading;

namespace ConsoleApp
{
    internal class Program
    {
        static bool isStopping = false;

        private static void Main()
        {
            var connections = new[] {"conn1", "conn2", "conn3"};

            // use System.Timers.Timer to keep foreach loop running continuously in production
                foreach (var connectionString in connections)
                {
                    ThreadPool.QueueUserWorkItem(Go, connectionString);
                    // Don't need this sleep unless you are trying to offset the start of the threads
                    // Thread.Sleep(1000);
                }

            Console.ReadLine();
            isStopping = true;
        }

        private static void Go(object data) // data will be null with the first call.
        {
            try
            {
               Console.WriteLine("Started: {0}", data);

               var connectionString = (string) data;

               if (connectionString == "conn2")
                   Thread.Sleep(5000);

               Console.WriteLine("Finished: {0}", data);
            }
            catch
            {
               // Do something here for unhandled errors
            }
            finally
            {
               if (! isStopping)
               {
                  ThreadPool.QueueUserWorkItem(Go, connectionString);
               }
            }
        }
    }
}

Open in new window

0
 

Author Comment

by:j055
ID: 35770541
Hi

Thanks for all the comments. I realize I can make the application better by implementing some of your suggestions. As far as the timer is concerned I want to make sure all the databases don't get hit at the same time. That's why I put a Sleep of a second in the loop. Timing precision is not important, i.e. the application only needs to connect to each database somewhere within a minute. If a cycle was missed completely in wouldn't be the end of the world. What's important is how the server resources get used. I'd like to spread the load as much as possible. Sleeping the thread doesn't seem to use any CPU. Is that correct? I can't find a better alternative to using Thread.Sleep on this basis.

I'm not sure yet whether I want to keep connections open or just open and close them immediately but I think the later at the moment.

Thanks
Andrew

0
 
LVL 8

Accepted Solution

by:
Volox earned 2000 total points
ID: 35773999
You are correct that Sleep doesn't consume CPU resources; that isn't the issue.  It's more of a matter of how it behaves... sleep supposedly stops the behind the scenes messaging (SendMessage and COM pumping) of the thread and can cause it to orphan if the main thread dies and / or can cause the application to hard lock up.  If you use Thread.CurrentThread.Join(thimeout) instead (so you are waiting the thread on itself), then the background messaging continues.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article describes a simple method to resize a control at runtime.  It includes ready-to-use source code and a complete sample demonstration application.  We'll also talk about C# Extension Methods. Introduction In one of my applications…
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question