Solved

Limiting the number of running threads in an asp .net c# web application.

Posted on 2009-04-08
13
810 Views
Last Modified: 2013-12-17

I've asked this question before but it seems I keep coming back around to it. I have an application that uses the WebClient class to load web site internally. It does this like so:

               Uri siteUri = new Uri("http://www.somewhere.com");
               client = new WebClient();
               String result = client.DownloadString(uri);

The site pointed to by siteUri is actually an internal web application that takes a url parameter that tells it to generate a specific document and place it in a given folder. The result of the page after it loads is extremely simple as the site was create with this purpose in mind. If it generated the doc successfully, it says success; otherwise, it says failure.

My issue is that I will have a batch of requests wherein anywhere from 1 to 300 documents need to be generated. Part of my requirements is that I cannot have more than 3 threads connected to this internal site at the same time. I think the spirit of this request is to ensure one application doesn't bog down the mainframe.

Currently I've implemented this in C# using a semaphore. This is my first experience with threading so I'm second guessing myself every step of the way. What I do is I have a class that creates an array of threads. Each thread is passed the id of the doc to generate and the method that is used to generate that document. The class code is below.

I'm open to any and all suggestions. Like I said, this is my first threaded application so I'll take all the help I can get. Just try to speak in short simple words easy to understand. (lol)
using System;

using System.Data;

using System.Configuration;

using System.Web;

using System.Web.Security;

using System.Web.UI;

using System.Web.UI.WebControls;

using System.Web.UI.WebControls.WebParts;

using System.Web.UI.HtmlControls;

using System.Collections.Generic;

using System.Text;

using System.Threading;

using System.Net;

using System.IO;
 
 

public class docGeneration {
 

     //Used for object locking for log file

     static Object writeLogLock = new Object();
 

     //Used for object locking for doc/thread status

     static Object statusLogLock = new Object();
 

     //Don't think this is used...

     static Object threadCountLock = new Object();
 

     //Maximum # of concurrent threads

     static int maxThreads = 3;

     

     //Controls max # of concurrent threads

     private static Semaphore semaphore ;
 

     private docGeneration() {

     }
 

     public docGeneration(List<String> lstSequences, String folder, String side) {
 

          if (docGeneration.status != null)

               docGeneration.status.Clear();
 

          this.Sequences = lstSequences;

          this.Folder = folder;

          this.Side = side;

     }
 

     //List that contains the results of each doc being processed. Used by Default.aspx.cs / calling code.

     public static List<String> status = new List<string>();
 

     public static String logPath = String.Empty;
 

     

     private Thread[] _requests;
 

     //Property so that calling code can know if threads are still being processed

     public Boolean ThreadsAlive {

          get {

               if (_requests != null) {

                    int cnt = _requests.Length;

                    for (int i = 0; i < cnt; i++) {

                         if (_requests[i].IsAlive) {

                              return true;

                         }

                    }

               }

               return false;

          }

     }
 

     private List<String> _sequences;
 

     public List<String> Sequences {

          get {

               return _sequences;

          }

          set {

               _sequences = value;

          }

     }
 

     private String _folder;
 

     public String Folder {

          get {

               return _folder;

          }

          set {

               _folder = value;

          }

     }
 

     private String _side;
 

     public String Side {

          get {

               return _side;

          }

          set {

               _side = value;

          }

     }
 

     /*

      * Method that queues each doc to be generated in its own thread. A Semaphore

      * is used to limit the maximum number of concurrent threads to 3 as that is

      * a requirement.

      **/ 

     public void RunRev() {

          /* hold all available semaphore slots till threads are created */

          try

          {

               semaphore = new Semaphore(0, maxThreads);

               int c = Sequences.Count;

               this._requests = new Thread[c];

               writeLog("   /*************************BEGIN ENTRY************* ");

               for (int i = 0; i < c; i++)

               {

                    String s = Sequences[i];

                    _requests[i] = new Thread(new ParameterizedThreadStart(grabDocsWebClient));

                    _requests[i].Name = s;

                    _requests[i].Start(s);

                    if((i + 1) == maxThreads)

                         semaphore.Release(maxThreads);

               }

               /* make maxThreads (3) number of threads available */

               //semaphore.Release(maxThreads);

          }

          catch (Exception e)

          {

               writeLog("ERROR: " + e.Message + ". Occurred in  of . \n STACK TRACE: " + e.StackTrace);

          }

     }

     

     //So Default.aspx has a way to know if docs weren't generated

     public static void writeStatus(String logMessage) {

          lock (statusLogLock) {

               status.Add(logMessage);

          }

     }
 

     //Writes message to a log file

     public static void writeLog(String logMessage) {

          lock (writeLogLock) {

               String logFile = docGeneration.logPath;

               using (StreamWriter logEntry = new StreamWriter(logFile, true)) {

                    logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);

                    logEntry.Close();

               }

          }

     }
 

     public static void writeLog(String path, String logMessage)

     {

          lock (writeLogLock)

          {

               String logFile = path;

               using (StreamWriter logEntry = new StreamWriter(logFile, true))

               {

                    logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);

                    logEntry.Close();

               }

          }

     }

     /*

      * Method loads another web page designed to retrieve these docs from the main frame and downloads

      * the results of the page generated. If the docs was generated, the word 'generated' will exist in the 

      * downloaded string.

      * 

      * Note that semaphore.WaitOne(); will force the thread to wait until there is a freed thread

      *  semaphore.Release(1); releases a spot in the allocated threads from the semaphore

      */

     public void grabDocsWebClient(Object strSequence) {
 

          WebClient client = null;

          String _strSequences = (String)strSequence;

          String seqNum = _strSequences.Substring(0, _strSequences.Length - 1);

          String folderName = this.Folder;   

          String err = "";

          String uri = null;

          try {
 

               //THREAD BLOCKING TESTING

               //writeLog(String.Format("docGeneration: {0} WAITING ", Thread.CurrentThread.Name + " STATE: " + Thread.CurrentThread.ThreadState));
 

               semaphore.WaitOne();

               uri = "http://internalsite/begin?" + seqNum;

               Uri siteUri = new Uri(uri);

               client = new WebClient();
 

               String result = client.DownloadString(uri);

               

               if (result != null && result.Contains("generated"))

               {

                    writeStatus(String.Format("SUCCESS: {0} ", _strSequences));

                    writeLog(String.Format("SUCCESS: {0} ", _strSequences));

               }

               else

               {

                    writeStatus(String.Format("FAILED: {0} ", _strSequences));

                    writeLog(String.Format("FAILED: {0} ", _strSequences));

               }      

               //client.DownloadStringAsync(siteUri);

               //client.DownloadStringCompleted += Test;

           

          } catch (Exception e) {

               err = e.Message;

               writeStatus(String.Format("ERROR: {0} ", _strSequences));

               writeLog(String.Format("ERROR: {0} ", _strSequences));

               writeLog("ERROR: " + e.Message + " FOR SEQUENCE: " + _strSequences + "\n" + "STACK TRACE: " + e.StackTrace);

          } finally {

               writeLog(String.Format(": {0} FINISHED. STATE: {1} ", Thread.CurrentThread.Name , Thread.CurrentThread.ThreadState));

               semaphore.Release(1);

          }

     }
 

     private void Test(Object sender, DownloadStringCompletedEventArgs args) 

     {

          String result = args.Result;
 

          if (result != null && result.Contains("generated"))

          {

               writeStatus(String.Format("SUCCESS: {0} ", result));

               writeLog(String.Format("SUCCESS: {0} ", result));

          }

          else

          {

               writeStatus(String.Format("FAILED: {0} ", result));

               writeLog(String.Format("FAILED: {0} ", result));

          }          

     }
 

}

Open in new window

0
Comment
Question by:baijajusav
  • 7
  • 4
  • 2
13 Comments
 
LVL 19

Expert Comment

by:daveamour
Comment Utility
This is an asp.net app right?
If so what is the context and scope of your docGeneration class and you didn't say what is wrong with your current code?
0
 
LVL 83

Accepted Solution

by:
CodeCruiser earned 500 total points
Comment Utility
Although it would require a radical change but i would have implemented it a bit differently. I would use a backgroudworker control because its easy to work with. I would use a queue to store the documents names that need to be processed. When starting the process, i would run the 3 background workers manualy using items from the queue. Then in the Complete event of each backgroundworker, i would check the queue to see if there are any more items to process and if there are then start the backgroundworker again with a new item to process.
Hope it makes sense.
0
 
LVL 19

Expert Comment

by:daveamour
Comment Utility
This reminds me of a question I answered a while ago.
Its worth having a look at that thread as it may give some inspiration as it's simillar in many ways.
http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_24053329.html?sfQueryTermInfo=1+1+10+counter+dtgurlgrid.rows.count
 
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility

daveamour:my issue is that my test environment is a bit finicky so I can't discern if my code is failing or if its just a bad environment. This being my first threaded app, I'm a bit self conscious about the code as I'm not sure what it's supposed to look like so to speak.

CodeCruiser:I like that idea. Can the background worker control be used in an asp .net application? I don't see it in the controls. Still, I see your point and I might try demoing an app up that works like you describe. I'll store the document id's in a list and lock the list when a thread goes to get the next doc id.

daveamour:Looking over your code now.

Thank you all! I know this question is both difficult (for me) and relative so I appreciate any and all input.
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility

To touch on CodeCruiser's idea, let me verify that I understand how to lock variables so that several threads don't access a variable at the same time.

From what I've read, the easiest way to do locking is declare a static object variable. Then whenever you want to lock something, use that variable. So, is the following going to prevent more than one thread from accessing it at a time?

Queue<String> qSequences = new Queue<string>();
static Object queueLock = new Object();

lock(queueLock )
{
      String docId = qSequences.Dequeue();
}

I figure that I will just start up 3 threads and let them run. They will call a method whose body is a loop. The loop will run so long there are items in the queue. Inside of the loop, I'll call the code to generate the documents and such. Err...did that make sense?

Thoughts?
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility


Okay, I've got a demo application up and it works awesomely. To work around the test environment issues, I made a down and dirty web app that simulates taking a parameter off the url and doing something with it (namely, it just writes it out to the page). This is so the WebClient's DownloadString method actually has something to do in my testing.

I don't know why I didn't think of this sooner. I had my mind so stuck on having 1 thread per document that I didn't see the simple solution of limiting the number of concurrent threads...simply only use 3 threads. Duh!!

I'll post code later on.

CodeCruiser, looks like you'll be getting the points.

Thanks for all your help guys!
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 83

Expert Comment

by:CodeCruiser
Comment Utility
Glad that my suggestion was useful for you.
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility

Okay, here is my demo code / arrived solution.

Thank you again for your participation. It's amazing how stepping back and rethinking the issue simplifies things.
using System;

using System.Data;

using System.Configuration;

using System.Web;

using System.Web.Security;

using System.Web.UI;

using System.Web.UI.WebControls;

using System.Web.UI.WebControls.WebParts;

using System.Web.UI.HtmlControls;

using System.Collections;

using System.Collections.Generic;

using System.Threading;

using System.Net;

using System.IO;
 

public partial class _Default : System.Web.UI.Page 

{

    //Used for object locking for log file

    static Object writeLogLock = new Object();
 

    //Used for object locking for image/thread status

    static Object statusLogLock = new Object();
 

    //Don't think this is used...

    static Object queueLock = new Object();
 

    Thread[] _threads;
 

    String path = "";
 

    //Property so that calling code can know if threads are still being processed

    public Boolean ThreadsAlive

    {

        get

        {

            if (_threads != null)

            {

                int cnt = _threads.Length;

                for (int i = 0; i < cnt; i++)

                {

                    if (_threads[i].IsAlive)

                    {

                        return true;

                    }

                }

            }

            return false;

        }

    }
 

    static Queue<String> qSequences = new Queue<string>();
 

    protected void Page_Load(object sender, EventArgs e)

    {

        qSequences.Clear();

        String appPath = this.Request.ApplicationPath;

        path = this.Request.MapPath(appPath);
 

        qSequences.Enqueue("123");

        qSequences.Enqueue("2");

        qSequences.Enqueue("3");

        qSequences.Enqueue("4");

        qSequences.Enqueue("5");

        qSequences.Enqueue("6");

        qSequences.Enqueue("7");

        qSequences.Enqueue("8");

        qSequences.Enqueue("9");

        qSequences.Enqueue("0");

        qSequences.Enqueue("01");

        qSequences.Enqueue("02");

        _threads = new Thread[3];

        for (int i = 0; i < 3; i++)

        {

            String docid = "";

            lock (queueLock)

            {

                docid = qSequences.Dequeue();

            }            

            _threads[i] = new Thread(new ParameterizedThreadStart(grabDocsWebClient));

            _threads[i].Name = docid;

            _threads[i].Start(docid);

        }
 

        while (ThreadsAlive)

        {

            Thread.Sleep(2000);

        }
 

    }
 
 

    public void writeLog(String logMessage)

    {

        lock (writeLogLock)

        {

            String logFile = path + "//ThreeThreads.txt";

            using (StreamWriter logEntry = new StreamWriter(logFile, true))

            {

                logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);

                logEntry.Close();

            }

        }

    }
 

   

    public void grabDocsWebClient(Object pDocId)

    {

        WebClient client = null;

        String docid = (String)pDocId;

        String uri = null;

        try

        {
 

            while (docid != null && !String.Empty.Equals(docid))

            {

		//Test web site to screen scrape

                uri = "http://localhost/WebClientTester?docid=" + docid;

                Uri siteUri = new Uri(uri);

                client = new WebClient();
 

                String result = client.DownloadString(uri);
 

                if (result != null && result.Length > 17)

                {

                    writeLog(String.Format("SUCCESS: {0} ", docid));

                }

                else

                {

                    writeLog(String.Format("FAILED: {0} ", docid));

                }
 

                lock (queueLock)

                {

                    if (qSequences.Count > 0)

                    {

                        docid = qSequences.Dequeue();

                    }

                    else

                    {

                        docid = null;

                    }

                }      

            }

        }

        catch (Exception e)

        {

            writeLog(String.Format("ERROR: {0} ", e.Message));

        }

        finally

        {

        }

    }

}

Open in new window

0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility

Okay, my demo app worked but now that I'm incorporating it into my main app, I'm getting a wierd behavior.

In my while loop of the method that is being called in each thread, the code:

lock (queueLock)
                {
                    if (qSequences.Count > 0)
                    {
                        docid = qSequences.Dequeue();
                    }
                    else
                    {
                        docid = null;
                    }
                }      

never gets hit. I've set break points and everything and it gets passed right over. Any thoughts? (even though the question is closed)
0
 
LVL 83

Expert Comment

by:CodeCruiser
Comment Utility
I will try to answer your question but right now my brother is on my head and we have to leave for a friend.
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility
TY! Hope you get your brother-on-head-itis issue taken care of.
0
 
LVL 3

Author Comment

by:baijajusav
Comment Utility

Found the issue. I needed a try/catch around the String result = client.DownloadString(uri) call because if it errored, the while loop would be escaped.

Problem solved.

Thanks. Again.
0
 
LVL 83

Expert Comment

by:CodeCruiser
Comment Utility
Glad to hear the problem is solved.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

A quick way to get a menu to work on our website, is using the Menu control and assign it to a web.sitemap using SiteMapDataSource. Example of web.sitemap file: (CODE) Sample code to add to the page menu: (CODE) Running the application, we wi…
User art_snob (http://www.experts-exchange.com/M_6114203.html) encountered strange behavior of Android Web browser on his Mobile Web site. It took a while to find the true cause. It happens so, that the Android Web browser (at least up to OS ver. 2.…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now