Solved

Limiting the number of running threads in an asp .net c# web application.

Posted on 2009-04-08
13
814 Views
Last Modified: 2013-12-17

I've asked this question before but it seems I keep coming back around to it. I have an application that uses the WebClient class to load web site internally. It does this like so:

               Uri siteUri = new Uri("http://www.somewhere.com");
               client = new WebClient();
               String result = client.DownloadString(uri);

The site pointed to by siteUri is actually an internal web application that takes a url parameter that tells it to generate a specific document and place it in a given folder. The result of the page after it loads is extremely simple as the site was create with this purpose in mind. If it generated the doc successfully, it says success; otherwise, it says failure.

My issue is that I will have a batch of requests wherein anywhere from 1 to 300 documents need to be generated. Part of my requirements is that I cannot have more than 3 threads connected to this internal site at the same time. I think the spirit of this request is to ensure one application doesn't bog down the mainframe.

Currently I've implemented this in C# using a semaphore. This is my first experience with threading so I'm second guessing myself every step of the way. What I do is I have a class that creates an array of threads. Each thread is passed the id of the doc to generate and the method that is used to generate that document. The class code is below.

I'm open to any and all suggestions. Like I said, this is my first threaded application so I'll take all the help I can get. Just try to speak in short simple words easy to understand. (lol)
using System;
using System.Data;
using System.Configuration;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
using System.IO;
 
 
public class docGeneration {
 
     //Used for object locking for log file
     static Object writeLogLock = new Object();
 
     //Used for object locking for doc/thread status
     static Object statusLogLock = new Object();
 
     //Don't think this is used...
     static Object threadCountLock = new Object();
 
     //Maximum # of concurrent threads
     static int maxThreads = 3;
     
     //Controls max # of concurrent threads
     private static Semaphore semaphore ;
 
     private docGeneration() {
     }
 
     public docGeneration(List<String> lstSequences, String folder, String side) {
 
          if (docGeneration.status != null)
               docGeneration.status.Clear();
 
          this.Sequences = lstSequences;
          this.Folder = folder;
          this.Side = side;
     }
 
     //List that contains the results of each doc being processed. Used by Default.aspx.cs / calling code.
     public static List<String> status = new List<string>();
 
     public static String logPath = String.Empty;
 
     
     private Thread[] _requests;
 
     //Property so that calling code can know if threads are still being processed
     public Boolean ThreadsAlive {
          get {
               if (_requests != null) {
                    int cnt = _requests.Length;
                    for (int i = 0; i < cnt; i++) {
                         if (_requests[i].IsAlive) {
                              return true;
                         }
                    }
               }
               return false;
          }
     }
 
     private List<String> _sequences;
 
     public List<String> Sequences {
          get {
               return _sequences;
          }
          set {
               _sequences = value;
          }
     }
 
     private String _folder;
 
     public String Folder {
          get {
               return _folder;
          }
          set {
               _folder = value;
          }
     }
 
     private String _side;
 
     public String Side {
          get {
               return _side;
          }
          set {
               _side = value;
          }
     }
 
     /*
      * Method that queues each doc to be generated in its own thread. A Semaphore
      * is used to limit the maximum number of concurrent threads to 3 as that is
      * a requirement.
      **/ 
     public void RunRev() {
          /* hold all available semaphore slots till threads are created */
          try
          {
               semaphore = new Semaphore(0, maxThreads);
               int c = Sequences.Count;
               this._requests = new Thread[c];
               writeLog("   /*************************BEGIN ENTRY************* ");
               for (int i = 0; i < c; i++)
               {
                    String s = Sequences[i];
                    _requests[i] = new Thread(new ParameterizedThreadStart(grabDocsWebClient));
                    _requests[i].Name = s;
                    _requests[i].Start(s);
                    if((i + 1) == maxThreads)
                         semaphore.Release(maxThreads);
               }
               /* make maxThreads (3) number of threads available */
               //semaphore.Release(maxThreads);
          }
          catch (Exception e)
          {
               writeLog("ERROR: " + e.Message + ". Occurred in  of . \n STACK TRACE: " + e.StackTrace);
          }
     }
     
     //So Default.aspx has a way to know if docs weren't generated
     public static void writeStatus(String logMessage) {
          lock (statusLogLock) {
               status.Add(logMessage);
          }
     }
 
     //Writes message to a log file
     public static void writeLog(String logMessage) {
          lock (writeLogLock) {
               String logFile = docGeneration.logPath;
               using (StreamWriter logEntry = new StreamWriter(logFile, true)) {
                    logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);
                    logEntry.Close();
               }
          }
     }
 
     public static void writeLog(String path, String logMessage)
     {
          lock (writeLogLock)
          {
               String logFile = path;
               using (StreamWriter logEntry = new StreamWriter(logFile, true))
               {
                    logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);
                    logEntry.Close();
               }
          }
     }
     /*
      * Method loads another web page designed to retrieve these docs from the main frame and downloads
      * the results of the page generated. If the docs was generated, the word 'generated' will exist in the 
      * downloaded string.
      * 
      * Note that semaphore.WaitOne(); will force the thread to wait until there is a freed thread
      *  semaphore.Release(1); releases a spot in the allocated threads from the semaphore
      */
     public void grabDocsWebClient(Object strSequence) {
 
          WebClient client = null;
          String _strSequences = (String)strSequence;
          String seqNum = _strSequences.Substring(0, _strSequences.Length - 1);
          String folderName = this.Folder;   
          String err = "";
          String uri = null;
          try {
 
               //THREAD BLOCKING TESTING
               //writeLog(String.Format("docGeneration: {0} WAITING ", Thread.CurrentThread.Name + " STATE: " + Thread.CurrentThread.ThreadState));
 
               semaphore.WaitOne();
               uri = "http://internalsite/begin?" + seqNum;
               Uri siteUri = new Uri(uri);
               client = new WebClient();
 
               String result = client.DownloadString(uri);
               
               if (result != null && result.Contains("generated"))
               {
                    writeStatus(String.Format("SUCCESS: {0} ", _strSequences));
                    writeLog(String.Format("SUCCESS: {0} ", _strSequences));
               }
               else
               {
                    writeStatus(String.Format("FAILED: {0} ", _strSequences));
                    writeLog(String.Format("FAILED: {0} ", _strSequences));
               }      
               //client.DownloadStringAsync(siteUri);
               //client.DownloadStringCompleted += Test;
           
          } catch (Exception e) {
               err = e.Message;
               writeStatus(String.Format("ERROR: {0} ", _strSequences));
               writeLog(String.Format("ERROR: {0} ", _strSequences));
               writeLog("ERROR: " + e.Message + " FOR SEQUENCE: " + _strSequences + "\n" + "STACK TRACE: " + e.StackTrace);
          } finally {
               writeLog(String.Format(": {0} FINISHED. STATE: {1} ", Thread.CurrentThread.Name , Thread.CurrentThread.ThreadState));
               semaphore.Release(1);
          }
     }
 
     private void Test(Object sender, DownloadStringCompletedEventArgs args) 
     {
          String result = args.Result;
 
          if (result != null && result.Contains("generated"))
          {
               writeStatus(String.Format("SUCCESS: {0} ", result));
               writeLog(String.Format("SUCCESS: {0} ", result));
          }
          else
          {
               writeStatus(String.Format("FAILED: {0} ", result));
               writeLog(String.Format("FAILED: {0} ", result));
          }          
     }
 
}

Open in new window

0
Comment
Question by:baijajusav
  • 7
  • 4
  • 2
13 Comments
 
LVL 19

Expert Comment

by:daveamour
ID: 24104192
This is an asp.net app right?
If so what is the context and scope of your docGeneration class and you didn't say what is wrong with your current code?
0
 
LVL 83

Accepted Solution

by:
CodeCruiser earned 500 total points
ID: 24105105
Although it would require a radical change but i would have implemented it a bit differently. I would use a backgroudworker control because its easy to work with. I would use a queue to store the documents names that need to be processed. When starting the process, i would run the 3 background workers manualy using items from the queue. Then in the Complete event of each backgroundworker, i would check the queue to see if there are any more items to process and if there are then start the backgroundworker again with a new item to process.
Hope it makes sense.
0
 
LVL 19

Expert Comment

by:daveamour
ID: 24105154
This reminds me of a question I answered a while ago.
Its worth having a look at that thread as it may give some inspiration as it's simillar in many ways.
http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_24053329.html?sfQueryTermInfo=1+1+10+counter+dtgurlgrid.rows.count
 
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 3

Author Comment

by:baijajusav
ID: 24107032

daveamour:my issue is that my test environment is a bit finicky so I can't discern if my code is failing or if its just a bad environment. This being my first threaded app, I'm a bit self conscious about the code as I'm not sure what it's supposed to look like so to speak.

CodeCruiser:I like that idea. Can the background worker control be used in an asp .net application? I don't see it in the controls. Still, I see your point and I might try demoing an app up that works like you describe. I'll store the document id's in a list and lock the list when a thread goes to get the next doc id.

daveamour:Looking over your code now.

Thank you all! I know this question is both difficult (for me) and relative so I appreciate any and all input.
0
 
LVL 3

Author Comment

by:baijajusav
ID: 24107188

To touch on CodeCruiser's idea, let me verify that I understand how to lock variables so that several threads don't access a variable at the same time.

From what I've read, the easiest way to do locking is declare a static object variable. Then whenever you want to lock something, use that variable. So, is the following going to prevent more than one thread from accessing it at a time?

Queue<String> qSequences = new Queue<string>();
static Object queueLock = new Object();

lock(queueLock )
{
      String docId = qSequences.Dequeue();
}

I figure that I will just start up 3 threads and let them run. They will call a method whose body is a loop. The loop will run so long there are items in the queue. Inside of the loop, I'll call the code to generate the documents and such. Err...did that make sense?

Thoughts?
0
 
LVL 3

Author Comment

by:baijajusav
ID: 24107793


Okay, I've got a demo application up and it works awesomely. To work around the test environment issues, I made a down and dirty web app that simulates taking a parameter off the url and doing something with it (namely, it just writes it out to the page). This is so the WebClient's DownloadString method actually has something to do in my testing.

I don't know why I didn't think of this sooner. I had my mind so stuck on having 1 thread per document that I didn't see the simple solution of limiting the number of concurrent threads...simply only use 3 threads. Duh!!

I'll post code later on.

CodeCruiser, looks like you'll be getting the points.

Thanks for all your help guys!
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 24110701
Glad that my suggestion was useful for you.
0
 
LVL 3

Author Comment

by:baijajusav
ID: 24115772

Okay, here is my demo code / arrived solution.

Thank you again for your participation. It's amazing how stepping back and rethinking the issue simplifies things.
using System;
using System.Data;
using System.Configuration;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
using System.Collections;
using System.Collections.Generic;
using System.Threading;
using System.Net;
using System.IO;
 
public partial class _Default : System.Web.UI.Page 
{
    //Used for object locking for log file
    static Object writeLogLock = new Object();
 
    //Used for object locking for image/thread status
    static Object statusLogLock = new Object();
 
    //Don't think this is used...
    static Object queueLock = new Object();
 
    Thread[] _threads;
 
    String path = "";
 
    //Property so that calling code can know if threads are still being processed
    public Boolean ThreadsAlive
    {
        get
        {
            if (_threads != null)
            {
                int cnt = _threads.Length;
                for (int i = 0; i < cnt; i++)
                {
                    if (_threads[i].IsAlive)
                    {
                        return true;
                    }
                }
            }
            return false;
        }
    }
 
    static Queue<String> qSequences = new Queue<string>();
 
    protected void Page_Load(object sender, EventArgs e)
    {
        qSequences.Clear();
        String appPath = this.Request.ApplicationPath;
        path = this.Request.MapPath(appPath);
 
        qSequences.Enqueue("123");
        qSequences.Enqueue("2");
        qSequences.Enqueue("3");
        qSequences.Enqueue("4");
        qSequences.Enqueue("5");
        qSequences.Enqueue("6");
        qSequences.Enqueue("7");
        qSequences.Enqueue("8");
        qSequences.Enqueue("9");
        qSequences.Enqueue("0");
        qSequences.Enqueue("01");
        qSequences.Enqueue("02");
        _threads = new Thread[3];
        for (int i = 0; i < 3; i++)
        {
            String docid = "";
            lock (queueLock)
            {
                docid = qSequences.Dequeue();
            }            
            _threads[i] = new Thread(new ParameterizedThreadStart(grabDocsWebClient));
            _threads[i].Name = docid;
            _threads[i].Start(docid);
        }
 
        while (ThreadsAlive)
        {
            Thread.Sleep(2000);
        }
 
    }
 
 
    public void writeLog(String logMessage)
    {
        lock (writeLogLock)
        {
            String logFile = path + "//ThreeThreads.txt";
            using (StreamWriter logEntry = new StreamWriter(logFile, true))
            {
                logEntry.WriteLine(DateTime.Now.ToString() + " " + logMessage);
                logEntry.Close();
            }
        }
    }
 
   
    public void grabDocsWebClient(Object pDocId)
    {
        WebClient client = null;
        String docid = (String)pDocId;
        String uri = null;
        try
        {
 
            while (docid != null && !String.Empty.Equals(docid))
            {
		//Test web site to screen scrape
                uri = "http://localhost/WebClientTester?docid=" + docid;
                Uri siteUri = new Uri(uri);
                client = new WebClient();
 
                String result = client.DownloadString(uri);
 
                if (result != null && result.Length > 17)
                {
                    writeLog(String.Format("SUCCESS: {0} ", docid));
                }
                else
                {
                    writeLog(String.Format("FAILED: {0} ", docid));
                }
 
                lock (queueLock)
                {
                    if (qSequences.Count > 0)
                    {
                        docid = qSequences.Dequeue();
                    }
                    else
                    {
                        docid = null;
                    }
                }      
            }
        }
        catch (Exception e)
        {
            writeLog(String.Format("ERROR: {0} ", e.Message));
        }
        finally
        {
        }
    }
}

Open in new window

0
 
LVL 3

Author Comment

by:baijajusav
ID: 24118425

Okay, my demo app worked but now that I'm incorporating it into my main app, I'm getting a wierd behavior.

In my while loop of the method that is being called in each thread, the code:

lock (queueLock)
                {
                    if (qSequences.Count > 0)
                    {
                        docid = qSequences.Dequeue();
                    }
                    else
                    {
                        docid = null;
                    }
                }      

never gets hit. I've set break points and everything and it gets passed right over. Any thoughts? (even though the question is closed)
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 24118449
I will try to answer your question but right now my brother is on my head and we have to leave for a friend.
0
 
LVL 3

Author Comment

by:baijajusav
ID: 24118463
TY! Hope you get your brother-on-head-itis issue taken care of.
0
 
LVL 3

Author Comment

by:baijajusav
ID: 24118586

Found the issue. I needed a try/catch around the String result = client.DownloadString(uri) call because if it errored, the while loop would be escaped.

Problem solved.

Thanks. Again.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 24125440
Glad to hear the problem is solved.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Just a quick little trick I learned recently.  Now that I'm using jQuery with abandon in my asp.net applications, I have grown tired of the following syntax:      (CODE) I suppose it just offends my sense of decency to put inline VBScript on a…
For those of you who don't follow the news, or just happen to live under rocks, Microsoft Research released a beta SDK (http://www.microsoft.com/en-us/download/details.aspx?id=27876) for the Xbox 360 Kinect. If you don't know what a Kinect is (http:…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question