Solved

Multi Thread question?

Posted on 2011-09-30
11
477 Views
Last Modified: 2013-12-16
Hi,

I am trying to debug this multi thread app but it is really hard to follow.
My question are:
1)  can you tell how RunParser() methods gets call other than by StartParsing() Method?
From the debugger, it looks like RunPRaser() gets called several times.
I might be wrong.


Initially StartParsing() calls the RunParser() and it parse the given URL.
Is that only time it gets called?
So my given URL is a folder, say, http://mysite.com/webfolderName/
Would the RunParser() run once then calls the ParseFolder() which recursively parse the entire subfolders.

2)Also how does ThreadRunFunction() get called?
If I set the ThreadCount=10.
Is there when and where gets call that ThreadRunFunction()?
So one thread is available, it calls the ThreadRunFunction() which in turns calls the Dequeue method.
then next thread call ThreadRunFunction() which in turns calls the Dequeue method and so on.

I was a little confused how this threadRunFunction() gets called multiple times..

3) ThreadRunFunction calls Dequeue() then starts ParseURI() which parses the given file and put found links into the Enqueue(). That's end of the life of the ThreadRunFunction right?
Then how does each URI in queue gets parsed in order to find the subfolders?
What calls RunParser()/ParseFolder()?

Sorry if these makes sense.
private int nThreadCount;
    private int ThreadCount
    {
        get { return nThreadCount; }
        set
        {           
                for (int nIndex = 0; nIndex < value; nIndex++)
                {
                    // check if thread not created or not suspended
                    if (threadsRun[nIndex] == null || threadsRun[nIndex].ThreadState != ThreadState.Suspended)
                    {                        
                        threadsRun[nIndex] = new Thread(new ThreadStart(ThreadRunFunction));                      
                        threadsRun[nIndex].Start();                         
                    }
                    // check if the thread is suspended
                    else if (threadsRun[nIndex].ThreadState == ThreadState.Suspended)
                    {        
                        threadsRun[nIndex].Resume();
                    }
                }              
                nThreadCount = value;
            }           
        }
    }
    void StartParsing()
    {
        ThreadCount = 10;
        /* parsing starts and thread starts */
        if (threadParse == null || threadParse.ThreadState != ThreadState.Suspended)
        {        
            threadParse = new Thread(new ThreadStart(RunParser));         
        }
    }
    void RunParser()
    {

        ThreadsRunning = true;
        try
        {
            string baseURI = this.comboBoxWeb.Text.Trim();
            if (Directory.Exists(baseURI) == true)
                ParseFolder(baseURI, 0);
            else
            {
                if (File.Exists(baseURI) == false)
                {
                    Normalize(ref baseURI);
                    this.comboBoxWeb.Text = baseURI;
                }
                downloadBaseURI = new MyUri(baseURI);
                this.EnqueueUri(downloadBaseURI, false);
            }
        }
    
    }
    void ThreadRunFunction()
    {
        MyWebRequest request = null;
        while (ThreadsRunning && int.Parse(Thread.CurrentThread.Name) < this.ThreadCount)
        {
            
            MyUri uri = DequeueUri();        

            if (uri != null)
            {
                if (SleepConnectTime > 0)
                    Thread.Sleep(SleepConnectTime * 1000);
                ParseUri(uri, ref request);         

            }
            else
                Thread.Sleep(SleepFetchTime * 1000);
        }
       
    }
void ParseUri(MyUri uri, ref MyWebRequest request)
		{
			string strStatus = "";
			
			

			try
			{				
				request = MyWebRequest.Create(uri, request, KeepAlive);				
				request.Timeout = RequestTimeout*1000;				
				MyWebResponse response = request.GetResponse();				
				strStatus += request.Header+response.Header;	
				
				
                bool bParse = true;           							

				if(ThreadsRunning == true && bParse == true && uri.Depth < WebDepth)
				{
					strStatus += "\r\nParsing page ...\r\n";
					
					string strRef = @"(href|HREF|src|SRC)[ ]*=[ ]*[""'][^""'#>]+[""']";
					MatchCollection matches = new Regex(strRef).Matches(strResponse);
					strStatus += "Found: "+matches.Count+" ref(s)\r\n";
					URLCount += matches.Count;
					foreach(Match match in matches)
					{						
							MyUri newUri = new MyUri(strRef);							
							ithis.EnqueueUri(newUri, true);
							
						}
						
					}
				}
void ParseFolder(string folderName, int nDepth)
		{
			DirectoryInfo dirInfo = new DirectoryInfo(folderName);
			FileInfo[] fileInfo = dirInfo.GetFiles();
            //FileInfo[] fileInfo = dirInfo.GetFiles("*.txt");
			foreach(FileInfo f in fileInfo)
			{
				if(ThreadsRunning == false)
					break;
				MyUri uri = new MyUri(f.FullName);
				uri.Depth = nDepth;
				this.EnqueueUri(uri, true);
			}
			
			DirectoryInfo[] dirsInfo = dirInfo.GetDirectories();
			foreach(DirectoryInfo d in dirsInfo)
			{
				if(ThreadsRunning == false)
					break;
				ParseFolder(d.FullName, nDepth+1);
			}
		}
				LogUri(uri.AbsoluteUri, strStatus);
			}
			catch(Exception e)
			{
				LogError(uri.AbsoluteUri, strStatus+e.Message);
				request = null;
			}
			finally
			{
				EraseItem(itemLog);
			}
		}
}

Open in new window

0
Comment
Question by:dkim18
  • 6
  • 4
11 Comments
 

Author Comment

by:dkim18
ID: 36893025

I made correction.
In the ParseURI(), it gets added if new URI is not in the Queue already.

So the RunParser() gets called only once and extracts all the files and subfolders from there,
Is  that correct?


ParseURI()
{
....

foreach(Match match in matches)
{						
MyUri newUri = new MyUri(strRef);							
if(this.EnqueueUri(newUri, true) == true)
 
							
}

						
					}

Open in new window

0
 

Author Comment

by:dkim18
ID: 36893039
Enqueue() checks if uri is in the queue or not.
Sorry I forgot to mention that.
0
 
LVL 8

Assisted Solution

by:SunnyDark
SunnyDark earned 100 total points
ID: 36896545
The line ThreadCount = 10; (line 27) creates 10 new threads , that means calls ThreadRunFunction 10 times ,
RunParser only called once from StartParsing

ThreadRunFunction will live while ThreadsRunning is true...

This however seems as a mistake since by the looks of it should simply wait for the queue to empty...
 
0
 
LVL 8

Accepted Solution

by:
Volox earned 400 total points
ID: 36898032
Not to offend the author of the code (sort of assuming it wasn't you based on the questions you are asking) but this is a poor implementation.

To answer your questions...
1) Based on the code you have here, the RunParser method should only get called once; however I don't think you have all the code in your example since there is no containing class statement and there are several variable that appear to be class level members for which the definition isn't provided.  If all this is defined within a class and the varaibles are not static variables, then you could be seeing RunParser run multiple times if there are multiple instances of the class instanitated and called.

2) When ThreadCount is set to a number, the set property definition creates that number of threads and launches them to run the ThreadRunFunction.  This is part of why I say this is a poor implementation... It is generally bad practice to perform significant operations within the set property of a class and that is especially true when the operation involves launching threads.  The "right" way to do this would have been to make the set property private, make it a simple assignment to the nThreadCount variable, and have a method like LaunchThreads(int) that takes the number of threads to be launched, does that work, and assigns the value to nThreadCount.

The other reason that the implementation in the ThreadCount set property is bad is that it doesn't provide for thread cleanup if / when you reduce the thread count.  So if you call ThreadCount = 10 and it launches 10 threads and then call ThreadCount = 2, it still leaves 8 threads out there running and depends on the logic on line 56 to spin down the threads.  (And I'm failing to see where the threadsRun[] array gets instantiated and dimensioned so a ThreadCount = x statement that exceeds the size of the array would also spell trouble...)

The logic on line 56 is dangerous because there are no garuantees that thread names will be numeric nor that they will be sequential if they are numeric.  Instead, there should be an array that matches up with the threadsRun[] and contains wait events (such as ManualResetEvent) or boolean flags that are used to terminate the thread gracefully.

3) As SunnyDark points out, the ThreadRunFunction will keep repeating until the ThreadsRunning goes false - or - the ThreadCount variable is decreased such that it is less than the integer value represented by the thread name (already commented on the problem with that).

If there isn't anything left in the queue to process, the method simply sleeps and then loops back around to try again.  So if neither of those two triggers stop the looping, that method will essentially run forever.

Extra)  One other thing I'd point out... You have problem with thread safety in this code... When StartParsing() is called it sets the ThreadCount property first and THEN creates a thread for RunParser.  The problem here is that ThreadCount launchs the threads which run the ThreadRunFunction which checks the ThreadsRunning property.  The ThreadsRunning property is set inside of the RunParser method.  So any of the threads that ThreadCount[set] launches that get to line 56 before the RunParser method gets launched and performs line 37 will end up exiting (and I don't see anything that would re-launch them).

Likewise, within the ThreadCount[set] operation the threads are actually launched and started before the new value is stored in nThreadCount so again, if the threads get to line 56 before the nThreadCount value is stored, they will exit out.

On a system in 'normal' state, this "out of order" problem probably won't manifest because it will take longer for the runtime to allocate the resources for the threads and get them actually launched than it will take the methods to get through to the point where they set the variables.  However, when you get into a position where the system is boged down and doing a lot of thread context switching, this could really cause some incorrect / inconsistent behavior for you.
0
 

Author Comment

by:dkim18
ID: 36904147
I don't have much experience with the threading and I have been trying to understand what this app is doing but I had a hard time following it.
It worked fine in the beginning but now it is behaving inconsistenly.

Let me try to understand what Volox said and come back to this question.

1) Can I set this to single thread by setting the threadcount=1?
I tried it and the app didn't run.
it just went for a StartParsing but it never ran the ThreadRunFunction().
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 8

Expert Comment

by:Volox
ID: 36904748
Setting ThreadCount to 1 will essentially take you down to 2 threads.  It may very well have failed to run the ThreadRunFunction for the exact reasons I state in the "Extra" section of my last response.
0
 
LVL 8

Expert Comment

by:Volox
ID: 36904833
Here's a quick fix to the first couple of methods that should at least allow you to run it with the number of threads set to 1 (which will mean you will have two running threads).  It doesn't fix all the problems but it provides a quick hack fix to the order of operations and some of the thread safety issues.

I will apologize in advance for any syntax errors as I don't have my dev environment handy so I just did these edits real quick in notepad...
private int nThreadCount;
    private object launchLock = new object();
    private bool stopThreads = false;

    private int ThreadCount
    {
        get { return nThreadCount; }
    }
    private void LaunchThreads(int newThreadCount)
    {
        lock (launchLock)
        {       
                stopThreads = false;
                newThreadCount = value;
                for (int nIndex = 0; nIndex < newThreadCount; nIndex++)
                {
                    // check if thread not created or not suspended
                    if (threadsRun[nIndex] == null || threadsRun[nIndex].ThreadState != ThreadState.Suspended)
                    {                        
                        threadsRun[nIndex] = new Thread(new ThreadStart(ThreadRunFunction));                      
                        threadsRun[nIndex].Start();                         
                    }
                    // check if the thread is suspended
                    else if (threadsRun[nIndex].ThreadState == ThreadState.Suspended)
                    {        
                        threadsRun[nIndex].Resume();
                    }
                }
            }           
        }
    }
    void StartParsing()
    {
        LaunchThreads(10);
        /* parsing starts and thread starts */
        if (threadParse == null || threadParse.ThreadState != ThreadState.Suspended)
        {        
            threadParse = new Thread(new ThreadStart(RunParser));         
        }
    }
    void RunParser()
    {
        try
        {
            string baseURI = this.comboBoxWeb.Text.Trim();
            if (Directory.Exists(baseURI) == true)
                ParseFolder(baseURI, 0);
            else
            {
                if (File.Exists(baseURI) == false)
                {
                    Normalize(ref baseURI);
                    this.comboBoxWeb.Text = baseURI;
                }
                downloadBaseURI = new MyUri(baseURI);
                this.EnqueueUri(downloadBaseURI, false);
            }
        }
    
    }
    void ThreadRunFunction()
    {
        MyWebRequest request = null;
        while (!stopThreads && int.Parse(Thread.CurrentThread.Name) < this.ThreadCount)
        {
            
            MyUri uri = DequeueUri();        

            if (uri != null)
            {
                if (SleepConnectTime > 0)
                    Thread.Sleep(SleepConnectTime * 1000);
                ParseUri(uri, ref request);         

            }
            else
                Thread.Sleep(SleepFetchTime * 1000);
        }
    }

Open in new window

0
 

Author Comment

by:dkim18
ID: 36905413
So when & where does this stopThreads get changed?

Also my original code used ThreadsRunning = true in RunParser() method.


stopTheads has been used in the ThreadRunFuntion()
While (!stopThreads &&....

I am not seeing the difference.
Can you explain what it is doing please?
Thanks.
0
 

Author Comment

by:dkim18
ID: 36905624
Hm.
It sets the threadcount but it is not going any where.
it hangs.



into: Stepping over non-user code 'System.Windows.Forms.NativeWindow.DebuggableCallback'
Step into: Stepping over non-user code 'System.Windows.Forms.Control.ReflectMessageInternal'
Step into: Stepping over non-user code 'System.Windows.Forms.Control.WmCommand'
Step into: Stepping over non-user code 'System.Windows.Forms.Control.WndProc'
Step into: Stepping over non-user code 'System.Windows.Forms.NativeWindow.DebuggableCallback'
Step into: Stepping over non-user code 'System.Windows.Forms.NativeWindow.DefWndProc'
Step into: Stepping over non-user code 'System.Windows.Forms.Control.WmMouseUp'
Step into: Stepping over non-user code 'System.Windows.Forms.Control.WndProc'
Step into: Stepping over non-user code 'System.Windows.Forms.ToolBar.WndProc'
Step into: Stepping over non-user code 'System.Windows.Forms.NativeWindow.DebuggableCallback'
Step into: Stepping over non-user code 'System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop'
0
 

Author Comment

by:dkim18
ID: 36909792
Did you mean to leave out the setter?
 private int ThreadCount
    {
        get { return nThreadCount; }
    }


So how this ThreatCount gets Set then?
There is a ThreadCount in the ThreadRunFunction()
while (!stopThreads && int.Parse(Thread.CurrentThread.Name) < this.ThreadCount)
0
 
LVL 8

Expert Comment

by:Volox
ID: 36912311
Yes.  I removed the setter and put in the LaunchThreads method.  Although I do see one place where I messed up...  line 14 should have been:
nThreadCount = newThreadCount;

The stopThreads replaces the ThreadsRunning because this inversion causes less issues.  If you use ThreadsRunning, the logical thing to do is set it to true after the thread is running, however that is precisely what causes the out of order threading problems.  By using stopThreads, the value can be false from the get go and set to true when you want to bring the threads to a stop.

I didn't see where in your code the ThreadsRunning was ever being set to false, but you'd want to replace that with the stopThreads being set to true.  If you were using ThreadsRunning as an indicator of whether the thread had launched you can put it back in if you want.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This document covers how to connect to SQL Server and browse its contents.  It is meant for those new to Visual Studio and/or working with Microsoft SQL Server.  It is not a guide to building SQL Server database connections in your code.  This is mo…
Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now