C# Multi-threading problem

Hi,

I have a rqeuirement to implement multi-threading within an application which can execute multiple web queries simultaeneously.  I have written the code byut am getting some strange results that I would like some help/advice on.  

I want to crteate 'n' threads, each of which will call the same function to execute a web query and return their results.  I have added tracing to the thread function but this still gives odd results with regards to timings.  When multi-threaded I would expect performance to be improved by about a factor of the number of threads used however this seems not to be the case.

The code to create the threads is attached. My thread function is as follows:

private void QueryURL(object myObject)
{
  LogFile log = LogFile.Instance;
  DateTime startTime = DateTime.Now;
  EbayResult ebayResult = (EbayResult)myObject;

  try
  {
    ebayResult.ResultDocument.Load(ebayResult.Url);
  }

  catch (Exception ex)
  {
    log.Write("Thread terminated with error, text is " + ex.Message, false);
  }

  // Mark the thread as complete and return;
  DateTime endTime = DateTime.Now;
  TimeSpan elapsed = endTime - startTime;
  System.Diagnostics.Debug.WriteLine("Thread ID " + ebayResult.id + "Started at " + startTime.ToString() + " Completed at " + endTime.ToString() + " - Duration was " + elapsed.Milliseconds.ToString() + " milliseconds");
  ebayResult.ThreadComplete = true;
  return;
}

What is odd is that my trace shows output such as :

Thread ID 1 Started at 21/04/2010 18:18:32 Completed at 21/04/2010 18:18:40 - Duration was 390 milliseconds - which is very odd as the start and end times do not match the elapsed time reported.

Any ideas what I have done wrong???
DateTime endTime = DateTime.Now;
TimeSpan elapsed = endTime - startTime;
log.Write("Starting threads creation after " + elapsed.Milliseconds.ToString() + " milliseconds [" + endTime.ToString() + "]", false);

int thread = 1;
foreach (string url in urlList)
{
	EbayResult ebayResult = new EbayResult(url);
	ebayResult.id = thread.ToString();
	thread++;
	_listResults.Add(ebayResult);
	ThreadPool.QueueUserWorkItem(new WaitCallback(QueryURL) ,ebayResult);
}

endTime = DateTime.Now;
elapsed = endTime - startTime;
log.Write("All threads created after " + elapsed.Milliseconds.ToString() + " milliseconds [" + endTime.ToString() + "]", false);

// Wait for all threads to complete
bool allThreadsComplete;

do
{
  // Sleep for 1/10 second 
  Thread.Sleep(10);
  
  // Check for ANY active threads and loop back if any
  allThreadsComplete = true; 
  foreach (EbayResult ebayResult in _listResults)
  {
    if (!ebayResult.ThreadComplete)
    {
      allThreadsComplete = false;
      break;
    }
  }
}
while (!allThreadsComplete);

// All threads are now complete - concatenate the results from each thread
endTime = DateTime.Now;
elapsed = endTime - startTime;
log.Write("All threads complete after " + elapsed.Milliseconds.ToString() + " milliseconds [" + endTime.ToString() + "]", false);

Open in new window

ChrisMDrewAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lazyberezovskyCommented:
Consider using WaitHandle.WaitAll method for waiting all threads.
http://msdn.microsoft.com/en-us/library/z6w25xa6.aspx

For avoiding infinite waiting use WaitHandle.WaitAll(WaitHandle[], TimeSpan) method.
0
ChrisMDrewAuthor Commented:
I'd seen that - however I don't know how many threads I will be creating and it seems that this is required as the WaitAll requires an array of handles - I wonder if I could allocate say an array of 10 handles but only set some???  

Any ideas why my tracing is ghetting weird timings?  I'm pretty sure that the variables in the thread fuinction are allocated for each thread so there should not be any interaction there???
0
lazyberezovskyCommented:
For measuring elapsed time it's better to use Stopwatch:
Stopwatch watch = new Stopwatch();
watch.Start();
//use watch.Elapsed and watch.ElapsedMilliseconds
watch.Stop();
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

lazyberezovskyCommented:
I recommend you to use int as EbayResult Id.
And refactored code:
List<WaitHandle> waitHandles = new List<WaitHandle>(); // Also WaitHandle[] array could be used as you know urlList items count.
int index = 0;

foreach (string url in urlList)
{
    EbayResult ebayResult = new EbayResult(url);
    ebayResult.id = ++index;
    ManualResetEvent waitHandle = new ManualResetEvent(false);
    waitHandles.Add(waitHandle);                           
    ThreadPool.QueueUserWorkItem(delegate(object obj) { QueryURL(ebayResult, waitHandle); }, null);
}

// Waiting for all threads to complete their work
WaitHandle.WaitAll(waitHandles.ToArray()); // You can specify timeout here


// Mind that thread proc signature changed:
private void QueryURL(EbayResult result, ManualResetEvent waitHandle)
{
    // Do work here
    waitHandle.Set();
}

Open in new window

0
ChrisMDrewAuthor Commented:
Thanks for the suggestions.  I did try using 'WaitHandler.WaitAll but this failed with the error 'WaitAll for multiple handles on a STA thread is not supported'.  As i cannot be 100% certain whether how this function will be called I don't think I can use the WaitAll - I believe that I would need to ensure that the caller used the [MTAThread] attribute.

BTW - the primary issue with the odd timings appears to be an error on my part - I should log TOTALMILLISECONDS and not MILLISECONDS.  When I change the logging code at least the timings are correct suggesting that the mult-threading is working correctly after all.

I would be very interested in knowing if there is any way I can use the WaitAll function however within this class without having to modify the calling program.
0
lazyberezovskyCommented:
Simplest work-around is to call url's processing on MTA thread (if you don't want to change [STA] attribute of Main method:
// Caller
Thread thread = new Thread(delegate() { Query(urlList); });
thread.ApartmentState = ApartmentState.MTA;
thread.Start();           

// MTA method
private void Query(IEnumerable<string> urlList)
{
    // Use my previous code here
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ChrisMDrewAuthor Commented:
Thanks I think I have everything I need now!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C#

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.