asked on

Multi-File Download with C# WebClient...

Can the WebClient class successfully, in a multi-threaded application, download multiple files?

I have a multi-threaded application that allows the user to download up to 5 files at a time.

To accomplish this I run the download in 2 phases: Phase 1 calls DownloadFileAsync(uri,downloadpath,object) 5 times (I use AutoResetEvent objects for thread synchronization). Phase 1 runs with no problems.

Phase 2 is the handler for the AsyncCompletedEvent and where the problem occurs. When the program enters the download complete handle I am finding the file information contained in the response header for the sender object (ResponseHeaders["Content-Disposition"]), is different than the file in the AsyncCompletedEventHandler.UserState!

What I expected was completed events to be fired for each download request, but it looks like the completed event is using the first or the last request as the sender and eventually the sender value changes to null. As noted earlier, the sender never matches the event arg UserState.

If I download 1 file at a time, no problems - the response header content matches the UserState object.

Any ideas on what I am doing wrong would be greatly appreciated.

Thanks in advance.

Rick

Éric Moreau

chech http://neatupload.codeplex.com/

sadlermd

ASKER

emoreau:

just took a cursory glance at the link - does it work with downloading files?

Éric Moreau

You may be right. I used it to upload but I never tried to download! Sorry.

Sudhakar Pulivarthi

Hi,

public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. So you need to make sure on call to .Download() like this methods to synchronize.
http://msdn.microsoft.com/en-us/library/system.net.webclient(v=VS.80).aspx

sadlermd

ASKER

Hello Sudhakar:

I am doing that. My code is similar to this snippet taken from the MS website:

// this method will be called up to 5 times from another thread
public static void DownLoadFileInBackground2 (string address)
{
WebClient client = new WebClient ();
Uri uri = new Uri(address);

// Specify that the DownloadFileCallback method gets called
// when the download completes.
client.DownloadFileCompleted += new AsyncCompletedEventHandler (DownloadFileCallback2);
// Specify a progress notification handler.
client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressCallback);

client.DownloadFileAsync (uri, "serverdata.txt", "myDataObject");
}

The question is can the WebClient object handle scenarios where multiple requests are being made from threads. In this scenario I am finding that when the download complete event is fired, the sender and the user state don't match...

Thanks!

Russell_Venable

You need to incorporate "asyccallback" and "lock" your threading so they don't cross thread. Every new download should be called by asynccallback.

sadlermd

ASKER

Russell:

I will research your suggestion - however, just off the top of my head, I think the net of what you are saying would be the equivalent of downloading one file at a time.

Correct me where I am wrong: If a download should only occur on a callback, then that suggests a download has finished, or am I missing your point?

Thanks for your help,

Rick

Russell_Venable

Correct, I was thinking about raw sockets using IAsyncResult and Asynccallback. I did however find a good example of what your are trying to do.

Have you tried looking at a project like this one posted on Codeproject.
WebClientAsyncDownloader

sadlermd

ASKER

Russell:

Pretty interesting - I will check it out!

Rick

Todd Gerbert

The WebClient doesn't support concurrent operations; you need one WebClient for each file download.

Think of one WebClient object as kinda-sorta equal to one tab in Internet Explorer. If you type an address and hit enter IE will start downloading the page - if you immediately type a new address and hit enter again the first page is aborted and IE downloads the second address you typed.

If it's working for you now it's likely because some of your downloads are happening fast enough that they're not really running concurrently.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.ComponentModel;

namespace ConsoleApplication1
{
	class Program
	{
		static void Main(string[] args)
		{
			InitiateDownload("http://intranet/test1.pdf", @"C:\test1.pdf", FileDownloadCompleted, "test1.pdf");
			InitiateDownload("http://intranet/test2.pdf", @"C:\test2.pdf", FileDownloadCompleted, "test2.pdf");

			Console.ReadKey();
		}

		static void InitiateDownload(string RemoteAddress, string LocalFile, AsyncCompletedEventHandler CompletedCallback, object userToken)
		{
			WebClient wc = new WebClient();
			wc.DownloadFileCompleted += CompletedCallback;
			wc.DownloadFileAsync(new Uri(RemoteAddress), LocalFile, userToken);
		}

		static void FileDownloadCompleted(object sender, AsyncCompletedEventArgs e)
		{
			if (e.Error != null)
				Console.WriteLine("Error downloading {0}: {1}", e.UserState, e.Error);
			else if (e.Cancelled)
				Console.WriteLine("Download of {0} cancelled.", e.UserState);
			else
				Console.WriteLine("File with userToke {0} completed.", e.UserState);
		}
	}
}

Open in new window

sadlermd

ASKER

tgerbert:

You are on the right track to replicating my problem.

My code is the same as your snippet except I am making 5 download calls in succession. For me, things breakdown in the FileDownloadCompleted() method.

So if I place in the user token the filenames ("test1.pdf", "test2.pdf", etc.), what I expect to see in the download complete event is sender=test1.pdf and userstate=test1.pdf, etc...

Instead, when downloading 3,4 or more files at a time, eventually i get results like "sender=test1.pdf" and "userstate=test5.pdf"! To make matters more confusing, sender will often be null!

Thoughts?

Rick

sadlermd

ASKER

Let me add that my application will often need to download thousands of files at the time.

My application works *perfectly* as long as I download 1 file at a time :-(

Rick

Todd Gerbert

You're creating a new instance of WebClient for each file download? Are you creating thousands of WebClients at once, or a few at a time? Are you Dispose()'ing every WebClient when you're done with it?

sadlermd

ASKER

I'm creating a max of 5 webclient objects and i instantiate each instance in a using statement - I don't explicitly call dispose.

sadlermd

ASKER

The following is happening consistently:

1. Downloading a total of 5 files - A.pdf, B.pdf, C.pdf, E.pdf, F.pdf
2. Download 3 at a time. (Create a thread event to call the download process and stop when count >= 3)
3. Files A,B,C are downloaded by DownloadFileAsync()
4. DownloadComplete receives 3 events - all are for C
5. Files E,F are downloaded by DownloadFileAsync()
6. DownloadComplete receives 2 events - all are for F
7. A,B,E are the only files that actually downloaded to my machine.

I would be interested in hearing if this is happening to anyone else...

ASKER CERTIFIED SOLUTION

Todd Gerbert

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Russell_Venable

I wrote something simular yesterday but I didn't post it cause I could not get it to work. Not sure why it is not working either.

sadlermd

ASKER

My personal feeling is one of 2 things is happening:

1. I have a bug in my program - I have several other things running in other threads.

2. There is something about the web server response mechanics that I am not aware of.

Either way, this is beginning to feel like a run down a rabbit hole.

I will put together a piece of code that closely replicates my application.

Rick

sadlermd

ASKER

tgerbert:

Your code snippet is the same logic I employ except for 4 subtle differences:

1. I fire off the InitiateDownload() calls from inside a BackgroundWorker thread
2. I create the WebClient instance in a using statement.
3. I synchronize the calls to InitiateDownload with AutoResetEvent instead of a custom static object.
4. I don't call Dispose on the webclient.

Of these differences number 3 and 4 are the most intriguing. Anyway, I will apply those differences in my code and see what happens!

Thanks again,

Rick

Todd Gerbert

Maybe it's the "using" statement. Remember that at the end of a "using" block Dispose() is called on the object declared in the using statement.

using (WebClient wc = new WebClient())
{
  wc.DownloadFileCompleted += downloadEventHandler;
  wc.DownloadFileAsync(new Uri(someUrl));
} // wc.Dispose() is called here by compiler, but DownloadFileAsync may not have completed yet

Open in new window

sadlermd

ASKER

Thanks for the help!

sadlermd

ASKER

tgerbert:

For clarification, I need to elaborate on the fix. For anyone who needs it, your code snippet does work. But the problem with my program was a bug in my code.

The content I was passing in the UserToken was a reference type; I was reusing it instead of creating a new instance of the object - that's why the UserState was always the value of the last download request.

Rick

Todd Gerbert

Keep in mind my comments regarding the "using" statement with asynchronous method calls - that still applies.