Multi-File Download with C# WebClient...

Can the WebClient class successfully, in a multi-threaded application, download multiple files?

I have a multi-threaded application that allows the user to download up to 5 files at a time.

To accomplish this I run the download in 2 phases: Phase 1 calls DownloadFileAsync(uri,downloadpath,object) 5 times (I use AutoResetEvent objects for thread synchronization). Phase 1 runs with no problems.

Phase 2 is the handler for the AsyncCompletedEvent and where the problem occurs. When the program enters the download complete handle I am finding the file information contained in the response header for the sender object (ResponseHeaders["Content-Disposition"]), is different than the file in the AsyncCompletedEventHandler.UserState!

What I expected was completed events to be fired for each download request, but it looks like the completed event is using the first or the last request as the sender and eventually the sender value changes to null. As noted earlier, the sender never matches the event arg UserState.

If I download 1 file at a time, no problems - the response header content matches the UserState object.

Any ideas on what I am doing wrong would be greatly appreciated.

Thanks in advance.

Rick
sadlermdAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Éric MoreauSenior .Net ConsultantCommented:
0
sadlermdAuthor Commented:
emoreau:

just took a cursory glance at the link - does it work with downloading files?
0
Éric MoreauSenior .Net ConsultantCommented:
You may be right. I used it to upload but I never tried to download! Sorry.
0
Why Diversity in Tech Matters

Kesha Williams, certified professional and software developer, explores the imbalance of diversity in the world of technology -- especially when it comes to hiring women. She showcases ways she's making a difference through the Colors of STEM program.

Sudhakar PulivarthiProject Lead - EngineeringCommented:
Hi,

public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. So you need to make sure on call to .Download() like this methods to synchronize.
http://msdn.microsoft.com/en-us/library/system.net.webclient(v=VS.80).aspx
0
sadlermdAuthor Commented:
Hello Sudhakar:

I am doing that. My code is similar to this snippet taken from the MS website:

// this method will be called up to 5 times from another thread
public static void DownLoadFileInBackground2 (string address)
{
    WebClient client = new WebClient ();
    Uri uri = new Uri(address);

    // Specify that the DownloadFileCallback method gets called
    // when the download completes.
    client.DownloadFileCompleted += new AsyncCompletedEventHandler (DownloadFileCallback2);
    // Specify a progress notification handler.
    client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressCallback);

    client.DownloadFileAsync (uri, "serverdata.txt", "myDataObject");
}

The question is can the WebClient object handle scenarios where multiple requests are being made from threads. In this scenario I am finding that when the download complete event is fired, the sender and the user state don't match...

Thanks!

0
Russell_VenableCommented:
You need to incorporate "asyccallback" and "lock" your threading  so they don't cross thread. Every new download should be called by asynccallback.
0
sadlermdAuthor Commented:
Russell:

I will research your suggestion - however, just off the top of my head, I think the net of what you are saying would be the equivalent of downloading one file at a time.

Correct me where I am wrong: If a download should only occur on a callback, then that suggests a download has finished, or am I missing your point?

Thanks for your help,

Rick
0
Russell_VenableCommented:
Correct, I was thinking about raw sockets using IAsyncResult and Asynccallback. I did however find a good example of what your are trying to do.

Have you tried looking at a project like this one posted on Codeproject.
WebClientAsyncDownloader
0
sadlermdAuthor Commented:
Russell:

Pretty interesting - I will check it out!

Rick
0
Todd GerbertIT ConsultantCommented:
The WebClient doesn't support concurrent operations; you need one WebClient for each file download.

Think of one WebClient object as kinda-sorta equal to one tab in Internet Explorer.  If you type an address and hit enter IE will start downloading the page - if you immediately type a new address and hit enter again the first page is aborted and IE downloads the second address you typed.

If it's working for you now it's likely because some of your downloads are happening fast enough that they're not really running concurrently.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.ComponentModel;

namespace ConsoleApplication1
{
	class Program
	{
		static void Main(string[] args)
		{
			InitiateDownload("http://intranet/test1.pdf", @"C:\test1.pdf", FileDownloadCompleted, "test1.pdf");
			InitiateDownload("http://intranet/test2.pdf", @"C:\test2.pdf", FileDownloadCompleted, "test2.pdf");

			Console.ReadKey();
		}

		static void InitiateDownload(string RemoteAddress, string LocalFile, AsyncCompletedEventHandler CompletedCallback, object userToken)
		{
			WebClient wc = new WebClient();
			wc.DownloadFileCompleted += CompletedCallback;
			wc.DownloadFileAsync(new Uri(RemoteAddress), LocalFile, userToken);
		}

		static void FileDownloadCompleted(object sender, AsyncCompletedEventArgs e)
		{
			if (e.Error != null)
				Console.WriteLine("Error downloading {0}: {1}", e.UserState, e.Error);
			else if (e.Cancelled)
				Console.WriteLine("Download of {0} cancelled.", e.UserState);
			else
				Console.WriteLine("File with userToke {0} completed.", e.UserState);
		}
	}
}

Open in new window

0
sadlermdAuthor Commented:
tgerbert:

You are on the right track to replicating my problem.

My code is the same as your snippet except I am making 5 download calls in succession. For me, things breakdown in the FileDownloadCompleted() method.

So if I place in the user token the filenames ("test1.pdf", "test2.pdf", etc.), what I expect to see in the download complete event is sender=test1.pdf and userstate=test1.pdf, etc...

Instead, when downloading 3,4 or more files at a time, eventually i get results like "sender=test1.pdf" and "userstate=test5.pdf"! To make matters more confusing, sender will often be null!

Thoughts?

Rick



0
sadlermdAuthor Commented:
Let me add that my application will often need to download thousands of files at the time.

My application works *perfectly* as long as I download 1 file at a time :-(


Rick
0
Todd GerbertIT ConsultantCommented:
You're creating a new instance of WebClient for each file download? Are you creating thousands of WebClients at once, or a few at a time? Are you Dispose()'ing every WebClient when you're done with it?
0
sadlermdAuthor Commented:
I'm creating a max of 5 webclient objects and i instantiate each instance in a using statement - I don't explicitly call dispose.



0
sadlermdAuthor Commented:
The following is happening consistently:

1. Downloading a total of 5 files - A.pdf, B.pdf, C.pdf, E.pdf, F.pdf
2. Download 3 at a time. (Create a thread event to call the download process and stop when count >= 3)
3. Files A,B,C are downloaded by DownloadFileAsync()
4. DownloadComplete receives 3 events - all are for C
5. Files E,F are downloaded by DownloadFileAsync()
6. DownloadComplete receives 2 events - all are for F
7. A,B,E are the only files that actually downloaded to my machine.

I would be interested in hearing if this is happening to anyone else...
0
Todd GerbertIT ConsultantCommented:
I think you're gonna need to post your code - I'm having a hard thinking of what might be the cause of your issue without seeing it.

The code snippet below works, but don't use it as is (it needs some thread sync/error checking/etc).  It produces this output (note that the files complete in a different order than they were specified so I know async worked well):
File with userToke http://intranet/i.pdf completed.
File with userToke http://intranet/e.pdf completed.
File with userToke http://intranet/g.pdf completed.
File with userToke http://intranet/a.pdf completed.
File with userToke http://intranet/f.pdf completed.
File with userToke http://intranet/c.pdf completed.
File with userToke http://intranet/j.pdf completed.
File with userToke http://intranet/h.pdf completed.
File with userToke http://intranet/l.pdf completed.
File with userToke http://intranet/k.pdf completed.
File with userToke http://intranet/b.pdf completed.
File with userToke http://intranet/d.pdf completed.

Open in new window


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.ComponentModel;

namespace ConsoleApplication1
{
    class Program
    {
        static Dictionary<string, string> fileList = new Dictionary<string, string>();

        static void Main(string[] args)
        {
            fileList.Add("http://intranet/a.pdf", @"C:\test\a.pdf");
            fileList.Add("http://intranet/b.pdf", @"C:\test\b.pdf");
            fileList.Add("http://intranet/c.pdf", @"C:\test\c.pdf");
            fileList.Add("http://intranet/d.pdf", @"C:\test\d.pdf");
            fileList.Add("http://intranet/e.pdf", @"C:\test\e.pdf");
            fileList.Add("http://intranet/f.pdf", @"C:\test\f.pdf");
            fileList.Add("http://intranet/g.pdf", @"C:\test\g.pdf");
            fileList.Add("http://intranet/h.pdf", @"C:\test\h.pdf");
            fileList.Add("http://intranet/i.pdf", @"C:\test\i.pdf");
            fileList.Add("http://intranet/j.pdf", @"C:\test\j.pdf");
            fileList.Add("http://intranet/k.pdf", @"C:\test\k.pdf");
            fileList.Add("http://intranet/l.pdf", @"C:\test\l.pdf");

            // Start the first 5 - which should keep no more than 5 running at any given time
            for (int i = 0; i < 5; i++)
            {
                KeyValuePair<string, string> file = fileList.ElementAt(i);
                InitiateDownload(file.Key, file.Value, FileDownloadCompleted, file.Key);
            }

            Console.ReadKey();
        }

        static void InitiateDownload(string RemoteAddress, string LocalFile, AsyncCompletedEventHandler CompletedCallback, object userToken)
        {
            // Remove file from queue
            fileList.Remove(userToken.ToString());

            WebClient wc = new WebClient();
            wc.DownloadFileCompleted += CompletedCallback;
            wc.DownloadFileAsync(new Uri(RemoteAddress), LocalFile, userToken);
        }

        static void FileDownloadCompleted(object sender, AsyncCompletedEventArgs e)
        {
            if (e.Error != null)
                Console.WriteLine("Error downloading {0}: {1}", e.UserState, e.Error);
            else if (e.Cancelled)
                Console.WriteLine("Download of {0} cancelled.", e.UserState);
            else
                Console.WriteLine("File with userToke {0} completed.", e.UserState);

            // if there are files left in queue start another
            if (fileList.Count > 0)
                InitiateDownload(fileList.First().Key, fileList.First().Value, FileDownloadCompleted, fileList.First().Key);

            // Dispose of the WebClient that just finished
            ((IDisposable)sender).Dispose();
        }
    }
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Russell_VenableCommented:
I wrote something simular yesterday but I didn't post it cause I could not get it to work. Not sure why it is not working either.
0
sadlermdAuthor Commented:
My personal feeling is one of 2 things is happening:

1. I have a bug in my program - I have several other things running in other threads.

2. There is something about the web server response mechanics that I am not aware of.

Either way, this is beginning to feel like a run down a rabbit hole.

I will put together a piece of code that closely replicates my application.

Rick
0
sadlermdAuthor Commented:
tgerbert:

Your code snippet is the same logic I employ except for 4 subtle differences:

1. I fire off the InitiateDownload() calls from inside a BackgroundWorker thread
2. I create the WebClient instance in a using statement.
3. I synchronize the calls to InitiateDownload with AutoResetEvent instead of a custom static object.
4. I don't call Dispose on the webclient.

Of these differences number 3 and 4 are the most intriguing. Anyway, I will apply those differences in my code and see what happens!

Thanks again,

Rick

0
Todd GerbertIT ConsultantCommented:
Maybe it's the "using" statement.  Remember that at the end of a "using" block Dispose() is called on the object declared in the using statement.

using (WebClient wc = new WebClient())
{
  wc.DownloadFileCompleted += downloadEventHandler;
  wc.DownloadFileAsync(new Uri(someUrl));
} // wc.Dispose() is called here by compiler, but DownloadFileAsync may not have completed yet

Open in new window

0
sadlermdAuthor Commented:
Thanks for the help!
0
sadlermdAuthor Commented:
tgerbert:

For clarification, I need to elaborate on the fix. For anyone who needs it, your code snippet does work. But the problem with my program was a bug in my code.

The content I was passing in the UserToken was a reference type; I was reusing it instead of creating a new instance of the object - that's why the UserState was always the value of the last download request.

Rick
0
Todd GerbertIT ConsultantCommented:
Keep in mind my comments regarding the "using" statement with asynchronous method calls - that still applies.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C#

From novice to tech pro — start learning today.