Solved

Multi-File Download with C# WebClient...

Posted on 2011-02-17
23
5,177 Views
Last Modified: 2012-05-11
Can the WebClient class successfully, in a multi-threaded application, download multiple files?

I have a multi-threaded application that allows the user to download up to 5 files at a time.

To accomplish this I run the download in 2 phases: Phase 1 calls DownloadFileAsync(uri,downloadpath,object) 5 times (I use AutoResetEvent objects for thread synchronization). Phase 1 runs with no problems.

Phase 2 is the handler for the AsyncCompletedEvent and where the problem occurs. When the program enters the download complete handle I am finding the file information contained in the response header for the sender object (ResponseHeaders["Content-Disposition"]), is different than the file in the AsyncCompletedEventHandler.UserState!

What I expected was completed events to be fired for each download request, but it looks like the completed event is using the first or the last request as the sender and eventually the sender value changes to null. As noted earlier, the sender never matches the event arg UserState.

If I download 1 file at a time, no problems - the response header content matches the UserState object.

Any ideas on what I am doing wrong would be greatly appreciated.

Thanks in advance.

Rick
0
Comment
Question by:sadlermd
  • 12
  • 5
  • 3
  • +2
23 Comments
 
LVL 69

Expert Comment

by:Éric Moreau
ID: 34916975
0
 

Author Comment

by:sadlermd
ID: 34918046
emoreau:

just took a cursory glance at the link - does it work with downloading files?
0
 
LVL 69

Expert Comment

by:Éric Moreau
ID: 34918432
You may be right. I used it to upload but I never tried to download! Sorry.
0
 
LVL 11

Expert Comment

by:Sudhakar Pulivarthi
ID: 34923481
Hi,

public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. So you need to make sure on call to .Download() like this methods to synchronize.
http://msdn.microsoft.com/en-us/library/system.net.webclient(v=VS.80).aspx
0
 

Author Comment

by:sadlermd
ID: 34926484
Hello Sudhakar:

I am doing that. My code is similar to this snippet taken from the MS website:

// this method will be called up to 5 times from another thread
public static void DownLoadFileInBackground2 (string address)
{
    WebClient client = new WebClient ();
    Uri uri = new Uri(address);

    // Specify that the DownloadFileCallback method gets called
    // when the download completes.
    client.DownloadFileCompleted += new AsyncCompletedEventHandler (DownloadFileCallback2);
    // Specify a progress notification handler.
    client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressCallback);

    client.DownloadFileAsync (uri, "serverdata.txt", "myDataObject");
}

The question is can the WebClient object handle scenarios where multiple requests are being made from threads. In this scenario I am finding that when the download complete event is fired, the sender and the user state don't match...

Thanks!

0
 
LVL 15

Expert Comment

by:Russell_Venable
ID: 34928923
You need to incorporate "asyccallback" and "lock" your threading  so they don't cross thread. Every new download should be called by asynccallback.
0
 

Author Comment

by:sadlermd
ID: 34929020
Russell:

I will research your suggestion - however, just off the top of my head, I think the net of what you are saying would be the equivalent of downloading one file at a time.

Correct me where I am wrong: If a download should only occur on a callback, then that suggests a download has finished, or am I missing your point?

Thanks for your help,

Rick
0
 
LVL 15

Expert Comment

by:Russell_Venable
ID: 34930925
Correct, I was thinking about raw sockets using IAsyncResult and Asynccallback. I did however find a good example of what your are trying to do.

Have you tried looking at a project like this one posted on Codeproject.
WebClientAsyncDownloader
0
 

Author Comment

by:sadlermd
ID: 34930963
Russell:

Pretty interesting - I will check it out!

Rick
0
 
LVL 33

Expert Comment

by:Todd Gerbert
ID: 34932301
The WebClient doesn't support concurrent operations; you need one WebClient for each file download.

Think of one WebClient object as kinda-sorta equal to one tab in Internet Explorer.  If you type an address and hit enter IE will start downloading the page - if you immediately type a new address and hit enter again the first page is aborted and IE downloads the second address you typed.

If it's working for you now it's likely because some of your downloads are happening fast enough that they're not really running concurrently.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.ComponentModel;

namespace ConsoleApplication1
{
	class Program
	{
		static void Main(string[] args)
		{
			InitiateDownload("http://intranet/test1.pdf", @"C:\test1.pdf", FileDownloadCompleted, "test1.pdf");
			InitiateDownload("http://intranet/test2.pdf", @"C:\test2.pdf", FileDownloadCompleted, "test2.pdf");

			Console.ReadKey();
		}

		static void InitiateDownload(string RemoteAddress, string LocalFile, AsyncCompletedEventHandler CompletedCallback, object userToken)
		{
			WebClient wc = new WebClient();
			wc.DownloadFileCompleted += CompletedCallback;
			wc.DownloadFileAsync(new Uri(RemoteAddress), LocalFile, userToken);
		}

		static void FileDownloadCompleted(object sender, AsyncCompletedEventArgs e)
		{
			if (e.Error != null)
				Console.WriteLine("Error downloading {0}: {1}", e.UserState, e.Error);
			else if (e.Cancelled)
				Console.WriteLine("Download of {0} cancelled.", e.UserState);
			else
				Console.WriteLine("File with userToke {0} completed.", e.UserState);
		}
	}
}

Open in new window

0
 

Author Comment

by:sadlermd
ID: 34933090
tgerbert:

You are on the right track to replicating my problem.

My code is the same as your snippet except I am making 5 download calls in succession. For me, things breakdown in the FileDownloadCompleted() method.

So if I place in the user token the filenames ("test1.pdf", "test2.pdf", etc.), what I expect to see in the download complete event is sender=test1.pdf and userstate=test1.pdf, etc...

Instead, when downloading 3,4 or more files at a time, eventually i get results like "sender=test1.pdf" and "userstate=test5.pdf"! To make matters more confusing, sender will often be null!

Thoughts?

Rick



0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:sadlermd
ID: 34933125
Let me add that my application will often need to download thousands of files at the time.

My application works *perfectly* as long as I download 1 file at a time :-(


Rick
0
 
LVL 33

Expert Comment

by:Todd Gerbert
ID: 34933558
You're creating a new instance of WebClient for each file download? Are you creating thousands of WebClients at once, or a few at a time? Are you Dispose()'ing every WebClient when you're done with it?
0
 

Author Comment

by:sadlermd
ID: 34933822
I'm creating a max of 5 webclient objects and i instantiate each instance in a using statement - I don't explicitly call dispose.



0
 

Author Comment

by:sadlermd
ID: 34934440
The following is happening consistently:

1. Downloading a total of 5 files - A.pdf, B.pdf, C.pdf, E.pdf, F.pdf
2. Download 3 at a time. (Create a thread event to call the download process and stop when count >= 3)
3. Files A,B,C are downloaded by DownloadFileAsync()
4. DownloadComplete receives 3 events - all are for C
5. Files E,F are downloaded by DownloadFileAsync()
6. DownloadComplete receives 2 events - all are for F
7. A,B,E are the only files that actually downloaded to my machine.

I would be interested in hearing if this is happening to anyone else...
0
 
LVL 33

Accepted Solution

by:
Todd Gerbert earned 500 total points
ID: 34934664
I think you're gonna need to post your code - I'm having a hard thinking of what might be the cause of your issue without seeing it.

The code snippet below works, but don't use it as is (it needs some thread sync/error checking/etc).  It produces this output (note that the files complete in a different order than they were specified so I know async worked well):
File with userToke http://intranet/i.pdf completed.
File with userToke http://intranet/e.pdf completed.
File with userToke http://intranet/g.pdf completed.
File with userToke http://intranet/a.pdf completed.
File with userToke http://intranet/f.pdf completed.
File with userToke http://intranet/c.pdf completed.
File with userToke http://intranet/j.pdf completed.
File with userToke http://intranet/h.pdf completed.
File with userToke http://intranet/l.pdf completed.
File with userToke http://intranet/k.pdf completed.
File with userToke http://intranet/b.pdf completed.
File with userToke http://intranet/d.pdf completed.

Open in new window


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.ComponentModel;

namespace ConsoleApplication1
{
    class Program
    {
        static Dictionary<string, string> fileList = new Dictionary<string, string>();

        static void Main(string[] args)
        {
            fileList.Add("http://intranet/a.pdf", @"C:\test\a.pdf");
            fileList.Add("http://intranet/b.pdf", @"C:\test\b.pdf");
            fileList.Add("http://intranet/c.pdf", @"C:\test\c.pdf");
            fileList.Add("http://intranet/d.pdf", @"C:\test\d.pdf");
            fileList.Add("http://intranet/e.pdf", @"C:\test\e.pdf");
            fileList.Add("http://intranet/f.pdf", @"C:\test\f.pdf");
            fileList.Add("http://intranet/g.pdf", @"C:\test\g.pdf");
            fileList.Add("http://intranet/h.pdf", @"C:\test\h.pdf");
            fileList.Add("http://intranet/i.pdf", @"C:\test\i.pdf");
            fileList.Add("http://intranet/j.pdf", @"C:\test\j.pdf");
            fileList.Add("http://intranet/k.pdf", @"C:\test\k.pdf");
            fileList.Add("http://intranet/l.pdf", @"C:\test\l.pdf");

            // Start the first 5 - which should keep no more than 5 running at any given time
            for (int i = 0; i < 5; i++)
            {
                KeyValuePair<string, string> file = fileList.ElementAt(i);
                InitiateDownload(file.Key, file.Value, FileDownloadCompleted, file.Key);
            }

            Console.ReadKey();
        }

        static void InitiateDownload(string RemoteAddress, string LocalFile, AsyncCompletedEventHandler CompletedCallback, object userToken)
        {
            // Remove file from queue
            fileList.Remove(userToken.ToString());

            WebClient wc = new WebClient();
            wc.DownloadFileCompleted += CompletedCallback;
            wc.DownloadFileAsync(new Uri(RemoteAddress), LocalFile, userToken);
        }

        static void FileDownloadCompleted(object sender, AsyncCompletedEventArgs e)
        {
            if (e.Error != null)
                Console.WriteLine("Error downloading {0}: {1}", e.UserState, e.Error);
            else if (e.Cancelled)
                Console.WriteLine("Download of {0} cancelled.", e.UserState);
            else
                Console.WriteLine("File with userToke {0} completed.", e.UserState);

            // if there are files left in queue start another
            if (fileList.Count > 0)
                InitiateDownload(fileList.First().Key, fileList.First().Value, FileDownloadCompleted, fileList.First().Key);

            // Dispose of the WebClient that just finished
            ((IDisposable)sender).Dispose();
        }
    }
}

Open in new window

0
 
LVL 15

Expert Comment

by:Russell_Venable
ID: 34934751
I wrote something simular yesterday but I didn't post it cause I could not get it to work. Not sure why it is not working either.
0
 

Author Comment

by:sadlermd
ID: 34935004
My personal feeling is one of 2 things is happening:

1. I have a bug in my program - I have several other things running in other threads.

2. There is something about the web server response mechanics that I am not aware of.

Either way, this is beginning to feel like a run down a rabbit hole.

I will put together a piece of code that closely replicates my application.

Rick
0
 

Author Comment

by:sadlermd
ID: 34937679
tgerbert:

Your code snippet is the same logic I employ except for 4 subtle differences:

1. I fire off the InitiateDownload() calls from inside a BackgroundWorker thread
2. I create the WebClient instance in a using statement.
3. I synchronize the calls to InitiateDownload with AutoResetEvent instead of a custom static object.
4. I don't call Dispose on the webclient.

Of these differences number 3 and 4 are the most intriguing. Anyway, I will apply those differences in my code and see what happens!

Thanks again,

Rick

0
 
LVL 33

Expert Comment

by:Todd Gerbert
ID: 34937753
Maybe it's the "using" statement.  Remember that at the end of a "using" block Dispose() is called on the object declared in the using statement.

using (WebClient wc = new WebClient())
{
  wc.DownloadFileCompleted += downloadEventHandler;
  wc.DownloadFileAsync(new Uri(someUrl));
} // wc.Dispose() is called here by compiler, but DownloadFileAsync may not have completed yet

Open in new window

0
 

Author Closing Comment

by:sadlermd
ID: 34938486
Thanks for the help!
0
 

Author Comment

by:sadlermd
ID: 34939533
tgerbert:

For clarification, I need to elaborate on the fix. For anyone who needs it, your code snippet does work. But the problem with my program was a bug in my code.

The content I was passing in the UserToken was a reference type; I was reusing it instead of creating a new instance of the object - that's why the UserState was always the value of the last download request.

Rick
0
 
LVL 33

Expert Comment

by:Todd Gerbert
ID: 34943932
Keep in mind my comments regarding the "using" statement with asynchronous method calls - that still applies.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Article by: Ivo
C# And Nullable Types Since 2.0 C# has Nullable(T) Generic Structure. The idea behind is to allow value type objects to have null values just like reference types have. This concerns scenarios where not all data sources have values (like a databa…
This article describes a simple method to resize a control at runtime.  It includes ready-to-use source code and a complete sample demonstration application.  We'll also talk about C# Extension Methods. Introduction In one of my applications…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now