• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 463
  • Last Modified:

best way to implement a download manager

I need to write a download manager.  Nothing outrageous.  It does not need (necessarily) to support resumable downloads (the files will be coming from apache 1.3x) but it should show files progress and update a given gridview row when they are complete.  It doesn't even need to persist across executions of the application so no local db is needed - though this will likely be nice in the future.  I plan on using indy9's http client to download the files and I do have experience with it (though not a huge amount).  

Essentially users would be presented with a list of files to download.  When each one is clicked it shoudl create a new row in the gridview (I'm just fine with the actual adding) and create a new http client (indy?) to download it.    I'm just not sure which technique would be best to update the download manager.  Perhaps something like this:?

1. user clicks, triggering download procedure
2. download procedure generates a random id (or other serial number) that is linked to this particular download.

This is where I go wonky. I'm a php guy. In php I would build a multidimensional array of $downloads["id"]["url"]  and could access / update information about this download by accessing it's array elements.    In delphi I do not know how to do this as the closest thing to an array I've found is the tstringlist and that is one dimensional.  

3. on work in each indy client, a procedure would be called, passing the unique id generated in 2 that would update the gridview row's download progress bar, or showing errors / complete status.

There are plenty of holes in my technique and I would love to know how you folks have implemented things like this in the past.   I appreciate it folks.
0
hibbidiji
Asked:
hibbidiji
  • 2
  • 2
1 Solution
 
TheRealLokiSenior DeveloperCommented:
You _can_ have dynamic arrays in delphi, but you'll find that an object list works better. You can have a list in the list even
I'd try to make it as Object Oriented as possible from the get-go.
e.g. make a "download object" that holds info about the download
store each download in a list
you might want to have more than 1 list
e.g. a list of "available" downlaods
and a list of "downloads being downloaded"
that will actually use less resources than a dynamic array of all possibilities.

here's a simple example of how to use an objectlist

e.g.

uses contnrs;

type
    TDownloadObject = class // just an example class
    public
        uniqueid: integer;
        url: string;
        totalsize: int64;
        bytesdownloaded: int64
        starttime: tdatetime;
        otherlist: tlist; // in case you did want another list in here for any reason, e.g. a list of failed attempts, what-have-you
        constructor Create(uniqueid_: integer; url_: string; totalsize_: int64; starttime_: tdatetime);
        destructor Destroy; override;
    end;

constructor TDownloadObject.create(uniqueid_: integer; url_: string; totalsize_: int64; starttime_: tdatetime);
    begin
        inherited Create;
        uniqueid := uniqueid_;
        url := url_;
        totalsize := totalsize_;
        starttime := starttime_;
        otherlist := TList.create;
    end;

destructor TDownloadObject.Destroy;
    begin
        otherlist.clear;
        otherlist.free;
        inherited Destory;
    end;

// example of using the object list
var
    mylist: TObjectList;
    i: integer;
begin
    mylist := TObjectList.create;
    try
        mylist.add( TDownloadObject.Create(1, url, totalsize, starttime );
        mylist.add( TDownloadObject.Create(2, url, totalsize, starttime );
       
// now loop through the list
        for i := 0 to pred(mylist.count) do //lists are zero based and go to count - 1
        begin
            memo1.lines.add('url ' +inttostr(i) + ' is: ' + TDownloadObject(mylist[i]).URL);
        end;
    finally
        mylist.free; // actually calls the destroy method of each object in the list for you
    end;

I haven't actually written a download manager, but to do what you say, I'd have a list of "downloadable" files

type TDownloadableFile = class
public
    filename: string;
    url: string;
etc...
end;

and a list of "downloadobjects" similar to the TDownloadObject at the top of this reply, but I'd actually put the HTTP client there in a thread that lives in this object, so you can ask the object to "pause" "resume" "cancel" etc easily
0
 
hibbidijiAuthor Commented:
That was a fantastic reply!   I can't say I've ever seen a more comprehensive answer to a question.   I would appreciate it if you were able to elaborate on your last comments about putting the http client inside the object.   I'm afraid I'm not sure how I would do that.  All my OOP experience is PHP and it's just different enough to frustrate me with this.   If you would prefer me to split it to a seperate question I'll happily award you the points and point to the new question.
0
 
TheRealLokiSenior DeveloperCommented:
There are dozens of ways to do this, and my method is by no means the best.
This sample code will show you how to do what you want with the Indy 9 TidHTTPclient in a thread, living inside a "TDownloadableObject".
The thread will "Post Messages" to the main form about it's progress, failure and success.
You can then handle that in any way you like
I wrote this code today with Delphi 7 and Indy 9, and it does work
I've only included the code you need to get your head around the threading and OO.
Should be enough for you to see and learn how to do things.

If you'd like the complete code, best to put it all in a different question, and I'll add the loading from file, selection, and a nice gui, but like I said, you should be able to work things out from the code below

//declare some messages that the download thread will send us to let us know how it is doing
const
  WM_HTTPClientDownloadBeginWork = WM_user + 100;
  WM_HTTPClientDownloadWork = WM_user + 101;
  WM_HTTPClientDownloadEndWork = WM_user + 102;
  WM_HTTPDownloadSucceeded = WM_user + 103;
  WM_HTTPDownloadFailed = WM_user + 104;

// these are to represent what "state" the download is in, i.e. not started, or part way through, or succeeded
type TDownloadStateForImages = (si_Blank, si_Failed, si_Succeeded, si_Downloading_Animation1, si_Downloading_Animation2 = 4);

// This is the thread that does the actual downloading. You tell it what to get and where to put it, and it reports back proress and status. As soon as you create this thread, it will start downloading the file
type TIndyInAThread = class(TThread)
  private
      UniqueID: integer;
      URL: string;
      LocalFilename: string;
      ShowProgress: boolean;
      HandleToPostTo: THandle;
      HTTPClientInsideAThread: TIdHTTP;
  public
// create the IdHTTPClient events for "file progress" so we can use a progress bar if we want
      procedure HTTPClientInsideThreadWorkBegin(Sender: TObject; AWorkMode: TWorkMode; const AWorkCountMax: Integer);
      procedure HTTPClientInsideThreadWorkEnd(Sender: TObject; AWorkMode: TWorkMode);
      procedure HTTPClientInsideThreadWork(Sender: TObject; AWorkMode: TWorkMode; const AWorkCount: Integer);
      constructor Create(URL_: string; LocalFilename_: string; UniqueID_: integer; HandleToPostTo_: THandle);
      procedure Execute; override;
  end;

// This is _1_ file that is in the "download queue". calling "StartDownload" will do just that
type TDownloadableFile = class
  private
      fURL: string;
      procedure SetURL(const Value: string);
  public
      Filename: string;
      UniqueID: integer; // unique to this session only
      TotalFileSize: int64; // use 0 if unknown
      BytesDownloaded: int64;
      State: TDownloadStateForImages;
      property URL: string read fURL write SetURL; // works out and sets "filename" when you set "URL"
      constructor Create(URL_: string; UniqueID_: integer);
      function DownloadPercent: integer;
      procedure StartDownload;
end;



... in the TForm1 Public block, put these procedures. They are to catch the essages sent from the download threads
    Procedure HTTPClientDownloadBeginWork(var Msg:TMessage);Message WM_HTTPClientDownloadBeginWork;
    Procedure HTTPClientDownloadWork(var Msg:TMessage);Message WM_HTTPClientDownloadWork;
    Procedure HTTPClientDownloadEndWork(var Msg:TMessage);Message WM_HTTPClientDownloadEndWork;
    Procedure HTTPDownloadSucceeded(var Msg:TMessage);Message WM_HTTPDownloadSucceeded;
    Procedure HTTPDownloadFailed(var Msg:TMessage);Message WM_HTTPDownloadFailed;




{ TDownloadableFile }

constructor TDownloadableFile.Create(URL_: string; UniqueID_: integer);
    begin
        inherited Create;
        URL := URL_;
        UniqueID := UniqueID_;
        State := si_Blank;
    end;

function TDownloadableFile.DownloadPercent: integer;
    begin
        if TotalFileSize = 0 then
          result := 0
        else
        begin
            result := (BytesDownloaded div TotalFileSize);
            if result > 100 then result := 100; //occasionally the TidHTTPClient does some "extra" bytes for "requesting, etc". You can tidy that up later
        end;
    end;

//work out the filename from the url if we can. You should do far more testing in this, in case you get bogus names in the url!
procedure TDownloadableFile.SetURL(const Value: string);
    var
        s: string;
    begin
        fURL := Value;
        s := Value;
        while pos('/', s) > 0 do
        begin
            delete(s, 1, pos('/', s));
        end;
        while pos('\', s) > 0 do
        begin
            delete(s, 1, pos('\', s));
        end;

        Filename := s;
    end;

//GO! this will save in a \IN\ subdirectory
procedure TDownloadableFile.StartDownload;
    begin
        if State in [si_Blank, si_Failed] then
        begin // only start the download if we are not already downloading
            TIndyInAThread.Create(self.URL, extractFilePath(Paramstr(0)) + 'In\' + self.Filename, self.UniqueID, Form1.Handle);
        end;
    end;

{ TIndyInAThread }

constructor TIndyInAThread.Create(URL_: string; LocalFilename_: string; UniqueID_: integer; HandleToPostTo_: THandle);
    begin
        inherited Create(True);
            FreeOnTerminate := True;
        URL := URL_;
        LocalFilename := LocalFilename_;
        UniqueID := UniqueID_;
        HandleToPostTo := HandleToPostTo_;
        ShowProgress := False;
        HTTPClientInsideAThread := TIdHTTP.Create(nil);

    //    HTTPClientInsideAThread.MaxLineAction := maException;
        HTTPClientInsideAThread.ReadTimeout := 0;
        HTTPClientInsideAThread.AllowCookies := True;
        HTTPClientInsideAThread.ProxyParams.BasicAuthentication := False;
        HTTPClientInsideAThread.ProxyParams.ProxyPort := 0;
        HTTPClientInsideAThread.Request.ContentLength := -1;
        HTTPClientInsideAThread.Request.ContentRangeEnd := 0;
        HTTPClientInsideAThread.Request.ContentRangeStart := 0;
        HTTPClientInsideAThread.Request.ContentType := 'text/html';
        HTTPClientInsideAThread.Request.Accept := 'text/html, */*';
        HTTPClientInsideAThread.Request.BasicAuthentication := False;
        HTTPClientInsideAThread.Request.UserAgent := 'Mozilla/3.0 (compatible; Indy Library)';
        HTTPClientInsideAThread.HTTPOptions := [hoForceEncodeParams];
// you may want to set the HandleRedirects to true... up to you though
        HTTPClientInsideAThread.OnWorkBegin := HTTPClientInsideThreadWorkBegin;
        HTTPClientInsideAThread.OnWork := HTTPClientInsideThreadWork;
        HTTPClientInsideAThread.OnWorkEnd := HTTPClientInsideThreadWorkEnd;

        Resume; // start the thread now, which will start the download
    end;

// this is the bit which does the downloading
procedure TIndyInAThread.Execute;
    var
        FS: TFileStream;
    begin
        try
            ForceDirectories(ExtractFilePath(LocalFilename));
            FS := TFileStream.Create(LocalFilename, fmCreate);
            ShowProgress := True;
            try
                HTTPClientInsideAThread.Get(URL, FS);
                PostMessage(HandleToPostTo, WM_HTTPDownloadSucceeded, UniqueID, 0);
// flesh this out a bit with some testing of the file etc...
            finally
                FS.Free;
            end;
        except
            on e: exception do
              PostMessage(HandleToPostTo, WM_HTTPDownloadFailed, UniqueID, 0);
        end;

        try
            HTTPClientInsideAThread.Free;
        except // do not care about any errors yet, maybe log them later
        end;
        ShowProgress := False;
    end;


procedure TIndyInAThread.HTTPClientInsideThreadWorkBegin(Sender: TObject; AWorkMode: TWorkMode; const AWorkCountMax: Integer);
    begin
        if ShowProgress then
        begin
            PostMessage(HandleToPostTo, WM_HTTPClientDownloadBeginWork, UniqueID, AWorkCountMax);
        end;
    end;

procedure TIndyInAThread.HTTPClientInsideThreadWork(Sender: TObject; AWorkMode: TWorkMode; const AWorkCount: Integer);
    begin
        if ShowProgress then
        begin
            PostMessage(HandleToPostTo, WM_HTTPClientDownloadWork, UniqueID, AWorkCount);
        end;
    end;

procedure TIndyInAThread.HTTPClientInsideThreadWorkEnd(Sender: TObject; AWorkMode: TWorkMode);
    begin
        if ShowProgress then
        begin
            PostMessage(HandleToPostTo, WM_HTTPClientDownloadEndWork, UniqueID, 0);
        end;
    end;

// these procedures catch the windows messages from teh threads and change properties in your download object.
// you can also take this chance to update any progress bars or lists you may be using
Procedure TForm1.HTTPClientDownloadBeginWork(var Msg:TMessage);
    var
        uniqueid_: integer;
        totalbytes_: int64;
        i: integer;
    begin
        uniqueid_ := Msg.wparam;
        totalbytes_ := Msg.LParam;
        i := IndexOfDownloadableFile(uniqueid_);
        if i <> -1 then
        begin
            TDownloadableFile(DownloadableFileList[i]).BytesDownloaded := 0;
            TDownloadableFile(DownloadableFileList[i]).TotalFileSize := totalbytes_;
            TDownloadableFile(DownloadableFileList[i]).State := si_Downloading_Animation1;
            lvDownloadableFiles.Repaint; // just an example, you cna show the state and progress in a tlistview
        end;
    end;

Procedure TForm1.HTTPClientDownloadWork(var Msg:TMessage);
    var
        uniqueid_, bytesinthisblock: integer;
        i: integer;
    begin
        uniqueid_ := Msg.wparam;
        bytesinthisblock := Msg.LParam;
        i := IndexOfDownloadableFile(uniqueid_);
        if i <> -1 then
        begin
            TDownloadableFile(DownloadableFileList[i]).BytesDownloaded :=
              TDownloadableFile(DownloadableFileList[i]).BytesDownloaded + bytesinthisblock;
            lvDownloadableFiles.Repaint; // just an example, you cna show the state and progress in a tlistview
        end;
    end;

Procedure TForm1.HTTPClientDownloadEndWork(var Msg:TMessage);
    var
        uniqueid_: integer;
        i: integer;
    begin
        uniqueid_ := Msg.wparam;
        i := IndexOfDownloadableFile(uniqueid_);
        if i <> -1 then
        begin
// nothing to do really, maybe hide any progress bar you may be using
            lvDownloadableFiles.Repaint; // just an example, you cna show the state and progress in a tlistview
        end;
    end;

procedure TForm1.HTTPDownloadFailed(var Msg: TMessage);
    var
        uniqueid_: integer;
        i: integer;
    begin
        uniqueid_ := Msg.wparam;
        i := IndexOfDownloadableFile(uniqueid_);
        if i <> -1 then
        begin
            TDownloadableFile(DownloadableFileList[i]).State := si_Failed;
            lvDownloadableFiles.Repaint; // just an example, you cna show the state and progress in a tlistview
        end;
    end;

procedure TForm1.HTTPDownloadSucceeded(var Msg: TMessage);
    var
        uniqueid_: integer;
        i: integer;
    begin
        uniqueid_ := Msg.wparam;
        i := IndexOfDownloadableFile(uniqueid_);
        if i <> -1 then
        begin
            TDownloadableFile(DownloadableFileList[i]).State := si_Succeeded;
            lvDownloadableFiles.Repaint; // just an example, you cna show the state and progress in a tlistview
        end;
    end;
0
 
hibbidijiAuthor Commented:
I apologize for the lateness of my reply-  for some reason the ee alerts were spam listed in my email.

Here is the new question with the request for the complete source - just for the sake of completeness

http://www.experts-exchange.com/Programming/Programming_Languages/Delphi/Q_21865836.html

Thanks!
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now