?
Solved

Download a tricky file with Indy in Delphi 2009

Posted on 2009-12-16
19
Medium Priority
?
929 Views
Last Modified: 2012-05-11
If I paste this link into my browser, I eventually get a "download/open" dialog box so I can save the xml file
I want to automate this in Delphi 2009 using Indy
e.g. TidHTTP

The link is
http://download.xmltv.co.nz/archive=none&icons=large&links=remote&provider=freeview/TC-TV1A/TC-TV2S/FV-tv3plus1/TC-TV4/TVGuide.xml

It should be easy, but I get a "page not found" when I try...
note: I can download other files easily enough :-/

0
Comment
Question by:TheRealLoki
  • 7
  • 6
  • 4
  • +1
19 Comments
 
LVL 25

Expert Comment

by:epasquier
ID: 26070259
can you provide a sample of the code you are using so that we might start from there to test and possibly fix what's wrong ?
0
 
LVL 13

Accepted Solution

by:
ThievingSix earned 1200 total points
ID: 26070342
Do you have to use indy?
program Project61;

{$APPTYPE CONSOLE}

uses
  Windows,
  SysUtils,
  WinInet;

function DownloadURL(const aUrl: string): AnsiString;
var
  hSession: HINTERNET;
  hService: HINTERNET;
  lpBuffer: array[0..1024 + 1] of AnsiChar;
  dwBytesRead: DWORD;
begin
  Result := '';
  hSession := InternetOpen('MyApp', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  try
    if Assigned(hSession) then
    begin
      hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, 0, 0);
      if Assigned(hService) then
        try
          while True do
          begin
            dwBytesRead := 1024;
            InternetReadFile(hService, @lpBuffer, 1024, dwBytesRead);
            if dwBytesRead = 0 then break;
            lpBuffer[dwBytesRead] := #0;
            Result := Result + lpBuffer;
          end;
        finally
          InternetCloseHandle(hService);
        end;
    end;
  finally
    InternetCloseHandle(hSession);
  end;
end;

var
  Test : String;
begin
  Writeln(String(DownloadURL('http://download.xmltv.co.nz/archive=none&icons=large&links=remote&provider=freeview/TC-TV1A/TC-TV2S/FV-tv3plus1/TC-TV4/TVGuide.xml')));
  readln;
end.

Open in new window

0
 
LVL 25

Expert Comment

by:epasquier
ID: 26070385
Nice one ThievingSix !
I'm going to copy this in my toolbox as it is a most easy way.
I'm not too sure about the While True part and break if BytesRead=0, it could end too soon in some cases no ? Will have to check in InternetReadFile how things are handled.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 13

Expert Comment

by:ThievingSix
ID: 26070408
This was just a quick find from http://www.swissdelphicenter.ch/torry/showcode.php?id=412 to see if Loki needed to use indy or if something like this was acceptable.

"To ensure all data is retrieved, an application must continue to call the InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero."

The code above isn't perfect, but like I said in the beginning I was just probing for info.
0
 
LVL 25

Expert Comment

by:epasquier
ID: 26070496
:o) How us poor programmers where doing before google (or altavista) ? Life is almost too easy now
0
 
LVL 13

Expert Comment

by:ThievingSix
ID: 26070563
We had bookmarks ;). You know, the real kind!
0
 
LVL 25

Expert Comment

by:epasquier
ID: 26070608
ok, I rephrase : How us programmers where doing before Internet ?
Well, I suppose we had no problems with file downloading then :o))
0
 
LVL 23

Assisted Solution

by:Ferruccio Accalai
Ferruccio Accalai earned 800 total points
ID: 26070894
This is an example using Indy 10 TIdHttp, wich save your TVGuide.xml in C:\ clicking on Button1

FORM
object Form1: TForm1
  Left = 196
  Top = 92
  Caption = 'Form1'
  ClientHeight = 444
  ClientWidth = 680
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object Button1: TButton
    Left = 374
    Top = 64
    Width = 75
    Height = 25
    Caption = 'Button1'
    TabOrder = 0
    OnClick = Button1Click
  end
  object IdHTTP1: TIdHTTP
    AllowCookies = True
    ProxyParams.BasicAuthentication = False
    ProxyParams.ProxyPort = 0
    Request.ContentLength = -1
    Request.Accept = 'text/html, */*'
    Request.BasicAuthentication = False
    Request.UserAgent = 'Mozilla/3.0 (compatible; Indy Library)'
    HTTPOptions = [hoForceEncodeParams]
    OnHeadersAvailable = IdHTTP1HeadersAvailable
    Left = 284
    Top = 74
  end
end
UNIT
unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, IdBaseComponent, IdComponent, IdTCPConnection, IdTCPClient, IdHTTP,
  StdCtrls, IdHeaderList;

type
  TForm1 = class(TForm)
    IdHTTP1: TIdHTTP;
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
    procedure IdHTTP1HeadersAvailable(Sender: TObject; AHeaders: TIdHeaderList;
      var VContinue: Boolean);
  private
    { Private declarations }
    FileName: string;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.Button1Click(Sender: TObject);
var
  MemStream: TMemoryStream;
  FileStream: TFileStream;
begin
  MemStream := TMemoryStream.Create;
  try
    IdHTTP1.Get('http://download.xmltv.co.nz/archive=none&icons=large&links=remote&provider=freeview/TC-TV1A/TC-TV2S/FV-tv3plus1/TC-TV4/TVGuide.xml', MemStream);
    MemStream.Position := 0;
    FileStream := TFileStream.Create(FileName, fmCreate);
    try
      MemStream.SaveToStream(FileStream);
    finally
      FileStream.Free;
    end;
  finally
    MemStream.Free;
  end;
end;

procedure TForm1.IdHTTP1HeadersAvailable(Sender: TObject;
  AHeaders: TIdHeaderList; var VContinue: Boolean);
var
  ContentDisposition: string;
begin
  ContentDisposition := AHeaders.Values['Content-Disposition'];
  ContentDisposition := Copy(ContentDisposition, 1 + Pos(';', ContentDisposition), MaxInt);
  ContentDisposition := StringReplace(ContentDisposition, 'filename=', '', [rfIgnoreCase]);
  ContentDisposition := StringReplace(ContentDisposition, '"', '', [rfReplaceAll]);
  FileName := 'C:\'+ContentDisposition;
end;

end.

Open in new window

0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26076715
Hi Ferruccio68
Your source is similar to what I already had.

It works for some things, but not for the tvguide

I get an "HTTP/1.2 404 not found" error when I try

here's my modified sample, showing some urls that work, and the tvguide one that doesn't



object Form1: TForm1
  Left = 196
  Top = 92
  Caption = 'Form1'
  ClientHeight = 444
  ClientWidth = 680
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object Button1: TButton
    Left = 374
    Top = 64
    Width = 75
    Height = 25
    Caption = 'Button1'
    TabOrder = 0
    OnClick = Button1Click
  end
  object Memo1: TMemo
    Left = 32
    Top = 112
    Width = 473
    Height = 273
    Lines.Strings = (
      'Memo1')
    TabOrder = 1
  end
  object ComboBox1: TComboBox
    Left = 32
    Top = 8
    Width = 473
    Height = 21
    ItemHeight = 13
    TabOrder = 2
    Text = 
      'http://metvuw.com/forecast/2009121718/rain-nzni-2009121718-06.gi' +
      'f'
    Items.Strings = (
      
        'http://metvuw.com/forecast/2009121718/rain-nzni-2009121718-06.gi' +
        'f'
      'http://xmltv.co.nz/uploads/images/donate3.gif'
      
        'http://download.xmltv.co.nz/archive=none&icons=large&links=remot' +
        'e&provider=freeview/TC-TV1A/TC-TV2S/FV-tv3plus1/TC-TV4/TVGuide.x' +
        'ml'
      
        'http://socialmedialawstudent.com/wp-content/uploads/2009/10/vade' +
        'r-fail1.jpg')
  end
  object IdHTTP1: TIdHTTP
    AllowCookies = True
    RedirectMaximum = 100
    ProxyParams.BasicAuthentication = False
    ProxyParams.ProxyPort = 0
    Request.ContentLength = -1
    Request.Accept = 'text/html, */*'
    Request.BasicAuthentication = False
    Request.UserAgent = 'Mozilla/3.0 (compatible; Indy Library)'
    HTTPOptions = [hoForceEncodeParams]
    OnHeadersAvailable = IdHTTP1HeadersAvailable
    Left = 284
    Top = 74
  end
end

UNIT

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, IdBaseComponent, IdComponent, IdTCPConnection, IdTCPClient, IdHTTP,
  StdCtrls, IdHeaderList;

type
  TForm1 = class(TForm)
    IdHTTP1: TIdHTTP;
    Button1: TButton;
    Memo1: TMemo;
    ComboBox1: TComboBox;
    procedure Button1Click(Sender: TObject);
    procedure IdHTTP1HeadersAvailable(Sender: TObject; AHeaders: TIdHeaderList;
      var VContinue: Boolean);
  private
    { Private declarations }
    FileName: string;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.Button1Click(Sender: TObject);
var
  MemStream: TMemoryStream;
  FileStream: TFileStream;
begin
  IdHTTP1.HandleRedirects := true;

  MemStream := TMemoryStream.Create;
  try
    IdHTTP1.Get(Combobox1.Text, MemStream);
    MemStream.Position := 0;
    FileStream := TFileStream.Create(FileName, fmCreate);
    try
      MemStream.SaveToStream(FileStream);
    finally
      FileStream.Free;
    end;
  finally
    MemStream.Free;
  end;
end;

procedure TForm1.IdHTTP1HeadersAvailable(Sender: TObject;
  AHeaders: TIdHeaderList; var VContinue: Boolean);
var
  ContentDisposition: string;
begin
  ContentDisposition := AHeaders.Values['Content-Disposition'];
  memo1.lines.text := ContentDisposition;
  ContentDisposition := Copy(ContentDisposition, 1 + Pos(';', ContentDisposition), MaxInt);
  ContentDisposition := StringReplace(ContentDisposition, 'filename=', '', [rfIgnoreCase]);
  ContentDisposition := StringReplace(ContentDisposition, '"', '', [rfReplaceAll]);
  if (ContentDisposition <> '') then
    FileName := 'C:\'+ContentDisposition
  else
    Filename := 'C:\file.dat';

end;

end.

Open in new window

0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26076766
Hi ThievingSix, long time no see...
Your source works just fine.
I would have liked to be able to do it in Indy, as I have proxy stuff and other routines hinging on it
But if an Indy solution can't be found, I'll accept your answer as the solution, and rework my existing code somehow
0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26076778
typo above btw
I get an "HTTP/1.2 404 not found" error when I try
actual error is (of course)
"HTTP/1.1 404 not found"
0
 
LVL 13

Expert Comment

by:ThievingSix
ID: 26076834
Long time no see =O?

I believe you can use a proxy with my example, but after school I'll go ahead and give indy a try. You are using indy 10 correct?
0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26076988
yep, Delphi 2009, with default Indy 10 except for that SMTP attachment fix (unrelated).
long time, I don't get on here as much as I used to, and used to answer the same questions as you until real life kicked in...
0
 
LVL 23

Expert Comment

by:Ferruccio Accalai
ID: 26078570
Strange thing, I can download it with that code, in fact i got TVGuide.xml in my C:\

I use Indy 10 and Delphi 2010
0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26085712
maybe it's a Delphi 2009 issue, but unfortunately, that's what I'm using
0
 
LVL 13

Expert Comment

by:ThievingSix
ID: 26085839
Hmm, I doubt it's a Delphi 2009 issue, have you tried updating your indy snapshot?
0
 
LVL 17

Author Comment

by:TheRealLoki
ID: 26108061
I guess it must be an Indy bug then.
I work in a shared environment and cant update Indy for a while, so I've modified ThievingSix's sample (to use streams) and it's working except for proxy, which I'll have to learn how to do later.
Points split
0
 
LVL 17

Author Closing Comment

by:TheRealLoki
ID: 31667025
Indy source identical to mine. Issue is with Indy. Used ThievingSix's code as an alternate
0
 
LVL 13

Expert Comment

by:ThievingSix
ID: 26108153
hSession: = InternetOpen ( 'MyApp', INTERNET_OPEN_TYPE_PROXY, 'proxy: 8080', nil, 0);

You can use InternetSetOption to set the username/password for the proxy. http://msdn.microsoft.com/en-us/library/aa385384%28VS.85%29.aspx
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question