Indy HTTP component and UserAgent

I am using Delphi 5 and my application uses the Indy HTTP component that came with the Pro version.

I am having a problem with the retrieved text from a web page.  I am trying to retrieve a web page at amazon.com.  The problem is that the retrieved data does not match what I get when I use Internet Explorer 6's "View > Source" option.  There are alot of differences between the two text results.

I thought it might have to do with the TIdHTTP's .Request.UserAgent value, which is natively set to 'Mozilla/3.0 (compatible; Indy Library)'.   I have changed this value to:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

...first, but that didn't make any difference, so I set it to...

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; T312461; .NET CLR 1.1.4322)

...(which is the thumbprint that my IE6 leaves behind in a web log on my server).  This too did not make any difference.

Is there something I am overlooking?

Thanks.
Monroe406Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Wim ten BrinkSelf-employed developerCommented:
Cookies, perhaps?
0
geobulCommented:
A solution eithout using Indy:

uses URLMon;

procedure TForm1.Button1Click(Sender: TObject);
begin
  // replace the URL with yours and the local filename also in the line below
  if URLDownloadToFile(nil, 'http://www.experts-exchange.com/Programming/Programming_Languages/Delphi/', 'c:\Experts.html', 0, nil) <> 0 then
    MessageBox(Handle, 'Cannot download the file.', PChar(Application.Title), MB_ICONERROR or MB_OK);
end;

Regards, Geo
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Monroe406Author Commented:
No, cookies are not the answer.  I just searched and deleted any cookies I could find containing the string "ebay" (I meant to say ebay.com, not amazon.com in my original posting).   Deleting the cookies made no difference.
0
Monroe406Author Commented:
Neither comment worked.  The resulting text from the "GET" still is different from what I get using IE 6 and Firefox 0.9.

I am beginning to suspect that there is some redirection going on behind the scenes that the browsers can handle, but the Indy and URLDownloadToFile are not responding to.

To this end, perhaps I need to find a utility that will tell me what GET requests are occuring between my browser and the website?  Does anyone know of any such tools that will tell me what URL is actually being displayed within IE?
0
Wim ten BrinkSelf-employed developerCommented:
There's one tool that might help but it captures a bit more than you might prefer. http://www.ethereal.com/ Ethereal is a tool that captures all network traffic between you and the rest of the network and I assume you're already familiar with it. If not, download it (it's free) and compare the network traffic of IE with the network traffic of Indy. Then you might notice where things go different.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Delphi

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.