Delphi HTTPSendRequest failure on HTTPS only

While using HTTPSendRequest, I get a lot of connection issues with secure sites. I'm posting the applicable code up to the error from HTTPSendRequest. The error I get is 12031. It runs perfectly fine on standard HTTP, but when it's working on some HTTPS connections, I get this error. For example, downloading URLs recursively from https://gmail.com will throw this error after the first URL or so. I'm using a custom InetCrackURL routine, as InternetCrackUrl() was returning invalid hosts and invalid file/params for GET requests (although I confirmed manually that the URLs were entirely valid, and very simple URLs. None of them had any complex parameters. For example: http://video.google.com?tw=2 would get corrupted by InternetCrackUrl()). Regardless, I have tried without the wPort := <const> section.. I added that only recently, but did not solve the problem. Any idea why this fails on HTTPS?

procedure InetCrackURL(const AURL: String; out AHost, AFile: String; out ASecure: Boolean);
var
  URI: TIdURI;
  sFile: String;
begin
  URI := TIdURI.Create(AURL);
  try
    ASecure := UpperCase(URI.Protocol) = 'HTTPS';
    AHost   := URI.Host;
    AFile   := URI.Path + URI.Document + URI.Params;
  finally
    FreeAndNil(URI);
  end;
end;

function DownloadSecureURLEx(const AURL: String; var AOutFile, AMimeType: String): Boolean;
var
  hConnect, hFile, hInet: HINTERNET;
  dwBytesRead: DWORD;
  readBuffer: packed array[0..$1FFF] of Byte;
  sHeader: String;
  sHost, sFile: String;
  bIsSecure: Boolean;
 
  fsOut: TFileStream;
  lpstrAccept: LPSTR;
  lpstrReferer: LPSTR;
  wPort: Word;
  dwFlag, dwFlagSize: DWORD;
const
  C_USER_AGENT    = 'User-Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)';
  C_CONNECT_FLAGS = INTERNET_FLAG_DONT_CACHE or
                    INTERNET_FLAG_IGNORE_CERT_CN_INVALID or
                    INTERNET_FLAG_IGNORE_CERT_DATE_INVALID or
                    INTERNET_FLAG_NO_UI;
begin
  hConnect := nil;
  hFile    := nil;
  hInet    := nil;

  Result   := False;
  SetLength(AMimeType, 0);

  //set up some basic URL crack data
  InetCrackURL(AURL, sHost, sFile, bIsSecure);
  frmMain.Memo1.Lines.Add('------');
  frmMain.Memo1.Lines.Add('URL: ' + AURL);
  frmMain.Memo1.Lines.Add('Host: ' + sHost);
  frmMain.Memo1.Lines.Add('File: ' + sFile);
  frmMain.Memo1.Lines.Add('');

  try
    hInet := InternetOpen(C_USER_AGENT, INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, INTERNET_SERVICE_URL);
    if (hInet = nil) then
      raise Exception.Create('hInternet = nil from InternetOpen()');

    //set timeout option
    dwFlag := 3000;
    dwFlagSize := SizeOf(dwFlag);

    InternetSetOption(hInet, INTERNET_OPTION_CONNECT_TIMEOUT, @dwFlag, dwFlagSize);

    //only recently added.. same error before this explicit addition
    if (bIsSecure) then
      wPort := INTERNET_DEFAULT_HTTPS_PORT
    else
      wPort := INTERNET_INVALID_PORT_NUMBER;
     
    //now make a connection to the actual host
    hConnect := InternetConnect(hInet, PChar(sHost), wPort, nil, nil, INTERNET_SERVICE_HTTP, 0, 0);
    if (hConnect = nil) then
      raise Exception.CreateFmt('Unable to connect to server %s', [sHost]);

    //open a request and set up the GET data request
    lpstrAccept := PChar('Accept: *.*');
    lpstrReferer := PChar(AURL);

    //on a nil value for lpzVersion, HTTP/1.1 is used per KB
    //http://msdn2.microsoft.com/en-US/library/aa384233.aspx
    dwFlag := C_CONNECT_FLAGS;
    if (bIsSecure) then
      dwFlag := dwFlag or INTERNET_FLAG_SECURE;

    hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
      lpstrReferer, @lpstrAccept, C_CONNECT_FLAGS, 0
    );

    if (hFile = nil) then
      raise Exception.Create('Could not open HTTP request via HTTPOpenRequest().');

    //Add any extra headers to the request, raising an exception if not siucessful
    //if not HTTPAddRequestHeaders(aHFile, PChar(s), length(s), HTTP_ADDREQ_FLAG_ADD) then
    //raise(EHTTPAddReqError.create('Could not add HTTP request header'));

    // Send the request, raising an exception if not successful

    //pass the original requesting URL as the referrer
    frmMain.memo1.lines.add('URL: ' + AURL);
    if (not HTTPSendRequest(hFile, nil, 0, nil, 0)) then
    begin
      ShowMessage(Inttostr(getlasterror()) + ' ' + sHost);
      raise Exception.Create('Could not send HTTP request via HTTPSendRequest().');
    end;
akede2001Asked:
Who is Participating?
 
Computer101Connect With a Mentor Commented:
PAQed with points refunded (500)

Computer101
EE Admin
0
 
developmentguruPresidentCommented:
 I use Indy for HTTPs and it works for me.  Why are you wanting to "reinvent the wheel"?  I would try it using Indy.  If it works then you can trace into their code and see what they did differently.  If ti does not then you know there is a problem (most likely router issue or similar) outside of your code.
0
 
akede2001Author Commented:
This function is being used in an asynchronous IInternetProtocol interface. Previously, I was using a function wrapped around TIdHTTP using SSL intercepts.. but it was too slow, and doesn't work great when using the objects with asynchronous downloading via Start, Read, and Terminate events received via IInternetProtocol. The increase in download speed is notable, which leads me to believe that the TIdHTTP component is poorly developed. Indy has also confirmed significant and intentional changes that would add more stability and error notification at an exchange of speed.
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
akede2001Author Commented:
Ok, I fixed it. It's actually just an oversight when I was writing the routine.

    dwFlag := C_CONNECT_FLAGS;
    if (bIsSecure) then
      dwFlag := dwFlag or INTERNET_FLAG_SECURE;

    hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
      lpstrReferer, @lpstrAccept, C_CONNECT_FLAGS, 0
    );

Should be:

    hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
      lpstrReferer, @lpstrAccept, dwFlag, 0
    );


When I wrote the routine to check for a secure connection, it updates a dwFlag value with the standard flags and sets INTERNET_FLAG_SECURE. Although this was being done, I was still passing the standard connection flags via C_CONNECT_FLAGS.

Problem solved, routine works perfectly now with HTTP and HTTPS, without the overhead of Indy. I don't recall who said it, but there was an official response from Indy that confirms newer versions are slower so they are more lenient on errors, missing or invalid data, slow server responses, and variations in server responses.
0
 
akede2001Author Commented:
I'm new here.. trying to figure out how to close this question now. No more review is needed, it's working now.
0
 
developmentguruPresidentCommented:
Before you close it I would be interested in knowing what kind of speed increase you are seeing...
0
 
akede2001Author Commented:
An interesting thread regarding the topic:

http://groups.google.com/group/borland.public.delphi.internet.winsock/browse_thread/thread/20e53df351c10cc2/b4e0564f1cb2a40a?lnk=st&q=delphi+indy+slow+TidHTTP&rnum=2&hl=en#b4e0564f1cb2a40a

With some details about what's slowing it up, and why. In addition to that, I've found that when I browse some sites that don't return valid timestamps, Indy tries to convert the time stamp anyway and it throws an exception, which is unhandled in Indy. Just a lot of little things like that which you'd need to look into before being able to use a component that seems to change drastically every time it's released. I'm not even sure what version of Indy I have installed now, and I've lost count of how many times I've had to check out which way to implement the SSL handlers around the standard HTTP components to get HTTPS.
0
All Courses

From novice to tech pro — start learning today.