Delphi HTTPSendRequest failure on HTTPS only

Posted on 2007-07-31
Last Modified: 2008-03-26
While using HTTPSendRequest, I get a lot of connection issues with secure sites. I'm posting the applicable code up to the error from HTTPSendRequest. The error I get is 12031. It runs perfectly fine on standard HTTP, but when it's working on some HTTPS connections, I get this error. For example, downloading URLs recursively from will throw this error after the first URL or so. I'm using a custom InetCrackURL routine, as InternetCrackUrl() was returning invalid hosts and invalid file/params for GET requests (although I confirmed manually that the URLs were entirely valid, and very simple URLs. None of them had any complex parameters. For example: would get corrupted by InternetCrackUrl()). Regardless, I have tried without the wPort := <const> section.. I added that only recently, but did not solve the problem. Any idea why this fails on HTTPS?

procedure InetCrackURL(const AURL: String; out AHost, AFile: String; out ASecure: Boolean);
  sFile: String;
  URI := TIdURI.Create(AURL);
    ASecure := UpperCase(URI.Protocol) = 'HTTPS';
    AHost   := URI.Host;
    AFile   := URI.Path + URI.Document + URI.Params;

function DownloadSecureURLEx(const AURL: String; var AOutFile, AMimeType: String): Boolean;
  hConnect, hFile, hInet: HINTERNET;
  dwBytesRead: DWORD;
  readBuffer: packed array[0..$1FFF] of Byte;
  sHeader: String;
  sHost, sFile: String;
  bIsSecure: Boolean;
  fsOut: TFileStream;
  lpstrAccept: LPSTR;
  lpstrReferer: LPSTR;
  wPort: Word;
  dwFlag, dwFlagSize: DWORD;
  C_USER_AGENT    = 'User-Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)';
  hConnect := nil;
  hFile    := nil;
  hInet    := nil;

  Result   := False;
  SetLength(AMimeType, 0);

  //set up some basic URL crack data
  InetCrackURL(AURL, sHost, sFile, bIsSecure);
  frmMain.Memo1.Lines.Add('URL: ' + AURL);
  frmMain.Memo1.Lines.Add('Host: ' + sHost);
  frmMain.Memo1.Lines.Add('File: ' + sFile);

    if (hInet = nil) then
      raise Exception.Create('hInternet = nil from InternetOpen()');

    //set timeout option
    dwFlag := 3000;
    dwFlagSize := SizeOf(dwFlag);

    InternetSetOption(hInet, INTERNET_OPTION_CONNECT_TIMEOUT, @dwFlag, dwFlagSize);

    //only recently added.. same error before this explicit addition
    if (bIsSecure) then
    //now make a connection to the actual host
    hConnect := InternetConnect(hInet, PChar(sHost), wPort, nil, nil, INTERNET_SERVICE_HTTP, 0, 0);
    if (hConnect = nil) then
      raise Exception.CreateFmt('Unable to connect to server %s', [sHost]);

    //open a request and set up the GET data request
    lpstrAccept := PChar('Accept: *.*');
    lpstrReferer := PChar(AURL);

    //on a nil value for lpzVersion, HTTP/1.1 is used per KB
    dwFlag := C_CONNECT_FLAGS;
    if (bIsSecure) then
      dwFlag := dwFlag or INTERNET_FLAG_SECURE;

    hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
      lpstrReferer, @lpstrAccept, C_CONNECT_FLAGS, 0

    if (hFile = nil) then
      raise Exception.Create('Could not open HTTP request via HTTPOpenRequest().');

    //Add any extra headers to the request, raising an exception if not siucessful
    //if not HTTPAddRequestHeaders(aHFile, PChar(s), length(s), HTTP_ADDREQ_FLAG_ADD) then
    //raise(EHTTPAddReqError.create('Could not add HTTP request header'));

    // Send the request, raising an exception if not successful

    //pass the original requesting URL as the referrer
    frmMain.memo1.lines.add('URL: ' + AURL);
    if (not HTTPSendRequest(hFile, nil, 0, nil, 0)) then
      ShowMessage(Inttostr(getlasterror()) + ' ' + sHost);
      raise Exception.Create('Could not send HTTP request via HTTPSendRequest().');
Question by:akede2001
    LVL 21

    Expert Comment

     I use Indy for HTTPs and it works for me.  Why are you wanting to "reinvent the wheel"?  I would try it using Indy.  If it works then you can trace into their code and see what they did differently.  If ti does not then you know there is a problem (most likely router issue or similar) outside of your code.

    Author Comment

    This function is being used in an asynchronous IInternetProtocol interface. Previously, I was using a function wrapped around TIdHTTP using SSL intercepts.. but it was too slow, and doesn't work great when using the objects with asynchronous downloading via Start, Read, and Terminate events received via IInternetProtocol. The increase in download speed is notable, which leads me to believe that the TIdHTTP component is poorly developed. Indy has also confirmed significant and intentional changes that would add more stability and error notification at an exchange of speed.

    Author Comment

    Ok, I fixed it. It's actually just an oversight when I was writing the routine.

        dwFlag := C_CONNECT_FLAGS;
        if (bIsSecure) then
          dwFlag := dwFlag or INTERNET_FLAG_SECURE;

        hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
          lpstrReferer, @lpstrAccept, C_CONNECT_FLAGS, 0

    Should be:

        hFile := HTTPOpenRequest(hConnect, PChar('GET'), PChar(sFile), nil,
          lpstrReferer, @lpstrAccept, dwFlag, 0

    When I wrote the routine to check for a secure connection, it updates a dwFlag value with the standard flags and sets INTERNET_FLAG_SECURE. Although this was being done, I was still passing the standard connection flags via C_CONNECT_FLAGS.

    Problem solved, routine works perfectly now with HTTP and HTTPS, without the overhead of Indy. I don't recall who said it, but there was an official response from Indy that confirms newer versions are slower so they are more lenient on errors, missing or invalid data, slow server responses, and variations in server responses.

    Author Comment

    I'm new here.. trying to figure out how to close this question now. No more review is needed, it's working now.
    LVL 21

    Expert Comment

    Before you close it I would be interested in knowing what kind of speed increase you are seeing...

    Author Comment

    An interesting thread regarding the topic:

    With some details about what's slowing it up, and why. In addition to that, I've found that when I browse some sites that don't return valid timestamps, Indy tries to convert the time stamp anyway and it throws an exception, which is unhandled in Indy. Just a lot of little things like that which you'd need to look into before being able to use a component that seems to change drastically every time it's released. I'm not even sure what version of Indy I have installed now, and I've lost count of how many times I've had to check out which way to implement the SSL handlers around the standard HTTP components to get HTTPS.
    LVL 1

    Accepted Solution

    PAQed with points refunded (500)

    EE Admin

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    6 Surprising Benefits of Threat Intelligence

    All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

    In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
    Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
    Need more eyes on your posted question? Go ahead and follow the quick steps in this video to learn how to Request Attention to your question. *Log into your Experts Exchange account *Find the question you want to Request Attention for *Go to the e…
    Sending a Secure fax is easy with eFax Corporate ( First, Just open a new email message.  In the To field, type your recipient's fax number You can even send a secure international fax — just include t…

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    17 Experts available now in Live!

    Get 1:1 Help Now