Solved

curl problem with some HTTPS sites

Posted on 2014-03-10
11
1,197 Views
Last Modified: 2014-03-26
I have an odd problem with the code below (that I got from another question here).  It works so far on 'http' sites and on most 'https' sites.  But not on one of my own sites and a client site I need it to connect to.  On those two sites I get a blank screen.  I re-arranged the original code to at least print out the url so I could see that it ran.
<?php
// url
$url = 'https://www.dibsiam.com/';
echo $url;

// Create a curl handle
$ch = curl_init($url);

$options = array(
        CURLOPT_FOLLOWLOCATION => true,         // follow redirects
        CURLOPT_ENCODING       => "",           // handle all encodings
        CURLOPT_USERAGENT      => "Mozilla/5.0",     // who am i
        CURLOPT_SSL_VERIFYHOST => 0,            // don't verify ssl
        CURLOPT_SSL_VERIFYPEER => true,        //
        CURLOPT_VERBOSE        => 1                //
    );
curl_setopt_array($ch,$options);

// Execute
curl_exec($ch);

// Check if any error occurred
if(!curl_errno($ch)) {
	$info = curl_getinfo($ch);

	echo 'Took ' . $info['total_time'] . ' seconds to send a request to ' . $info['url'];
	echo '<pre>';
	print_r($info);
	echo '</pre>';

}

// Close handle
curl_close($ch);
?>

Open in new window

0
Comment
Question by:Dave Baldwin
  • 6
  • 4
11 Comments
 
LVL 82

Author Comment

by:Dave Baldwin
ID: 39919057
Ok, here's version .9394959 of this code that appears to work.  However, I had to turn off SSL checking and set the SSL version.  I would prefer to have one that works with the checking turned back on.
<?php
// url
$url = 'https://www.dibsiam.com/';
echo $url.'<br>';

// Create a curl handle
$ch = curl_init($url);

$options = array(
        CURLOPT_FOLLOWLOCATION => true,         // follow redirects
        CURLOPT_ENCODING       => "",           // handle all encodings
        CURLOPT_USERAGENT      => "Mozilla/5.0",     // who am i
        CURLOPT_SSL_VERIFYHOST => 0,            // don't verify ssl
        CURLOPT_SSL_VERIFYPEER => false,        //
	CURLOPT_SSLVERSION     => 3,
        CURLOPT_VERBOSE        => 1                //
    );
curl_setopt_array($ch,$options);

// Execute
curl_exec($ch);

$err     = curl_errno($ch);
$errmsg  = curl_error($ch) ;
echo "Error # $err : Error message $errmsg";
$info = curl_getinfo($ch);
echo '<pre>';
print_r($info);
echo '</pre>';

//}

// Close handle
curl_close($ch);
?>

Open in new window

0
 
LVL 58

Expert Comment

by:Gary
ID: 39919117
No error message?
Do you get any headers?
Is it possible that curl is using outdated CA Roots? Might explain most sites work but some don't
http://curl.haxx.se/docs/caextract.html
0
 
LVL 82

Author Comment

by:Dave Baldwin
ID: 39919167
If I turn on the SSL verify, the error message shows in the second code.  And it did say there was a problem with the certificate.
Error # 60 : Error message SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I downloaded the 'pem' file but it's not clear where I should put it to use it.  The 'cURL' directory had newer versions of the dll's so I copied them over to the PHP directory.  Doesn't change anything.  I also tried this code on a newer version of PHP and it didn't change the problem.
0
 
LVL 58

Expert Comment

by:Gary
ID: 39919186
Well its definitely an issuing authority problem.
What are you running on ? Centos, Ubuntu? The path maybe different, I use Centos.
Sample update code from SSH (make a backup of whats there to start with just in case...)

curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt

Hold off on that command, what distro are you using?
0
 
LVL 82

Author Comment

by:Dave Baldwin
ID: 39919223
It turns out that my even newer versions of PHP for Windows have a newer version of cURL, 7.24.0, that works fine.  The versions that are not working have 7.21.0 so I guess it's time (again) to upgrade PHP on this machine.

I have to write a command line program that does uploads and downloads from 4 different sites and I want to do it over SSL/TLS of course.  I'll end up sticking it in a batch file so the office people can do it with just a double click and very little thought.  I couldn't find a PHP to EXE compiler that would do this.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 58

Expert Comment

by:Gary
ID: 39919243
cURL version wouldn't matter as its to do with the CA certs it is using and cURL doesn't come with them anymore so updating wouldn't update the roots.
Possible options, if you want to manually add the CA root.
http://curl.haxx.se/docs/sslcerts.html
0
 
LVL 82

Author Comment

by:Dave Baldwin
ID: 39919266
The methods implemented in the newer versions of cURL for PHP may be what's making the difference.  But in any case, I now have two ways of using cURL in PHP.  Note that 'cURL' in PHP is independent of the regular cURL program.
0
 
LVL 58

Expert Comment

by:Gary
ID: 39919278
Maybe so...
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39920234
Dave, I think the "encoding" is permissive, not restrictive.  This script worked for me on the Twitter SSL address.  See line 48 and 55-58
http://www.iconoun.com/demo/curl_get_example.php

<?php // /demo/curl_get_example.php
error_reporting(E_ALL);


// DEMONSTRATE THE BASICS OF CURL
// SOMETHING LIKE demo/curl_get_example.php?url=http://twitter.com


// YOU COULD HAVE SOMETHING LIKE THIS
$url = isset($_GET["url"]) ? $_GET["url"] : 'http://twitter.com';

// BUT SINCE IT IS ON MY SERVER, I HAVE HARD-CODED THIS
$url = 'https://twitter.com/RayPaseur';

// TRY THE REMOTE WEB SERVICE
$htm = my_curl($url);

// SHOW THE WORK PRODUCT OR BARK OUT ERROR MESSAGES
echo "<pre>";
echo PHP_EOL . '<strong>' . $url . '</strong>' . PHP_EOL;
echo PHP_EOL . htmlentities($htm);
echo PHP_EOL;


// A FUNCTION TO RUN A CURL-GET CLIENT CALL TO A FOREIGN SERVER
function my_curl
( $url
, $timeout=5
, $error_report=TRUE
)
{
    $curl = curl_init();

    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE THIS BLANK

    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
    curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );

    // ADDED FOR DAVE BALDWIN
    curl_setopt( $curl, CURLOPT_SSL_VERIFYHOST, FALSE  );
    curl_setopt( $curl, CURLOPT_SSL_VERIFYPEER, FALSE  );
	curl_setopt( $curl, CURLOPT_SSLVERSION,     3      );
    curl_setopt( $curl, CURLOPT_VERBOSE,        TRUE   );

    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            var_dump($inf);
        }
        curl_close($curl);
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}

Open in new window

HTH, ~Ray
0
 
LVL 82

Accepted Solution

by:
Dave Baldwin earned 0 total points
ID: 39921169
Thanks, I'll give it a try later.  There were only two sites that gave me a problem with the code I posted.  All the rest (so far) worked fine and it appears to be a difference in the version of 'libcurl' that PHP is using.
0
 
LVL 82

Author Closing Comment

by:Dave Baldwin
ID: 39955451
After testing, I found that I was only having this problem on PHP 5.2.17 on Windows.  PHP 5.3 and 5.4 on Windows worked fine.  Thanks for your help but clearly PHP 5.2 won't be receiving any updates anymore.  The 'solution' is just to use a more recent version that works right.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

These days socially coordinated efforts have turned into a critical requirement for enterprises.
This article discusses four methods for overlaying images in a container on a web page
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now