Solved

Curl access https/ssl site with password encryption

Posted on 2014-02-14
9
3,089 Views
Last Modified: 2014-02-15
Dear Experts,

During these two years, I've used the following php curl code to access https/ssl site that is successful and no issue at all, I could collect anything I want after logged into https site.

In the code, I have extracted all hidden keys and cookie info from login form and then
post submit with those key  to final https security check sites with my login name and password. The key/query string for https site setup by php code is matched to what I see
on one of proxy debugger such as Chales or Fidder. And cookie transfer from one
site to next https site is also okay and saved it into one cookie file in my code.

But Now today, I could not access the https site, finally found the reason from proxy debugger,the password is encryption with 128bit . At the beginning or last two days, the password is not encrypted that is why I could access the site in last two days.

The question is what I should do:

Question-1: Who is responsible to do client password encryption on https site? Is it
from browser (Chrome or IE)  with the trusted ssl cert  or from javascript code in the login form ?

Question-2: How can I do the exact password encryption on my php code  at my server
that is exactly same as it is done my browser so that I could pass the
security check for https site through php curl code

Question-3: there is a lots of option for curl_setopt about ssl option setup such as
CURLOPT_CAINFO, CURLOPT_SSLKEY, CURLOPT_SSH_HOST_PUBLIC_KEY_MD5, CURLOPT_SSH_PUBLIC_KEYFILE, CURLOPT_SSH_PRIVATE_KEYFILE  ,etc... from http://hk1.php.net/curl_setopt. Those option is helpful to set password
encryption on my server ? Do I need to copy my https  trusted ssl cert on my browser
and then used it with curl code ? How ?

I have read similar article at
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_25111542.html

but that article is not dealing for the case when  the password is with 128-bit encryption on https site, any other similar and good  article is suggest and now I don't know what I should do next


Please advise
Duncan
<?php
chdir(dirname(__FILE__));
error_reporting(E_ALL);
$cookiefile="cookiefile.txt";@unlink($cookiefile);$handle=fopen($cookiefile, 'w+');
login($cookiefile);

function login($cookiefile){

$url="https://example.com/login.htm?lang=en_US";  //-login-form
$file_url=open_https_url($url,$cookiefile,0,0);  //curl and its setup;
while($file_url==false){echo "\r\nLogin Form-1 again....\r\n";$file_url=open_https_url($url,$cookiefile,0,0);}

//-Extract all hidden key or cookie from the login form first before submit
$data=explode("requestId = '",$file_url);
$r=substr($data[1],0,stripos($data[1],"'"));
$data=explode('name="t" value="',$file_url);
$t=substr($data[1],0,stripos($data[1],'"'));
$data=explode('name="s" value='."'",$file_url);
$s=substr($data[1],0,stripos($data[1],"'"));

//-Setup postfields for the new https site
$postfields="lang=en_US&username=mylogin&password=mypassword";
$postfields=$postfields."&r=".$r."&t=".$t."&s=".$s;

//-Submit the form to next new https site with all key and cookie ready
$url="https://example.com/securitycheck.htm";
$file_url=open_https_url($url,$cookiefile,$postfields,1);
$temp=stripos($file_url,"Welcome mylogin");
while ($temp==false ||$temp==0 ){echo "Submit https site-2 again......\n";$file_url=open_https_url($url,$cookiefile,$postfields,1);
$temp=stripos($file_url,"Welcome mylogin");
} 
echo "Success Login initiation\n"; 
//if the page has "Welcome mylogin" word that proved https access from curl is successful, otherwise looping;

return "success";
}
function open_https_url($url,$cookiefile,$postfields,$postenable) { 
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
$header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";
curl_setopt( $ch, CURLOPT_URL,            $url  );
curl_setopt( $ch, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );
curl_setopt( $ch, CURLOPT_HTTPHEADER,     $header  );
curl_setopt( $ch, CURLOPT_REFERER,        'http://www.google.com'  );
curl_setopt( $ch, CURLOPT_ENCODING,       'gzip,deflate'  );
curl_setopt( $ch, CURLOPT_AUTOREFERER,    TRUE  );
	
if ($postenable==1){         
curl_setopt($ch, CURLOPT_POST, 1); 
curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields); 
}
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/ca.p7b"); 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); 
curl_setopt($ch, CURLOPT_HEADER, 0); 
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile); 
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);    
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); 
$result =curl_exec($ch); 
$err = curl_errno($ch);
$inf = curl_getinfo($ch);
//print_r($inf);
if ($result === FALSE){echo "\n curl fail: $url CURL_ERRNO=$err ";var_dump($inf);}
curl_close ($ch); 
return $result; 
} 

?>

Open in new window

0
Comment
Question by:duncanb7
  • 5
  • 4
9 Comments
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39861225
I got a neglected question alert on this one, and I have a bad feeling about the approach you were using.  It looks like you may have been "cut off."  Here is what I would do:

The first thing I would do is contact the publisher of the web site and tell them about the trouble you are having because of their change.  They may have a formal API that you can use to get the data you want, and if they do, that will be a better solution than trying to use an automated login.  APIs are typically licensed, metered and versioned, so the publisher has a "contract" with the audience and will not break an API without formal notice, probably including deprecation.  When you access a web site without an API, the "scraper scripts" often get broken by even small changes in the HTML document, so an API is the way to go.  You may be required to pay a license fee to use the API.

It is possible that the publisher made this change to prevent automated access to its data.  A few years ago, Google made a change like this, loading the DOM with JavaScript instead of producing clear-text HTML documents.  This frustrated a lot of people who had been scraping Google pages (without permission) but it helped Google keep control of its data and preserve the value in its web service.
0
 
LVL 13

Author Comment

by:duncanb7
ID: 39861242
Dear Ray,

thanks for your reply. I am getting confusing with https/SSL access. There is a few question
for two cases. Hope you can kindly answer this that I am appreciated

Case-1: successful logged in  last years
=======================================
Supposed on browser will do information encryption according to the site SSL cert for https/SSL access, Right ? but why I could access that https site without password encryption in last two years on my server in which I just input php code for  postfield as $postfields="lang=en_US&username=mylogin&password=mypassword";  ?
Whether php curl code NO need to do any encryption on password or login username for accessing https site succesfully ?

Like the case or code in
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_25111542.html  in which there is NO any password
encryption on php curl code  for accessing http/SSL site, Why if it is https site ?

Case-2 Fail logged today
========================
On Charles proxy debugger, I just only found the password is encrypted but
other query string or form hidden key or username all are NOT encrypted. So
you might be right there is API javascript code in the login form  to encrypt
password to 128 bit before accessing https. Why the company needs to do
two times encryption on passsword
? one from API javascript code on login page and second is from origin SSL browser encryption

Please help to point out my concept mistake if any , thanks

Duncan
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39861272
1. There is nothing about HTTPS/SSL that will interfere with cURL.  A cURL request is just like a browser request.  It can be made over HTTP or HTTPS and both will work correctly.

2. "Why the company needs to do two times encryption on password..." That is a question you should direct to the publisher of the web site.  My answer can only be speculative, but it would be "To reduce the risk of unwanted automated access to the web site."
0
 
LVL 13

Author Comment

by:duncanb7
ID: 39861296
Dear Ray,

I have No question on the publisher for additional encryption on password by javascript
API code in login form page.
But my concept to curl on HTTPS/SSL is still confusing even it seems simple.

1. There is nothing about HTTPS/SSL that will interfere with cURL.  A cURL request is just like a browser request.  It can be made over HTTP or HTTPS and both will work correctly.

if you said correct on pt-1, so the info sent by curl request is NOT encrypted/protected ? Is it dangerous ?
As I know when I access https/SSL site on my Chrome browser, the information
I submitted will be encrypted by the browser with the trusted SSL cert before
sending the info data to remote https site. Right ? and the pulisher will do
de-cryption on those info to collect final data at its side. If so, why we don't
need to do the same encryption for curl request on my server to achieve
the same action
?

Hope you understand what I am confusing on difference between curl request
and client browser request for HTTPS site

Duncan
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 108

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 39861348
The transmission is encrypted.  The clear-text data is presented to the script (just as it is presented to the browser).  This explanation from the security firm, Symantec, seems to be on point:
http://www.symantec.com/business/support/index?page=content&id=TECH180521

Nothing special is required by either the browser client or the cURL client.  Please see http://www.laprbass.com/RAY_curl_get_example.php where we read the Twitter web site over HTTPS.

<?php // RAY_curl_get_example.php
error_reporting(E_ALL);


// DEMONSTRATE THE BASICS OF CURL
// SOMETHING LIKE RAY_curl_get_example.php?url=http://twitter.com


// YOU COULD HAVE SOMETHING LIKE THIS
$url = isset($_GET["url"]) ? $_GET["url"] : 'http://twitter.com';

// BUT SINCE IT IS ON MY SERVER, I HAVE HARD-CODED THIS
$url = 'https://twitter.com/RayPaseur';

// TRY THE REMOTE WEB SERVICE
$htm = my_curl($url);

// SHOW THE WORK PRODUCT OR BARK OUT ERROR MESSAGES
echo "<pre>";
echo PHP_EOL . '<strong>' . $url . '</strong>';
echo PHP_EOL . htmlentities($htm);
echo PHP_EOL;


// A FUNCTION TO RUN A CURL-GET CLIENT CALL TO A FOREIGN SERVER
function my_curl
( $url
, $timeout=5
, $error_report=TRUE
)
{
    $curl = curl_init();

    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE THIS BLANK

    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  ); // ANCIENT HISTORY
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
    curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );

    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            var_dump($inf);
        }
        curl_close($curl);
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}

Open in new window

0
 
LVL 13

Author Comment

by:duncanb7
ID: 39861401
Dear Ray,

thanks for your reply and attached articles.
Now it seem It is better for me. if the  following what I said it is correct.

php curl request  or client PC browser request for https is nothing special and they
don't do any data info encryption during transmission. The transmission
encryption on the request data is done by SSL engine, for example, on
my server, Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips mod_bwlimited/1.4 PHP/5.3.21,

the request data encryption is done by my server openSSL enginner or module (not by php curl code) when https request is enabled. And the remote site of https , pulishers'
server will de-crypte the data (encrypted from my server) from pubishers' SSL engine or module. Last words, both algorithm for encryption from my server and decryption from publisher https site are recognized  and is  one of standard of SSL engine with cryption algorithm. In othe words, there is no doubt my server could communicate with the
https site  with SSL data encryption at both sides(my server and remote https site)

I will close this thread if what I said in this post is correct or close to correct

And the additional password encryption API javascript function is found on the remote site
login form page mentioned at top post , and probably I need Node.js help on my server to do the additional password encryption before curl https request on my server
that will simulate the same API action on login form with https request on my local PC browser such as on Chrome.
 

Please advise
Duncan
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39861431
The first part is certainly correct.  Not sure what node.js can do for you (I have not seen the login web page) but I'll take that on faith :-)
0
 
LVL 13

Author Closing Comment

by:duncanb7
ID: 39861447
Thanks for your detailed reply in this thread.
I will try Node.js first  on my server.
If it is not successful, I will switch to do it on my PC browser
for data collection daily to remote https site with using VBA code
that must work but it is not good to do it on my PC instead of my remoter server

Memo for me: And third party SSL cert , its encyrption, public or private keys are other issues and not related to this thread topic

Have a nice day

Duncan
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39861511
Thanks for the points and best of luck with the project, ~Ray
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

I imagine that there are some, like me, who require a way of getting currency exchange rates for implementation in web project from time to time, so I thought I would share a solution that I have developed for this purpose. It turns out that Yaho…
Imagine a situation that you have installed SSL (http://en.wikipedia.org/wiki/Secure_Sockets_Layer) Certificate on your Cisco ASA (Cisco Adaptive Security Appliance) firewall. Installation of SSL certificate on ASA is an another topic for which you …
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now