projects
asked on
How to secure ssl certs that use windows curl
Client using curl communicating with apache server over intranet and internet.
We have a problem where we have built a windows application which currently uses curl along with external ssl .crt and .key certs. (not sure we actually need both?)
This is a self signed client/server app that is our own meaning users have no need to have the certs for anything as only the client/server needs them. Additionally, we will change the certs now and then which are encrypted and packaged inside one of the executable s that is installed as part of the overall package.
The plan is to eventually move everything to libcurl which I think partially solves this problem but that is the point of my question.
The current testing method is that the encrypted certs are extracted from an exe, written to a temp directory (random) so that curl can use them, then the random dir is deleted.
One problem is that if someone really wanted those certs, they could watch files being written to disk, then later undelete the dir/files and gain access to the certs.
Not being use when we'll move to libcurl, the question is, how else could this be done so that the certs are secured?
Are both .crt and .key needed on the client?
And, when we move to libcurl, what should be keep in mind, best practice, methods, etc?
We have a problem where we have built a windows application which currently uses curl along with external ssl .crt and .key certs. (not sure we actually need both?)
This is a self signed client/server app that is our own meaning users have no need to have the certs for anything as only the client/server needs them. Additionally, we will change the certs now and then which are encrypted and packaged inside one of the executable s that is installed as part of the overall package.
The plan is to eventually move everything to libcurl which I think partially solves this problem but that is the point of my question.
The current testing method is that the encrypted certs are extracted from an exe, written to a temp directory (random) so that curl can use them, then the random dir is deleted.
One problem is that if someone really wanted those certs, they could watch files being written to disk, then later undelete the dir/files and gain access to the certs.
Not being use when we'll move to libcurl, the question is, how else could this be done so that the certs are secured?
Are both .crt and .key needed on the client?
And, when we move to libcurl, what should be keep in mind, best practice, methods, etc?
For the client certificates, your question basically boils down to "can I give someone something, but not let them have something". Even without writing to disk, if the client has the certificate, then the user running the client has the certificate. You can generate any number of obfuscation strategies as additional protection, but, in the end, they need to have what you don't want them to have.
Regarding what is actually needed, and their associated risks, there are two items of note:
1) the server's self-signed certificate. The server will use this to prove its own identity. Generally, you give the public key to the client, with which it will decrypt packets from the server, and possibly encrypt packets going to the server. The private key should be held on the server and never exposed to anyone else.
2) the client's certificate. The client will use this to prove its identity. The client will give out the public key. The private key is the part you want to keep hidden/unaccessible, but the client must have access to the private key in order to use the certificate.
If you wanted this as safe as possible, you would generate unique public/private key pairs for each authorized client. The certificates would have to be delivered separately from a "common" installer, though you could set up a web service to deliver them during installation.
Regarding what is actually needed, and their associated risks, there are two items of note:
1) the server's self-signed certificate. The server will use this to prove its own identity. Generally, you give the public key to the client, with which it will decrypt packets from the server, and possibly encrypt packets going to the server. The private key should be held on the server and never exposed to anyone else.
2) the client's certificate. The client will use this to prove its identity. The client will give out the public key. The private key is the part you want to keep hidden/unaccessible, but the client must have access to the private key in order to use the certificate.
If you wanted this as safe as possible, you would generate unique public/private key pairs for each authorized client. The certificates would have to be delivered separately from a "common" installer, though you could set up a web service to deliver them during installation.
such key should be exchanged out of band - not via the web channel for the user at large. The .CRT is used mainly for certificates holding the public keys. The concern will be more of safeguarding the .KEY extension which is used both for public and private PKCS#8 keys. The keys may be encoded as binary DER or as ASCII PEM.
This file is only protected by the password and stored in the web server or equivalent only. There should not any need for other to access that .key file as it is supposed to be unique to the server using it to handle the SSL from clients. The clients only need the .CRT.. the only worries is hardcoding the password to "open" up the access to .KEY. Need to minimise footprint for the hardcoding as it can be easily reverse engineered unless you do some encryptor on the code as in ensuring the executable with hardcoded password will not be easily RE.
Likewise, also go for DHE or ECDH and CURL latest version so that .KEY is not in a vulnerable state for compromised via known CVEs vulnerability like the case of POODLE (weak cipher SSLv3/v2, mainly on clients, related to BEAST/BREACH/CRIME), FREAK (weak RSA export), LogJam (weak DHE export). There is POODLE "fixes" stated for libcurl 7.15.x (mainly is to enforce TLSv1, try to go for TLSv1.2 where possible if it does not break your apps)
With DHE/ECDHE, the session key is unique so getting the .KEY will not be able to compromised sniffed SSL traffic (pcap) for any offline decrypt...
As a whole the use of libcURL), secure coding using its SSL opt should be considered too, have the necessary authentication and password requested rather than hardcode if possible e.g. use of "CURLOPT_KEYPASSWD". Also some starter to explore further for developer include CURLOPT_CAPATH (to verify remote), CURLOPT_SSL_VERIFYPEER (to verify peers prior exchange) and CURLOPT_CERT & CURLOPT_SSLKEY (to specify Authentication cert) etc.
This file is only protected by the password and stored in the web server or equivalent only. There should not any need for other to access that .key file as it is supposed to be unique to the server using it to handle the SSL from clients. The clients only need the .CRT.. the only worries is hardcoding the password to "open" up the access to .KEY. Need to minimise footprint for the hardcoding as it can be easily reverse engineered unless you do some encryptor on the code as in ensuring the executable with hardcoded password will not be easily RE.
Likewise, also go for DHE or ECDH and CURL latest version so that .KEY is not in a vulnerable state for compromised via known CVEs vulnerability like the case of POODLE (weak cipher SSLv3/v2, mainly on clients, related to BEAST/BREACH/CRIME), FREAK (weak RSA export), LogJam (weak DHE export). There is POODLE "fixes" stated for libcurl 7.15.x (mainly is to enforce TLSv1, try to go for TLSv1.2 where possible if it does not break your apps)
With DHE/ECDHE, the session key is unique so getting the .KEY will not be able to compromised sniffed SSL traffic (pcap) for any offline decrypt...
As a whole the use of libcURL), secure coding using its SSL opt should be considered too, have the necessary authentication and password requested rather than hardcode if possible e.g. use of "CURLOPT_KEYPASSWD". Also some starter to explore further for developer include CURLOPT_CAPATH (to verify remote), CURLOPT_SSL_VERIFYPEER (to verify peers prior exchange) and CURLOPT_CERT & CURLOPT_SSLKEY (to specify Authentication cert) etc.
ASKER
In fact, here is another bit of detail.
The only reason certs are being used is because when the client/server project started, it was believed that the only way to protect the data between client and server was to use https and that certs had to be used when using https.
This isn't true but it seems to have stuck with us all this time. There is no need to hand out certs in our case, we simply hand the user some software which only communicates with our home server or they install a piece of hardware which does the same.
Seems certs aren't really needed in this case?
>Is the curl client using certificate authentication?
No, the certs are only used because it was thought that a cert is needed in order to encrypt the traffic between client/server.
Both the client and the server have matching certs and the certs paths are included in the curl command line.
>If the reason is saving money, then a regular client request over TLS will generate an error, because the
>self-signed certificate deployed on the server is not signed by a known CA.
No errors are generated because the certs are included on the device running the software.
>The easy way is to use curl with the -k or --insecure flag. This is like seting the CURLOPT_SSL_VERIFYHOST
>flag to false in libcurl.
Yes, the command does include the insecure flag.
The only reason certs are being used is because when the client/server project started, it was believed that the only way to protect the data between client and server was to use https and that certs had to be used when using https.
This isn't true but it seems to have stuck with us all this time. There is no need to hand out certs in our case, we simply hand the user some software which only communicates with our home server or they install a piece of hardware which does the same.
Seems certs aren't really needed in this case?
>Is the curl client using certificate authentication?
No, the certs are only used because it was thought that a cert is needed in order to encrypt the traffic between client/server.
Both the client and the server have matching certs and the certs paths are included in the curl command line.
>If the reason is saving money, then a regular client request over TLS will generate an error, because the
>self-signed certificate deployed on the server is not signed by a known CA.
No errors are generated because the certs are included on the device running the software.
>The easy way is to use curl with the -k or --insecure flag. This is like seting the CURLOPT_SSL_VERIFYHOST
>flag to false in libcurl.
Yes, the command does include the insecure flag.
> it was believed that the only way to protect the data between client and server was to use https
: https is just secure channel, the minimal that client/server design should have. the key requirement is end to end encryption which the data should also be encrypted (another layer rather than point to point).
> Seems certs aren't really needed in this case?
: True that you do not need to exchange certificate as the private key is your server and client just need that server ssl cert in the client trusted root store for one way SSL, which is a more common deployment. For mutual authentication, a more secure assurance, the client will present its ssl upon server request which the client has its private key and before that the public key will already be shared out. Most likely, and typically, the pub cert (or keys) shared out in known public store directory for user or server to get from including revoked cert list.
>Both the client and the server have matching certs and the certs paths are included in the curl command line.
: Each entity should have their unique cert even if they are self signed for mutual authentication. The sharing of single self-signed cert for client and server seems to defeat the purpose of protecting the data since the private key is at both end. That sounds to me more of "pre-shared passkey" scheme instead of the true pki certificate authentication or even the pgp keyring concept.
> No errors are generated because the certs are included on the device running the software.
: but as you mentioned the cert is on the device itself, I supposed they each have their unique self signed cert still..
> Yes, the command does include the insecure flag.
: Do consider the exposure as those threat exploiting ssl vulnerability is real, openssl is still being on patching regime by most. Go for TLS v1 minimally
: https is just secure channel, the minimal that client/server design should have. the key requirement is end to end encryption which the data should also be encrypted (another layer rather than point to point).
> Seems certs aren't really needed in this case?
: True that you do not need to exchange certificate as the private key is your server and client just need that server ssl cert in the client trusted root store for one way SSL, which is a more common deployment. For mutual authentication, a more secure assurance, the client will present its ssl upon server request which the client has its private key and before that the public key will already be shared out. Most likely, and typically, the pub cert (or keys) shared out in known public store directory for user or server to get from including revoked cert list.
>Both the client and the server have matching certs and the certs paths are included in the curl command line.
: Each entity should have their unique cert even if they are self signed for mutual authentication. The sharing of single self-signed cert for client and server seems to defeat the purpose of protecting the data since the private key is at both end. That sounds to me more of "pre-shared passkey" scheme instead of the true pki certificate authentication or even the pgp keyring concept.
> No errors are generated because the certs are included on the device running the software.
: but as you mentioned the cert is on the device itself, I supposed they each have their unique self signed cert still..
> Yes, the command does include the insecure flag.
: Do consider the exposure as those threat exploiting ssl vulnerability is real, openssl is still being on patching regime by most. Go for TLS v1 minimally
ASKER
I have to re-read your post and it is a little over my head.
As for having both keys at either end, the reason any keys ended up on the client is because they run autonomously and no one is there to say 'yes, accept the insecure certificate'.
As for having both keys at either end, the reason any keys ended up on the client is because they run autonomously and no one is there to say 'yes, accept the insecure certificate'.
Noted - Pardon for the lengthy prev post. the whole point is client has its unique private .KEY and server has its unique .KEY. They do not share that if you really into certificate based authentication and to established SSL or any form of secure exchange channel.
Another is using DHE or ECDH (diffie hellmen which SSH leverage based on host based authentication). Otherwise, it is preshared password authentication , analogous to SSH password based auth. There is passwordless scheme (not advocated though) using SSH where the key is already done thru one-time setup "preset" and defined before hand. This still adhere to using unique keypair. http://www.linuxproblem.org/art_9.html
Pardon if I misunderstood
Another is using DHE or ECDH (diffie hellmen which SSH leverage based on host based authentication). Otherwise, it is preshared password authentication , analogous to SSH password based auth. There is passwordless scheme (not advocated though) using SSH where the key is already done thru one-time setup "preset" and defined before hand. This still adhere to using unique keypair. http://www.linuxproblem.org/art_9.html
Pardon if I misunderstood
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SSL definitely not for authentication, it is just to setup secure link client/server - the certificate (or key file) represent their identity. So what we are doing is just to make sure that those identity can be trusted. Some practices, on top of the expert shared, I suggest the below too
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
a) CURLOPT_PINNEDPUBLICKEY to set pinned public key which enforce that the pub key must be used otherwise drop the case - this is really being specific even if there are any other past "legit" pub cert used in past by server...
b) CURLOPT_SSLVERSION - set preferred TLS/SSL version, you can go for CURL_SSLVERSION_TLSv1 which is to support TLSv1.x where x is 0,1 or 2. SSLv2 is disabled by default since 7.18.1. SSLv3 is disabled by default since 7.39.0.
c) CURLOPT_SSL_OPTIONS in specific in 7.25, there is CURLSSLOPT_ALLOW_BEAST when set tell will have libcurl to not attempt to use any workarounds for a security flaw in the SSL3 and TLS1.0 protocols
d) CURLOPT_SSL_VERIFYPEER - verify the peer's SSL certificate. Minimally this should be set so that verification fails to prove that the certificate is authentic, the connection fails. Also to avoid man in the middle attempts especially when you are dealing with self signed.
e) CURLOPT_SSL_VERIFYHOST - verify the certificate's name against host. You can certificate must indicate that the server is the server to which you meant to connect, or the connection fails. Simply put, it means it has to have the same name in the certificate as is in the URL you operate against.
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
a) CURLOPT_PINNEDPUBLICKEY to set pinned public key which enforce that the pub key must be used otherwise drop the case - this is really being specific even if there are any other past "legit" pub cert used in past by server...
b) CURLOPT_SSLVERSION - set preferred TLS/SSL version, you can go for CURL_SSLVERSION_TLSv1 which is to support TLSv1.x where x is 0,1 or 2. SSLv2 is disabled by default since 7.18.1. SSLv3 is disabled by default since 7.39.0.
c) CURLOPT_SSL_OPTIONS in specific in 7.25, there is CURLSSLOPT_ALLOW_BEAST when set tell will have libcurl to not attempt to use any workarounds for a security flaw in the SSL3 and TLS1.0 protocols
d) CURLOPT_SSL_VERIFYPEER - verify the peer's SSL certificate. Minimally this should be set so that verification fails to prove that the certificate is authentic, the connection fails. Also to avoid man in the middle attempts especially when you are dealing with self signed.
e) CURLOPT_SSL_VERIFYHOST - verify the certificate's name against host. You can certificate must indicate that the server is the server to which you meant to connect, or the connection fails. Simply put, it means it has to have the same name in the certificate as is in the URL you operate against.
ASKER
I closed it too quickly I guess.
When I use the curl command without the .key file it works fine but only if I use -k option.
Without the -k option,
* Cert verify failed: BADCERT_CN_MISMATCH (I know why this one)
* Cert verify failed: BADCERT_NOT_TRUSTED (I know why this one)
curl: (51) Cert verify failed: BADCERT_EXPIRED
Guess it means the self signed certs have expired. I thought I created long term ones.
When I use the curl command without the .key file it works fine but only if I use -k option.
Without the -k option,
* Cert verify failed: BADCERT_CN_MISMATCH (I know why this one)
* Cert verify failed: BADCERT_NOT_TRUSTED (I know why this one)
curl: (51) Cert verify failed: BADCERT_EXPIRED
Guess it means the self signed certs have expired. I thought I created long term ones.
curl used to include a list of accepted CAs, but no longer bundles ANY CA certs. So by default it'll reject all SSL certificates as unverifiable. Also the cert CN subject does not match the cURL domain you entered likely...suggest you open another qns if needed...
ASKER
Wait now, are you saying that even if I general new certs, curl will still require the -k option as the cert is not usable?
Hey,
My answer told you to create your own CA, but I did not tell you how to do it. Here are the instructions.
You start by assigning a computer to be your CA server. It can be a server, or your PC.
1. On the CA (once)
==========
Create a private key
Create the Root Certificate
2. On the servers to secure via SSL (on each server)
==========================
Create CSR for server
3. On the CA (for each server)
=========================
Sign CSR with root CA key
4. On the clients (on each client)
==========================
Install the CA Root certificate, and then use the proper option to point out this CA cert for verification when connecting.
for libcurl: CURLOPT_CAPATH
for the curl command line: --cacert [file]
See this link for the actual command reference, except you don't have to import the CA root certificate into the browsers' CA store.
http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/
My answer told you to create your own CA, but I did not tell you how to do it. Here are the instructions.
You start by assigning a computer to be your CA server. It can be a server, or your PC.
1. On the CA (once)
==========
Create a private key
Create the Root Certificate
2. On the servers to secure via SSL (on each server)
==========================
Create CSR for server
3. On the CA (for each server)
=========================
Sign CSR with root CA key
4. On the clients (on each client)
==========================
Install the CA Root certificate, and then use the proper option to point out this CA cert for verification when connecting.
for libcurl: CURLOPT_CAPATH
for the curl command line: --cacert [file]
See this link for the actual command reference, except you don't have to import the CA root certificate into the browsers' CA store.
http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/
you need a CA not an -k. The option -k (or --insecure) is insecure and not advocated as we stated in the posts. Likewise not to disable both CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST. These all disable of options makes all SSL connections considered "insecure" although using them avoid the error. Regardless, unless you use those unsafe options, all SSL connections are attempted to be made secure by using the CA certificate bundle installed by default.
For self signed, see https://turboflash.wordpress.com/2009/06/23/curl-adding-installing-trusting-new-self-signed-certificate/
For self signed, see https://turboflash.wordpress.com/2009/06/23/curl-adding-installing-trusting-new-self-signed-certificate/
ASKER
I see mentions of CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST but are you explaining that these are options in the curl command?
The -k option was always used while testing because the certs were/are out of date. The plan was/is to use self signed because it is a closed application and not something where users need to have these certs, only the app needs it and is self contained.
Other than using -k because of out of date certs, it sounds like things are set up right then?
-We don't need to have the .key on the remote app, only the .crt.
-We don't want to use -k once the certs are up to date.
-We need to make sure the matching .crt and .key certs on the server, which we do.
However, still, even after all this info, not clear on how the certs are used in terms of encrypting the traffic between client and server. Most information given is way over my head and need someone to explain this in a nice easy for a beginner to understand way.
The -k option was always used while testing because the certs were/are out of date. The plan was/is to use self signed because it is a closed application and not something where users need to have these certs, only the app needs it and is self contained.
Other than using -k because of out of date certs, it sounds like things are set up right then?
-We don't need to have the .key on the remote app, only the .crt.
-We don't want to use -k once the certs are up to date.
-We need to make sure the matching .crt and .key certs on the server, which we do.
However, still, even after all this info, not clear on how the certs are used in terms of encrypting the traffic between client and server. Most information given is way over my head and need someone to explain this in a nice easy for a beginner to understand way.
those in libcurl. you call the options via curl_easy_setopt
Link is as per my last few postings http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
self-signed also need CA cert, and this case is itself. option "-k" just skip check. see http://curl.haxx.se/mail/archive-2011-02/0046.html
if need to suggest you have another question then
Link is as per my last few postings http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
self-signed also need CA cert, and this case is itself. option "-k" just skip check. see http://curl.haxx.se/mail/archive-2011-02/0046.html
if need to suggest you have another question then
Typically, when you use self-signed certs you do it for one of 2 reasons: saving money, or using certificate authentication.
If the reason is saving money, then a regular client request over TLS will generate an error, because the self-signed certificate deployed on the server is not signed by a known CA.
The easy way is to use curl with the -k or --insecure flag. This is like seting the CURLOPT_SSL_VERIFYHOST flag to false in libcurl.
A bit better way is to save your self created intermediate CA certificate into the curl intermediate store.
If you must use a client certificate for client authentication, then there I don't know an alternative to storing the certificate on disk.