When I encrypt and then decrypt a key using OPENSSL and Public and Private keys extracted from a Certificate, I get PKCS1 padding error

Hi All,

I am using some command line Open SSL commands to encrypt and decrypt data using Public and Private keys extracted from a Digital Cert.   When I try to decrypt I get PKCS padding errors.  Can someone tell me where I'm going wrong?

These are the command I've been using:

a) Extract Public key:  openssl x509 -pubkey -noout -in xxxxx.cer  > xxxxxpublickey.pem
b) Extract Private Key:openssl pkcs12 -in xxxxxx.pfx -nocerts -out xxxxxprivatekey.pem -nodes
c) Encypt a key (.bin file): openssl enc -aes-256-cbc -in kenkey.bin -out kenkey_Key -pass file:xxxxxpublickey.pem
d) Decrypt key produced in c) openssl rsautl -decrypt -hexdump -in kenkey_key -inkey xxxxxprivatekey.key -out aeskey.txt

This produces errors like this:
RSA operation error
3248:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:.\crypto\rsa\rsa_pk1.c:273:
3248:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:.\crypto\rsa\rsa_eay.c:602:

I'm still figuring out OpenSSL and encryption so I'm sure I'm doing something stupid.  Any help anyone can provide would be greatly appreciated.


Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Is the original file complete and not damaged?
btanExec ConsultantCommented:
Most of the time for such "RSA_padding_check_PKCS1_type_2 error ..." - you tends to see this with
(1) Encoding errors: instead of decrypting binary data, decrypting is done on (maybe) Base64 encoded data.
(2) Mismatched key pair or key itself: Public key does not matched to the Private key for decryption.

Maybe we can make sure the pair is not mismatch (2) too before saying the cipher load is not correct (1). Like below  in ref to https://www.sslshopper.com/ssl-converter.html

Convert PEM to DER: openssl x509 -outform der -in certificate.pem -out certificate.der
or if the cert is already in "der" format, can also be turn into "pem"
e.g. Convert DER to PEM: openssl x509 -inform der -in certificate.cer -out certificate.pem

Convert PEM to PFX: openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt
or if there is only "pfx" then can get "pem"
e.g. Convert PFX to PEM:  openssl pkcs12 -in certificate.pfx -out certificate.cer -nodes

After making sure we have the "pem", we can try the encrypt and decrypt as stated in

e.g. 1) openssl enc -base64 -d -in <file name> -out <decoded file name> where <decode file name> was created and it had binary contents.
e.g. 2) openssl rsautl -decrypt -inkey <privkey.pem> -out <output symmetric key file name> -pkcs
but in this case, consider trying to use -raw instead of -pkcs to decrypt data with server private

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Apache needs PEM (base64 - text file)
Managing Security & Risk at the Speed of Business

Gartner Research VP, Neil McDonald & AlgoSec CTO, Prof. Avishai Wool, discuss the business-driven approach to automated security policy management, its benefits and how to align security policy management with business processes to address today's security challenges.

btanExec ConsultantCommented:
will be easier to work with pem as you can also check the B64 textual content in the file - at least to me some sort of "assurance". But for IIS, they need the pfx so the convertor online I shared in the prev post link in sslhopper will be handy or use of openssl..but do avoid having the actual pfx to go through online conversion since it will gives trace of the upload files.

.or you can use the digicert tool if you are exporting the keys out from installed cert e.g. How to Export Your SSL Certificate w/Private Key Using the DigiCert Certificate Utility
This exports the following files that you need to copy to your Apache server:

Private Key: your_domain_com.key
Server Certificate: your_domain_com.crt
Intermediate Certificate: DigiCertCA.crt
Firebladeboy1993Author Commented:
incidentally,  c)  Was incorrect on this.  I should have encrypted as I decrypted, using  RSAUTL. What I am inadvertently doing in c) was trying to encrypt using a password from the Public key. Not my intent.
btanExec ConsultantCommented:
thanks for sharing - if I can sum it as an example below. This is the norm for keypair (asymm) to protect file encryption key (symm) and then use file encryption key (symm) to encrypt the actual file (payload). It is faster to use symm key for huge payload ...hope this help

>0 (asymm keypair)
openssl rsa -in id_rsa -outform pem > id_rsa.pem
openssl rsa -in id_rsa -pubout -outform pem > id_rsa.pub.pem

>1(symm key)
(generate an aes symm key to be use for encrypt)
openssl rand -base64 32 > key.bin

>2(protect symm key)
(using rsa pub key specifically therefore rsautl used to encrypt aes symm key)
openssl rsautl -encrypt -inkey id_rsa.pub.pem -pubin -in key.bin -out key.bin.enc

(using aes symm key specifically therefore enc used to encrypt the FILE)
openssl enc -aes-256-cbc -salt -in SECRET_FILE -out SECRET_FILE.enc -pass file:./key.bin

>3 (get back the symm key from the protected ver in -2, then use it to decrypt FILE encrypted in -2)
(using rsa prv key specifically therefore rsautl used to decrypt aes symm key)
openssl rsautl -decrypt -inkey id_rsa.pem -in key.bin.enc -out key.bin

(using aes symm key specifically therefore enc used to decrypt the FILE)
openssl enc -d -aes-256-cbc -in SECRET_FILE.enc -out SECRET_FILE -pass file:./key.bin
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.