OpenSSL - Decrypt padding errors - PKCS versus -Raw

Hi All,

I am receiving an encrypted file and it's key from a partner. The Key has itself been encrypted using our Digital Certificate Public Key.

When I attempt to decrypt the key using the following and our private key, I get a padding error as shown below:

C:\openssl rsautl -decrypt -in xxxx_Key -inkey xxxxprivatekey.pem -hexdump -out aeskey.txt Loading 'screen' into random state - done RSA operation error 5612:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding er ror:.\crypto\rsa\rsa_pk1.c:273: 5612:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:.\ crypto\rsa\rsa_eay.c:602:

If I add the -Raw switch to the decrypt, it appears to work but the resulting hexdump of the key is WAY larger than I'm expecting - 512 characters in fact.  Can anyone offer advice as to what may be going on here? Thanks!
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

btanExec ConsultantCommented:
Public-key crypto is not for encrypting arbitrarily long files. One uses a symmetric cipher for normal ops. Most of the time (which I shared in my prev post likewise on the steps) since we cannot use directly encrypt a large file using rsautl. Normally it is like the following:
1.Generate a key using openssl rand, eg. openssl rand 32 -out keyfile
2.Encrypt the key file using openssl rsautl
3.Encrypt the data using openssl enc, using the generated key from step 1.
4.Package encrypted key file with the encrypted data. Whoever received that package need to decrypt the key with their private key, then decrypt the data with the resulting key.
Probably has to ask the sender if that is what is done or it is differing ...
Firebladeboy1993Author Commented:
Hi,  Yes, it's exactly that process I'm attempting.  I send encrypted files just like that, and they other party responds the same way.  The problem I'm having is in decrypting the response.  As a test, I encrypted and decrypted a Key successfully using my Public and Private key.  When I try to do the same with the Key they send me however (which they have theoretically encrypted with my Public Key for me to Decrypt with my private i.e. the same process) I get the packing error.

I suppose I am looking for something they may have done during Encryption that could cause my decrypt to fail in this way.
btanExec ConsultantCommented:
Precisely, you should then share with them your scheme and commands used to be in sync, any diff from sender side will cannot be surfaced from your side..
SolarWinds® IP Control Bundle (IPCB)

Combines SolarWinds IP Address Manager and User Device Tracker to help detect IP conflicts, quickly identify affected systems, and help your team take near instantaneous action. Help improve visibility and enhance reliability with SolarWinds IP Control Bundle.

btanExec ConsultantCommented:
This link has been useful
When doing symmetric encryption, openssl enc, the command-line tool, uses its own non-standard, OpenSSL-specific format which does not match what SOAP uses.

...The steps are:
1.Copy Encrypted Session Key to a file and base64 decode it at same time

echo "TiMPCLfQgfw==" | base64 -d -i > sessionkey.enc
1.Decrypt session key using openssl and privatekey:

openssl rsautl -decrypt -in sessionkey.enc -out sessionkey.dec -inkey myprivatekey.key
1.Copy Encrypted Session Key to a file and base64 decode it at same time

echo "1qsIPulqkVQ3==" | base64 -d -i > body.enc
1.Read your session key in hex (am sure there's an easier way to do this but here's one way using hexdump unix command):

hexdump -C sessionkey.dec

And let's say it looks like this (this too short a key for aes256 but is just for demonstration purposes):
00000000  8a bc f2 86 25 4c 76 c4                           |....%Lv.|

1.Read your body in hex (again need to script this up):

hexdump -C body.enc

and let's say it looks like this:
00000000  80 33 28 0f 7a fa 73 f3  7e 4a 93 e6 a7 5f fe 2d  |.3(.z.s.~J..._.-|
00000010  29 88 84 4e e5 23 33 b8  0e 1a 43 fb 43 b1 83 ce  |)..N.#3...C.C...|

The first block (80 - f3) is the encryption iv
1.Decrypt the message using the hex version of the key and the hex version of the iv:

openssl enc -d -aes-256-cbc -in body.enc -K 8abcf286254c76c4 -iv 8033280f7afa73f3 -out body.dec

Note you will get some garbage at the beginning as you should strip off the iv from the beginning of the body before decrypting.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Firebladeboy1993Author Commented:
In fact I'm doing a variation of that and it's working fine in one case and not for another.  It seems there's nothing wrong with what I am doing but rather, a problem particular to one of the cases I'm dealing with, perhaps something to do with the Certificate the partner holds.  Thanks for the information on this btan.
btanExec ConsultantCommented:
thanks for sharing
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.