Link to home
Start Free TrialLog in
Avatar of Firebladeboy1993
Firebladeboy1993

asked on

OpenSSL - Decrypt padding errors - PKCS versus -Raw

Hi All,

I am receiving an encrypted file and it's key from a partner. The Key has itself been encrypted using our Digital Certificate Public Key.

When I attempt to decrypt the key using the following and our private key, I get a padding error as shown below:

C:\openssl rsautl -decrypt -in xxxx_Key -inkey xxxxprivatekey.pem -hexdump -out aeskey.txt Loading 'screen' into random state - done RSA operation error 5612:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding er ror:.\crypto\rsa\rsa_pk1.c:273: 5612:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:.\ crypto\rsa\rsa_eay.c:602:

If I add the -Raw switch to the decrypt, it appears to work but the resulting hexdump of the key is WAY larger than I'm expecting - 512 characters in fact.  Can anyone offer advice as to what may be going on here? Thanks!
Avatar of btan
btan

Public-key crypto is not for encrypting arbitrarily long files. One uses a symmetric cipher for normal ops. Most of the time (which I shared in my prev post likewise on the steps) since we cannot use directly encrypt a large file using rsautl. Normally it is like the following:
1.Generate a key using openssl rand, eg. openssl rand 32 -out keyfile
2.Encrypt the key file using openssl rsautl
3.Encrypt the data using openssl enc, using the generated key from step 1.
4.Package encrypted key file with the encrypted data. Whoever received that package need to decrypt the key with their private key, then decrypt the data with the resulting key.
Probably has to ask the sender if that is what is done or it is differing ...
Avatar of Firebladeboy1993

ASKER

Hi,  Yes, it's exactly that process I'm attempting.  I send encrypted files just like that, and they other party responds the same way.  The problem I'm having is in decrypting the response.  As a test, I encrypted and decrypted a Key successfully using my Public and Private key.  When I try to do the same with the Key they send me however (which they have theoretically encrypted with my Public Key for me to Decrypt with my private i.e. the same process) I get the packing error.

I suppose I am looking for something they may have done during Encryption that could cause my decrypt to fail in this way.
Precisely, you should then share with them your scheme and commands used to be in sync, any diff from sender side will cannot be surfaced from your side..
ASKER CERTIFIED SOLUTION
Avatar of btan
btan

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
In fact I'm doing a variation of that and it's working fine in one case and not for another.  It seems there's nothing wrong with what I am doing but rather, a problem particular to one of the cases I'm dealing with, perhaps something to do with the Certificate the partner holds.  Thanks for the information on this btan.
thanks for sharing