Outlook 2010 - W 2008 R2 set to encrypt e-mail aes 256 but encryption is always 168 bits 3DES

I am thinking we should clear up the ssl certificate in the personal store so that it will not be using an old certificate hence leading to wrong SMIME certificate, not supposed to. Otherwise, for GAL replacement will also ask to clear first to remove your existing published settings. I am thinking also if the machine support that level of cipher. May use IISCrypto to see and set the cipher to be  supported.

Regardless, the once certificate is installed, the usual setting applies

Part 2 of 4: Updating Security Settings to link the S/MIME certificate to Outlook profile
1. Launch Outlook 2010.
2. Select: File | Options
3. Select: Trust Center | Trust Center Settings
4. Select: E-mail Security.
5. Click on the settings button.
6.  Enter the below settings:
Security Setting Name:  Give the security setting a name. This is just a label.
Cryptographic Format: S/MIME should be selected.
Signing Certificate: Select your secure email certificate – click CHOOSE and select your certificate from the list.
Encryption Certificate: Select your secure email certificate – click CHOOSE and select your certificate from the list.
6. Click OK.  

I had this problem on another platform--- I'm not the expert here but maybe if I relate my case it will help you.----- I ran a CA -- and published certificates with 4096 bit SHA 512 and it would keep reverting to SHA1 n 1024 bit..--- that did not change until a new update was available, after I upgraded the system my initial settings showed up.

In other words it will not take unless your software is updated to address the higher encryption method.

Hi btan, I don't understand the "clear up the ssl certificate in the personal store" part.  Which "old" certificate could be found? The certificate used for the actual encrypted e-mail (3DES) is the right certificate as far as I can see in the e-mails properties.

Then again....I thought that NOT the certificate encrypts, but Outlook or OS will???  But correct me if I'm wrong......is the certificate deciding/determining which encryption can be used??

I'll check with IISCrypto btw....didn't think of using that angle yet.

The smime certificate should be installed in personal store and in the steps shared earlier chosen that same cert in the email setting. The cipher would have shown the supported one expected.

Outlook check for smime cert then use it to encrypt using the recipient smime public key and sign using your own smime private key.

I suspect the Outlook 2010 client yet to support sha 512. Another is using without Cached Exchange Mode. No workaround solution specifically..
https://social.technet.microsoft.com/Forums/ie/en-US/2c91188d-0a10-4fe9-89da-c6b16fa25232/how-to-get-aes-256-encryption-using-smime-with-outlook-2010?forum=outlook

Hi,

Tested this weekend without Cached Exchange mode : Same result....

So...when I choose the S/MIME security settings the first time....I choose the (our) certificate and when I do the "hash-algorithm" is set to SHA1 and the Encryption algorithm on AES 256.

When i send an e-mail to an external recipient ...who also has a certificate in "Contacts" the mail is being sent, signed AND encrypted but..... 168 Bits 3DES.

Help.......!

I need to get clear what's the cause. Our certificate, Recipients certificate, Outlook, Windows, Else....?


When I run IISCRYPTO it starts default with all settings "grey". I asume that if it's marked it's active?
Any tips regarding the use are welcome ....

Thanks in advance

Also ran listalgs.exe >
The screenshot is attached. What I want seems missing? (Not sure) But if.....how do I add that to the machine???
Algortithms_Few.png

When you are encrypting email to be sent to the recipient, you will be using the recipient certificate and if you are signing the email, then you be using your own local certificate. May need to test for signed only email and encrypted and signed email and encrypted email only. At least we can see if there are any differences.

For IISCrypto, you need to run with Administrator rights

IIS Crypto requires administrator privileges. If you are running under a non-administrator account, the GUI version will prompt for elevated permissions. The command line version must be run from a command line that already has elevated permissions.

@btan : thanx for the info!

In addition > i thought I'ld start over/clean....... deleted the published certificate in AD > New outlook profile......Internet Email this time, no Exchange.

What stands out here . I cannot Publish the certificate to the GAL anymore....... because of Internet Mail  >< Exchange Mail


Regarding IISCRYPTO > Are those settings for encrypting emails as well? Seems to be for SSL only?? (IIS)


What "properties" should the receivers public part need so outlook can encrypt AES 256?  

I can try to force the use of AES with GPO, but probably will get the error that Outlook is gonna use a different......bla bla...

I want to get clear WHY in this case it's just not functioning. Why is Outlook just sending all mails encrypted with 168 Bits 3DES?

Better to check with GPO as I suspect if there is a GPO at domain and a one at local policy hence the conflict that we see. The domain GPO will take precedence.

Hi All,
I've tested about all of your suggestions but still no solution. Nothing changes...sent mail is still 168 Bits 3DES.

So...Cached Exchange mode > no change (I removed the published to GAL ....can't publish it again while in non-cached mode) ...but it didn't make a difference at all......
I removed all of the certificates from the recipient, removed him as contact, re added via a received e-mail.
Then certificate is automatically added. No change.
Removed certificate again...imported in many ways....no change.

I did the gpo-stuff BUT without GPO since I didn't want to mess up the machine ... > Edited the registry instead.....which should lead to the same result!?
What i set was >
1.
 HKEY_CURRENT_USER\software\policies\microsoft\office\14.0\outlook\security > REG_DWORD adminsecuritymode set to "3" > listen to GPO (or the settings that it can alter by GPO)
2.
 HKEY_CURRENT_USER\software\policies\microsoft\office\14.0\outlook\security >REG_DWORD  minenckey set to 256 instead of the default 40
2.

I thought that maybe Outlook would now say that it would use less secure encryption with this setting, but nothing changed.
Maybe I need to set some more.....????

I also found an article stating that  >
"For unknown reason Microsoft has changed the behavior of Outlook 2010 on the encryption method. If there is no SMIME capabilities attribute in the user certificate Outlook will use an RC2 40bit encryption without any warning to the sender or recipient."
Linke here

I'm sure the recipients certificate has/is S/MIME but still........ is this some "bug"?

Any more ideas are welcome!

Thank you all in advance.

Sound like the case f a bug like the past case on the mentioned RC2, which is supposed to be fixed in this hotfix package (https://support.microsoft.com/en-us/help/2475877/description-of-the-outlook-2010-hotfix-package-outlook-x-none-msp-febr)
The certificate supposedly to have

- restricted with a Key Usage extension, then digitalSignature and/or nonRepudiation must be allowed explicitly in that extension.
- contains an Extended Key Usage extension, then it must contain either the id-kp-emailProtection OID (1.3.6.1.5.5.7.3.4) or the special all-purpose anyExtendedKeyUsage OID (2.5.29.37.0).
- contains extension with id_kp_client_Auth (1.3.6.1.5.5.7.3.2), the latter is for the SSL login on website.

There isnt other good means except to raise a MS support case. Meanwhile will look out on similar sharing in public

Hi btan,

First off....thanks for you fast replies.
I tested one more thing.
I reapplied the registry tweaks with 1 additional setting >

suitebmode

Details here

Now I get a new error when sending an e-mail >

The error "this message cannot be sign and or encrypt because your certificate is invalid" makes me wonder if there may be something wrong wit the recipients certificate after all?  Or is this now related to my own certificate (signing part).

I set everything back as it was and can mail again....encryption back to the default 3DES....

If this error somehow triggers you or someone else to maybe a new angle to a solution.........shoot!!!

Hi all,

I solved the problem after weeks of searching and testing all proposed solutions. The actual problem was.....in the recipients certificate!
Altough the certificate did meet the requirements, recently the receiving party reissued their Signing certificate WITHOUT any property regarding 3DES. Somehow ....when you add the contact from a received signed e-mail, Outlook sets/saves some settings regarding the encryption that this certificate can handle. And....the worst.......Outlook then decides to use the lowest :-(

After receiving the new signature certificate (bu a signed e-mail) i removed the existing Contact and re-added it from the new e-mail. After this Outlook did exactly what was expected.......AES256!!

So....all testing/config changes......had no effect...... :-(   On the other hand it made clearer that there must be something wrong with the receiving end's certificate. (I also tested with a new, other contact and certificate and that worked directly......)

So ...for eliminating/pointing towards the right conclusion BTAN assited me the best. Thanx again!

Solved by not giving up finding..............................