Require TLS Encryption Option Greyed Out of IIS 6 SMTP Relay Outbound Security Interface


I’m trying to enable TLS authentication on an IIS 6 SMTP relay virtual host, without success.  Although I’ve installed a self-signed cert, the option to “Require TLS Encryption” is greyed out.

To review, here are the steps I took.

1.)      Using IIS 7, I created a self-signed cert
2.)      I exported this cert to a folder on the server
3.)      I imported the cert into the Personal Cert store
4.)      I then tried to fully enable TLS but, the option to “Require TLS Encryption” is greyed out, suggesting the cert is either incorrectly formed, or, the virtual relay host does not see it.

Visuals of the steps I took...

Step One. Create Self-signed Cert via IIS 7 Manager

create self signed cert windows 2008

Step Two. Export the cert

export the certificate

Step Three. Import the Cert into the Personal Certificate Store

import the cert

Step Four. Secure IIS SMTP Virtual Host Using TLS

Require TLS option greyed out
Who is Participating?
Chris HConnect With a Mentor Infrastructure ManagerCommented:
Quote from here:

'When the IIS6 SMTP Server module looks for a certificate to use for TLS encryption, it seems checks the 'Local_Machine\my' store. I'm not sure where the OpenSSL certificate was placed by the system, but if it's not in the Personal certificates section, then the SMTP server won't be able to find it, and will therefore provide the error listed here.'
drmonoeAuthor Commented:
Yes, I found that during my Google searches but it wasn't immediately helpful.

So I suppose the next question is: how does Local Machine/My Store correspond to a drive location? Or does this refer to the HKLM reg hive?

I'll take a look in the reg to see if it reveals a location.
Chris HInfrastructure ManagerCommented:
It refers to the certificate store.  You need to click start, type mmc , push enter.

file, add/remove snap-in.

add certificates

local computer

then look at picture as to what it's referencing.  (local machine should be local computer)

My store is personal
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

drmonoeAuthor Commented:
Yes, overnight I reached that conclusion and added the cert to the correct store.

Cert Store
Unfortunately, this hasn't made the TLS encryption option available.  I've also tried resetting the virtual SMTP host individually and even an iisreset with no success.
Chris HInfrastructure ManagerCommented:
Investigate the registry keys this article says to disable TLS in IIS.  I'm wondering if an admin before you disabled.
Chris HInfrastructure ManagerCommented:
Also, maybe unrelated/non-correlated (but I've seen MS do worse).  Check your ie options listed here:
drmonoeAuthor Commented:
Hmmm, interesting.

I'll check those out and get back to you.  Thanks for the leads.
drmonoeAuthor Commented:
Unfortunately, none of those steps applied to this situation.  Truly a mystery as to why IIS is failing to see the cert (which seems to be properly installed and verified).
Chris HInfrastructure ManagerCommented:
Lol, it's not expired is it?  (had to ask)
drmonoeAuthor Commented:
Ha! That's a fair question (crossing all the T's, etc).

I'm sure the cert is still valid.  It was only issued a few days ago and expires a year from now.  Also, I confirmed it using a tool named SSLDiag.

Oh and I successfully used it to secure the default website with SSL.

So aside from the SMTP relay, all else seems good.
drmonoeAuthor Commented:
Yup, I verified that too.  

The cert's subject name definitely matches the virtual server's FQDN.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.