I've got a SBS2003 machine here that is having trouble with encrypted email. When the client sends the STARTTLS command Exchange replies with a 220, the client then sends the second EHLO, then the Exchange server drops the connection. I have tried this by connecting to exchange and typing the commands manually myself with putty and using ethereal to capture the mail stream of other mail servers connecting to our problem server. If exchange replied with a 550 I suspect the remote servers would fail back to un-encrypted, but since it's sending 220 ALL OK then dropping the connection, the remote servers don't fail back to unencrypted and we lose mail.
I've tried re-creating the self signed certificate using the sbs wizards with no success.
Full log below.
220 mail.problem-server.com.au Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at Tue, 10 Apr 2012 16:55:26 +0930
250-mail.problem-server.com.au Hello [203.xxx.xxx.39]
250-X-EXPS GSSAPI NTLM LOGIN
250-AUTH GSSAPI NTLM LOGIN
220 2.0.0 SMTP server ready
<connection closed by remote host>