TLS and SMTP Bridgehead Server not working right

I have an Exchange 2003 environment.  I have several routing group connectors set up to handle various connections. One is designed just for sending email to the Internet. The others are for connections to partner email servers.  All use the same SMTP Virtual Server.

I want to implement TLS to our email filtering service. I have had TLS working on outgoing emails for almost a year since the outgoing is controlled by the connector and TLS is set on the connector.  Our partners do not support TLS.  Now I wish to implement TLS incoming from the email filtering service.

Here's what I have done.  I have created a second SMTP Virtual Server and I have checked in the Authentication area for it to use TLS. I have applied our SSL cert to the VS as well. I then went to the routing group connector for the Internet email connection and I removed the Default SMTP VS and added in the new one. I then bounced both SMTP VS's.  Initially I still saw outgoing email being processed through the old VS. I reinstalled the old VS into the connector, waited for the email to clear and then removed it once again, bouncing the SMTP VS's between each change. Finally I saw outgoing emails being processed through the new SMTP VS.

Then we started our testing by enabling TLS on the email filtering service.  This failed. We could see TLS activity in Wireshark, but for whatever reason, it wasn't being processed into the mailbox. So we played around with it  and never figured that out. Anyway, we set the email filtering service back to a setting where TLS was preferred, but could send it by SMTP.  Then we started getting in emails via just plain old SMTP.  .  

Now, if I have the connector pointed to the SMTP VS that is set only to do TLS, how are we getting in regular SMTP emails?  

Subsequently we decided to just leave it for now and work on it later.  We found then that we could no longer process outgoing emails.  I checked the queues in the Exchange server and could see that outgoing emails were queued on the old SMTP VS, which is NOT any longer specified in the connector as the one to use.  I added back in the old VS so that both are in the connector and now emails are going both directions again.  Incoming TLS still doesn't work, but whatever...

So a second question is how are outgoing emails falling into the old VS when only the new one was specified in the connector?  Is there some other service that I should have bounced?  At one point we did bounce the SMTP Service.  I really don't want to reboot the entire server.  Any thoughts are appreciated.