How to set a TLS connection just for a specific domain when email is sent in Exchange 2010?

If we have a spam mail filter server you mean the TLS connection should be configured from the mail filter?

if exchange routes all outbound emails to the spam filter and the latter sends to the internet then yes the connection to your client will be initated by the spam filter and not from exchange, you should check with your spam filter on how to do this not on exchange

Just for my understanding why we can't do on Exchange when we have a mail filter? Is there any conflict?

it is not a question of conflict,
what you want is open a TLS with your CLIENT when your exchange is only connecting to your MAIL RELAY and your MAIL RELAY to your client

so connection is

Exchange ---> Mail Relay
Mail Relay ---> Client

so technically there is no connection between your Exchange and your Client.

Unless you want to open direct connection from exchange to your client and bypass your mail relay the setting should be done on your mail relay

If it is not clear please let me know

thanks

If I choose to create a separate send connector in Exchange for the specific client domain which smart host I should put?

unfortunately it is not as easy as this you should first give directaccess to ecchange for the SMTP protocol to the internet
second make sure that the public IP used by exchange for sending email is in your PTR records

 if you are willing to do all this then there is no need for the Smart host you can keep the connector configuration to Route by MX record

How can I know if my Exchange already sent by TLS if others MX server accept TLS connection?

The default of exchange is to use tls (as well as all new mail servers) it is called opportunistic tls...

However you can force exchange to use tls for a specific domain I sent the how-to before

Ok so by default Exchange 2010 try to send with TLS? Could you give a doc or link that explain that?

Thank you very much for your help!