Email not being sent by smarthost

I have an exchang 2003 server setup that has been running wihtout problems for a few years. Just this week, I changed the connector to forward mail to a smart host. This is a computer on my "optional" network as a Watchguard firewall would term it which has all ports opened between them on the firewall. Most email is being sent without problems, but I have a few that are either sitting in the queue on the smarthost (C:\Inetpub\mailroot\Queue). I have a user who got a NDR stating that his email was returned because "You do not have permission to send to this recipient".

I dont understand if that error message is coming from my side or theirs.

If I look in the event viewer on the smart host, I see a ton of smtpsvc notices that say: "Message delivery to the host 'x.x.x.x' failed while delivering to the remote domain 'x.com' for the following reason: The connection was dropped by the remote host."

What might cause this connection to be dropped?

I have run NSLOOKUP, SMTP DIAG, and checked to ensure I was setup for reverse dns. All came back saying all is OK. Both nslookup and smtp diag successfully translated the address and I am also able to
telnet to mail.domain.com 25

I dont understand enough about what may have happened while sending to the smarthost. Most email is being sent fine, but just a few are having problems.

Thanks.