How do I allow specific email addresses to send to a exchange POP\SMTP mailbox?

Running E2K3 on Server 2K3 R2.  I have created a user with a SMTP email account only.  All my other domain users have exchange accounts.  (I did this for reasons that would take too much space to explain, can't use OWA).  I want only three internal users to be able to send email to this address, then about five (this number will grow) external addresses (AOL, etc) to be able to send to the address.  We have our own domain, mx record etc, so I am not working though an ISP.  

In delivery restrictions, I checked the ONLY FROM box, and added the internal addresses.  For the external addresses, I created contacts for each address, then added those to the ONLY FROM box.  Internal works fine, users not added get a security message when they try to send, allowed users can send.  Problem is the external users get the following message, or something similar to it, I have tried several accounts:

This is an automatically generated Delivery Status Notification

Delivery to the following recipient failed permanently:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 sorry, no mailbox here by that name. (#5.1.1) (state 14).

If I change the restrictions to FROM EVERYONE, all external mail gets through without issues.

This is the first time I have setup an internal POP\SMTP mail box, so I could be doing any number of things wrong.  Any suggestions appreciated.
Thank you,
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

The error that you have posted is NOT an Exchange error. Does your email come in through another host or service? Exchange never says "State 14".

As far as I am aware, restricting the account to receive email from certain external addresses isn't possible. I certainly have not had it working correctly.

bobgrafAuthor Commented:
My email does not go through another host or service.  I have a registered domain and an MX record.

I now took all restrictions off this address.  External senders get the same message.  Let's say I do not want to block any external messages.  Also, I can send mail from this account to external addresses, but cannot receive.
It isn't an Exchange message - so something else is blocking the message. Could be AV, Antispam a firewall doing SMTP scanning, something like that. Something else is getting in the way.

Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

bobgrafAuthor Commented:
The external email I'm testing from is a GMAIL account.  I made sure on our spam server, and anti virus to whitelist the GMAIL domain.  Then went to my gmail account and sent two emails:

1.  to (this is an exchange account on our exchange server)
2.  to  (this is a POP/SMTP account on our exchange server)

The JDOE came through in seconds, no return emails in my gmail account.  The JOHND bounced back immediatly to my gmail account.  

I know you state this is not an exchange message, but I'm confused as to why the GMAIL account gets to one internal address and not the other.  Would it make more sense to post this to a different Zone?
So you have a spam server. Where I stated whether you have another host or server, that doesn't mean just internet services. Unless Exchange is directly exposed to the internet then the email is going through something else.
Does that antispam server do some kind of recipient validation? That is where I would be looking at for the cause of the problem.

bobgrafAuthor Commented:
I could not get the email box to work with the desired results.
Created a gmail account, and most of what I needed was completed.
Thank you

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Protocols

From novice to tech pro — start learning today.