Messages hang in the SMTP queue in Exchange System Manager - An SMTP protocol error has occurred

How do I disable application layer filtering of SMTP traffic on my PIX firewall from the PDM?  I need to do this in order to fix a problem where outbound emails through Microsoft Exchange 2003 sits in the SMTP queue and then comes back with an error message, "An SMTP protocol error has occurred."

Some of the email goes out fine, but depending on the domain name, it may or may not sit in the queue and never get delivered.
cullendreaAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lrmooreCommented:
With a PIX, there are a couple of issues that can cause this. Assuming you are running PIX 6.3x....
1 - smtp fixup. Did you disable this?
  no fixup protocol smtp
2 - outbound IP address. Is your exchange server's outgoing IP address the same as the inbound/MX record IP address?
 Example YES:
 static (inside,outside) <public ip> <private ip> netmask 255.255.255.255
 Example NO:
 global (outside) 1 interface
 nat (inside) 1 0 0 0
 static (inside,outside) tcp <MX public ip> smtp <private ip> smtp netmask 255.255.255.255

In the first example, all traffic both inbound and outbound from the server is bound to the same public IP. In the second example, inbound smtp traffic comes in to the MX public ip, but outbound email goes out as a dynamic xlate using the global ip address. Some upstream servers will not accept email coming from a different IP address than it finds with a MX record lookup for your domain.

3 - DNS fixup. Did you increase the dns size from default 512 to something bigger like 1024?
  fixup protocol dns maximum-length 512  <== Default
  fixup protocol dns maximum-length 1024 <== change to this

4 - If you are not using the latest PIX 6.3x OS, there are some bugs in the fixups. 6.3(1) is particularly buggy. Suggest updating to at very least 6.3(5) as most recent, most stable version.
5 - If you are using PIX 7.x OS, same issues with DNS inspect and the new esmtp inspect, same issue with inbound/outbound IP address differences..
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
cullendreaAuthor Commented:
Thank you for the suggestions.

I logged into the PIX and ran:
no fixup protocol smtp

The Exchange Server's outgoing IP address is different than the inbound MX record IP, but the nat rule and static commands had previously been run.

I did increase the DNS size to 1024 which helped with most but not all of the SMTP message queues.

The PIX OS is 6.3.4.

I still have the same problem only with wending to two different domain names now.
0
lrmooreCommented:
Progress is good, no?

>The Exchange Server's outgoing IP address is different than the inbound MX record IP,
As long as this is true, you will always have certain domains that simply will not accept email from you.
What does the current global/nat/static look like now?

>The PIX OS is 6.3.4
No big issues with this OS, but I'd still go ahead and upgrade to 6.3(5) if possible
0
cullendreaAuthor Commented:
The rest of the messages cleared out of the queue after a short period of time.  It sounds as if changing the maximum length of the fixup protocol for dns fixed it.  Thank you for your help.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Cisco

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.