Exchange 2003 - Emails Stuck in Internet Mail SMTP Queue

I hope it is not
Exchange queues fill with many non-delivery reports from the postmaster account in Small Business Server 2003
http://support.microsoft.com/kb/886208 

First step is to see if the domain's IP is blacklisted.
Check in http://www.mxtoolbox.com/blacklists.aspx - See if your public IP address is blacklisted.
If yes, it looks like someone might have done a Reverse NDR attack on your server.

In order to avoid such things from happening, you can use Antispam features on the Exchange.
Recipient filtering, sender filtering, IMF etc can be used which is pretty good solution for such spam attacks.

I have checked the IP's and they aren't blacklisted. I also checked the emails that are in the queue and they aren't NDRs. From what i can tell by looking at packet dumps and logs, exchange isn't forwarding emails to our smart host, which is a Cisco IronPort. I have looked at packet captures from both the exchange box and the IronPort appliance, and the SMTP connections are being terminated from the exchange box.

What do we see in NCSA logs? Do we even see a communication establishment?

NCSA?

In the SMTP logs, I see the SMTP connection to the smart host established and then terminated after the smart host responds to the helo command. the response from the smart host (IronPort) is the correct 250 response. the exchange server is the one closing the connection. I also saw this same conversation in the packet captures from both systems.

What are the third party antivirus that you see on the Exchange server?
It may be due to some third party antivirus.

Try to disable it and restart the SMTP service.

The AV solution is Symantec Endpoint Protection v11.0.6005.562. I checked the exceptions to make sure the SMTP Queue, Database are in the exclusion list and they are. I also disabled the AV and restarted the SMTP service. after the SMTP service restarted the messages were still stuck in the queue but the number of 'Delivery Failures' was reset.

ok, have a few questions here
1. By anychance do you see a 220************************************* banner when sending emails out to the smart host?
2. By anychance are you seeing only meeting invites by anychance stuck in the queue?
3. Is the queue in retry state? - if yes what is the additional queue information?

1. No I don't see that in the packet dumps or SMTP logs.
2. No, i have another regular email stuck in the queue.
3 Yes, see below

From Search results:
Message size: 2,456,876
Time Submitted: 7/14/2011 9:46AM
Time Received by server: 7/14/2011 9:46AM

From Message Properties:
Priority: Normal
Number of body recipients: N/A
Delivery failures: 244 (This count was reset when i restarted the SMTP Service)
Status: retry

Additional queue information is something that you see in the bottom of the screen for the queue.
Usually it says stuff like "Unable to open message for delivery" or "Connection dropped by remote host" or "Unable to bind to destination server in DNS" or "SMTP protocol error" etc.
what does it read?

You can use

http://support.microsoft.com/kb/823489

to troubleshoot the issue

The additional queue information says: "Unable to open the message for delivery"

I set the logging level to maximum and I am getting a bunch of those 327 messages.
Here is one of them:
Event Type:      Warning
Event Source:      MSExchangeTransport
Event Category:      Exchange Store Driver
Event ID:      327
Date:            7/15/2011
Time:            10:57:50 AM
User:            N/A
Computer:      Exchange
Description:
The following call : EcGetMime to the store failed. Error code : -2147467259. MDB : 176028cb-5077-442a-bff0-7a6d29b5c9b5. FID : 1-34. MID : 1-1F52826. File : .

For more information, click http://www.microsoft.com/contentredirect.asp.

What XCONBLR said is correct. The hotfix needs to be applied on the mailbox server. Make sure all the dlls mentioned in the hotfix are present in the bin directory once that is done restart the sa and iisadmin service. The old messages may be delivered sometimes it doesn't but new messages will not be stuck.

I looked at the file list for that hot fix and we have newer versions of most of those files and not all of them were in the exchsrv\bin directory, which we have on the D:\ drive.

Should I apply this hotfix anyway?

Here is a list of file versions we have:
File name                              File version
Cdo.dll                                    6.5.7654.12
Cdoex.dll                                      6.5.7654.12 (not in exchsrv\bin, found in Program files\Common Files\Microsoft Shared\CDO)
Davex.dll                                      6.5.7654.12
Excdo.dll                                      6.5.7654.12
Exoledb.dll                              6.5.7654.12
Lsmexhc.dll                              6.5.7654.12
Madfb.dll                                      6.5.7654.12
Massync.dll                              6.5.7638.1
Microsoft.exchange.oma.userinterface.dll      Not Found
Rtdsmcal.dll                              Not Found

Yes the hotfix needs to be applied and dlls should be present in bin. If required try to extract it and place them in bin. The missing ones need to be present and the higher ones need not to be changed.

In order others all dlls should be present the higher version ones may not be changed.

Ok, I have updates pending on the server now, I'll reboot the server tonight to finish that install and then apply the patch and recheck file versions. I'll keep you all posted on my progress and BIG BIG thanks for your responses!!!

How are the things going? Was there an update installed? Did that help? or are you facing the same issue again?

I installed the hotfix and applied the registry change. The messages stuck in the queue are gone and I don't see any more 327 events in the application log.

I'll keep an eye on it for a couple days and let you know if I have any issues.

BIG BIG THANKS for all you help!!

Well it's been almost a month and we haven't had any more email stuck in the SMTP queue.

BIG BIG BIG THANKS for all your help!!