Exchange Server 2013 - Mail stuck in queue for a handful of domains only...

I have a stand-alone Exchange 2013 (CU11) server.  All has been well for about 2 years now with all mail and any mail flowing well inbound and Outbound.  We do use Trend Micro Hosted Email Security for Inbound filtering.  As I just mentioned, we never had any issues with messages staying in the queue up until 5 days ago.  

Scenario:

1.  ALL inbound email is fine and flowing nicely.
2.  A handful of email domain suffixes will not go through.  For example, one of them is OUTBOUND mail to Sympatico.ca never leaves the Exchange Queue (also same with Bell.net and a few others).  90% of other outbound emails to other domains go through just fine.
3.  Tested DNS and appears to be resolving well
5.  Restarted the Transport Service numerous times to no avail
6.  No smoking gun in the Event Viewer
7.  Not using a smarthost on my Send Connector, using "MX record associated with recipient address" for delivery
8.  ISP even added a PTR record for my MX record.
9. get-ServerComponentState returns everything as Active

Really not sure what else to try or what typically causes only certain outbound email from specific domains from leaving the queue...  Here is what I see from one of the messages stuck in the queue (with replaced company specific info)...

================================================================
Identity: <InternalServerName>\2718\102499394519069
Subject: Test
Internet Message ID: <7656593dc5a2443a88144e4ef073a73b@<InternalServerName.DomainName.local>
From Address: XXXXXXXX@<ExternalDomainName.com>
Status: Ready
Size (KB): 20
Message Source Name: SMTP:Default <InternalServerName>
Source IP: 192.168.0.38
SCL: -1
Date Received: 9/4/2018 2:22:52 PM
Expiration Time: 9/8/2018 2:22:52 PM
Last Error:
Queue ID: <InternalServerName>\2718
Recipients:  XXXXXXXX@sympatico.ca;2;2;[{LRT=};{LED=};{FQDN=};{IP=}];0;CN=OutBound2015,CN=Connections,CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<DomainName>,DC=local;0
=======================================================================================================================
Can anyone suggest anything else to try or check?

Regards,

Dan
MuhnsterAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

timgreen7077Exchange EngineerCommented:
I would try to telnet to the server via port 25 for the recipient address. If the telnet unsuccessful then it may be your firewall or ISP blocking the communication from your exchange server to the recipient email domain. mxmta.owm.bell.net is the MX record for sympatico.ca, so i was able to successfully telnet to that server via port 25. In a cmd prompt on your exchange server run the following:

telnet mxmta.owm.bell.net 25.

See if it succeeds or fails, if it fails check your firewall to see if you guys are blocking it or its your ISP.
0
MuhnsterAuthor Commented:
Tim,
Much appreciate the quick comment and advice on this post!  I tried to telnet into mxmta.owm.bell.net 25 and I immediately received:

"421 Service not available"
"Connection to host lost"

BUT I was able to Telnet successfully from the same Exchange server command prompt to two other mail servers I manage outside of this customer's environment.  

This domain is not showing on ANY blastlists so how can I check if my ISP is blocking the connection to bell.net (I assume Sympatico would fall under a similar umbrella)?  

Thanks again for the quick response, really trying to get this resolved ASAP,

Dan
0
Saif ShaikhServer engineer Commented:
Hello,

Please increase the diagnostics login on send connector and check the send-receive logs in the exchange folder:\Exchange Server\V14\TransportRoles\Logs\ProtocolLog\SmtpSend

Set-SendConnector “Send Connector Name” -ProtocolLoggingLevel verbose

As per the NDR provided I do not see any last error: so we need to see where the connection is exactly failing in the send logs. Please paste the error here. Also are you using any transport agents on the exchange server. Try to disable the transport agents just for testing and check.

Get-transportagent - to view if you are using transport agents.

get-transportagent-disable-transportagent - to disable all transport agents
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

timgreen7077Exchange EngineerCommented:
if you are going directly out to the MX record without a smart host and the telnet failed, you may need to contact your ISP.
Can you receive emails from that domain, but the email would come through your inbound gateway so you maybe able to receive them fine, bit if sending is an issue you may need to check your ISP.
0
timgreen7077Exchange EngineerCommented:
also just a note are you using any DNS IP address on the internal or external DNS. If you log into the exchange admin center and go to servers, double click the exchange server and check under DNS. Do you have any IPs assigned or not?
0
MuhnsterAuthor Commented:
Saif,

Thank you very much for your direction.  Verbose logging was already enabled.  I "retried" the stuck queue in the queue viewer and collected the latest portion of the log it created.  This is V15 and this log was in a different location but I think I have the right one.  Getting a few hints about a connection refused due to a reputation issue with our external IP address.  The bell.net log simply says "421 service not available".

====================================================================================================
2018-09-04T23:53:35.638Z,OutBound2015,08D612BEF2F7CDFA,0,,72.167.238.32:25,*,,attempting to connect
2018-09-04T23:53:35.638Z,OutBound2015,08D612BEF2F7CDF9,3,192.168.0.38:61840,68.178.213.243:25,>,QUIT,
2018-09-04T23:53:35.638Z,OutBound2015,08D612BEF2F7CDF9,4,192.168.0.38:61840,68.178.213.243:25,-,,Remote
2018-09-04T23:53:35.712Z,OutBound2015,08D612BEF2F7CDFA,1,192.168.0.38:61841,72.167.238.32:25,+,,
2018-09-04T23:53:35.790Z,OutBound2015,08D612BEF2F7CDFA,2,192.168.0.38:61841,72.167.238.32:25,<,554 p3plibsmtp01-09.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:53:35.790Z,OutBound2015,08D612BEF2F7CDFB,0,,68.178.213.244:25,*,,attempting to connect
2018-09-04T23:53:35.791Z,OutBound2015,08D612BEF2F7CDFA,3,192.168.0.38:61841,72.167.238.32:25,>,QUIT,
2018-09-04T23:53:35.791Z,OutBound2015,08D612BEF2F7CDFA,4,192.168.0.38:61841,72.167.238.32:25,-,,Remote
2018-09-04T23:53:35.866Z,OutBound2015,08D612BEF2F7CDFB,1,192.168.0.38:61842,68.178.213.244:25,+,,
2018-09-04T23:53:35.948Z,OutBound2015,08D612BEF2F7CDFB,2,192.168.0.38:61842,68.178.213.244:25,<,554 p3plibsmtp03-10.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:53:35.948Z,OutBound2015,08D612BEF2F7CDFB,3,192.168.0.38:61842,68.178.213.244:25,>,QUIT,
2018-09-04T23:53:35.949Z,OutBound2015,08D612BEF2F7CDFB,4,192.168.0.38:61842,68.178.213.244:25,-,,Remote
2018-09-04T23:58:28.782Z,OutBound2015,08D612BEF2F7CDFF,0,,184.150.200.82:25,*,,attempting to connect
2018-09-04T23:58:28.787Z,OutBound2015,08D612BEF2F7CDFE,0,,184.150.200.82:25,*,,attempting to connect
2018-09-04T23:58:28.800Z,OutBound2015,08D612BEF2F7CDFF,1,192.168.0.38:62317,184.150.200.82:25,+,,
2018-09-04T23:58:28.804Z,OutBound2015,08D612BEF2F7CDFE,1,192.168.0.38:62318,184.150.200.82:25,+,,
2018-09-04T23:58:28.824Z,OutBound2015,08D612BEF2F7CDFF,2,192.168.0.38:62317,184.150.200.82:25,<,421 Service not available,
2018-09-04T23:58:28.824Z,OutBound2015,08D612BEF2F7CDFF,3,192.168.0.38:62317,184.150.200.82:25,>,QUIT,
2018-09-04T23:58:28.824Z,OutBound2015,08D612BEF2F7CDFF,4,192.168.0.38:62317,184.150.200.82:25,-,,Remote
2018-09-04T23:58:28.827Z,OutBound2015,08D612BEF2F7CDFE,2,192.168.0.38:62318,184.150.200.82:25,<,421 Service not available,
2018-09-04T23:58:28.827Z,OutBound2015,08D612BEF2F7CDFE,3,192.168.0.38:62318,184.150.200.82:25,>,QUIT,
2018-09-04T23:58:28.827Z,OutBound2015,08D612BEF2F7CDFE,4,192.168.0.38:62318,184.150.200.82:25,-,,Remote
2018-09-04T23:58:36.181Z,OutBound2015,08D612BEF2F7CE00,0,,72.167.238.29:25,*,,attempting to connect
2018-09-04T23:58:36.257Z,OutBound2015,08D612BEF2F7CE00,1,192.168.0.38:62319,72.167.238.29:25,+,,
2018-09-04T23:58:36.345Z,OutBound2015,08D612BEF2F7CE00,2,192.168.0.38:62319,72.167.238.29:25,<,554 p3plibsmtp01-08.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:36.345Z,OutBound2015,08D612BEF2F7CE01,0,,68.178.213.37:25,*,,attempting to connect
2018-09-04T23:58:36.346Z,OutBound2015,08D612BEF2F7CE00,3,192.168.0.38:62319,72.167.238.29:25,>,QUIT,
2018-09-04T23:58:36.346Z,OutBound2015,08D612BEF2F7CE00,4,192.168.0.38:62319,72.167.238.29:25,-,,Remote
2018-09-04T23:58:36.423Z,OutBound2015,08D612BEF2F7CE01,1,192.168.0.38:62320,68.178.213.37:25,+,,
2018-09-04T23:58:36.501Z,OutBound2015,08D612BEF2F7CE01,2,192.168.0.38:62320,68.178.213.37:25,<,554 p3plibsmtp02-01.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:36.501Z,OutBound2015,08D612BEF2F7CE02,0,,68.178.213.203:25,*,,attempting to connect
2018-09-04T23:58:36.501Z,OutBound2015,08D612BEF2F7CE01,3,192.168.0.38:62320,68.178.213.37:25,>,QUIT,
2018-09-04T23:58:36.501Z,OutBound2015,08D612BEF2F7CE01,4,192.168.0.38:62320,68.178.213.37:25,-,,Remote
2018-09-04T23:58:36.579Z,OutBound2015,08D612BEF2F7CE02,1,192.168.0.38:62322,68.178.213.203:25,+,,
2018-09-04T23:58:36.661Z,OutBound2015,08D612BEF2F7CE02,2,192.168.0.38:62322,68.178.213.203:25,<,554 p3plibsmtp03-04.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:36.661Z,OutBound2015,08D612BEF2F7CE03,0,,72.167.238.32:25,*,,attempting to connect
2018-09-04T23:58:36.662Z,OutBound2015,08D612BEF2F7CE02,3,192.168.0.38:62322,68.178.213.203:25,>,QUIT,
2018-09-04T23:58:36.662Z,OutBound2015,08D612BEF2F7CE02,4,192.168.0.38:62322,68.178.213.203:25,-,,Remote
2018-09-04T23:58:36.736Z,OutBound2015,08D612BEF2F7CE03,1,192.168.0.38:62323,72.167.238.32:25,+,,
2018-09-04T23:58:36.817Z,OutBound2015,08D612BEF2F7CE03,2,192.168.0.38:62323,72.167.238.32:25,<,554 p3plibsmtp01-12.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:36.818Z,OutBound2015,08D612BEF2F7CE04,0,,68.178.213.244:25,*,,attempting to connect
2018-09-04T23:58:36.818Z,OutBound2015,08D612BEF2F7CE03,3,192.168.0.38:62323,72.167.238.32:25,>,QUIT,
2018-09-04T23:58:36.818Z,OutBound2015,08D612BEF2F7CE03,4,192.168.0.38:62323,72.167.238.32:25,-,,Remote
2018-09-04T23:58:36.891Z,OutBound2015,08D612BEF2F7CE04,1,192.168.0.38:62324,68.178.213.244:25,+,,
2018-09-04T23:58:36.973Z,OutBound2015,08D612BEF2F7CE04,2,192.168.0.38:62324,68.178.213.244:25,<,554 p3plibsmtp03-13.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:36.974Z,OutBound2015,08D612BEF2F7CE05,0,,68.178.213.243:25,*,,attempting to connect
2018-09-04T23:58:36.974Z,OutBound2015,08D612BEF2F7CE04,3,192.168.0.38:62324,68.178.213.244:25,>,QUIT,
2018-09-04T23:58:36.974Z,OutBound2015,08D612BEF2F7CE04,4,192.168.0.38:62324,68.178.213.244:25,-,,Remote
2018-09-04T23:58:37.047Z,OutBound2015,08D612BEF2F7CE05,1,192.168.0.38:62325,68.178.213.243:25,+,,
2018-09-04T23:58:37.126Z,OutBound2015,08D612BEF2F7CE05,2,192.168.0.38:62325,68.178.213.243:25,<,554 p3plibsmtp02-05.prod.phx3.secureserver.net bizsmtp IB103. Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,
2018-09-04T23:58:37.126Z,OutBound2015,08D612BEF2F7CE05,3,192.168.0.38:62325,68.178.213.243:25,>,QUIT,
2018-09-04T23:58:37.127Z,OutBound2015,08D612BEF2F7CE05,4,192.168.0.38:62325,68.178.213.243:25,-,,Remote
================================================================================================

I do have a number of agents that are running but none of these are new and this was not an issue until approx. 5 days ago.

=====================================================================================================

Identity                                              Enabled         Priority
--------                                                   -------              --------
ScanMail SMTP Receive Agent         True            1
ScanMail Routing Agent                    True            2
Content Filter Agent                          True            3
Sender Id Agent                                 True            4
Sender Filter Agent                            True            5
Recipient Filter Agent                        True            6
Protocol Analysis Agent                     True            7
Transport Rule Agent                         True            8
Malware Agent                                    True            9
Text Messaging Routing Agent         True            10
Text Messaging Delivery Agent        True            11
System Probe Drop Smtp Agent      True            12
System Probe Drop Routing Agent  True            13
=======================================================================================

Thanks again!

Dan
0
MuhnsterAuthor Commented:
Thanks Tim,

We do receive email from the individuals that we are replying to from the domains in question.

Under the DNS Lookup section:

Both External and Internal DNS Lookups are manually configured to point to our internal DC which provides all internal and external resolution to clients.

Thanks!

Dan
0
timgreen7077Exchange EngineerCommented:
I would add your ISP for DNS and allow that to resolve the DNS externally. remove your internal DNS and add the IP of your ISP and test. add it all your exchange servers that send mail outbound.
0
MuhnsterAuthor Commented:
Ok thanks Tim,

I thought it would have been enough to use my ISP's DNS servers in my Forwarders on our DNS server and on the external firewall/router.  I will try adding them in the EAC under the External DNS in the DNS Lookup section.

Thanks again!

Dan
0
MuhnsterAuthor Commented:
Hey Tim and Saif,

Ok, so after reviewing the logs, I initiated the reputation list reset (automated, online and almost instant) AND customized the External DNS Lookup in the EAC (Under servers) to reflect the ISP's DNS IPs and within minutes my queues cleared!

I believe that both of these identified issues and recommendations had a contribution to resolution.  Thanks to both of you so very much.  You were both very much part of this resolution.

Regards,

Dan
0
MuhnsterAuthor Commented:
Is there a way to give credit for a solution to both of you?  There really were two issues resolved here and it certainly appears that both of you had a hand with resolving both identified issues...

Dan
0
Saif ShaikhServer engineer Commented:
That gr8 to here that your issue is resolved: The cause was
Connection refused. 45.72.190.206 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://csi.cloudmark.com/reset-request/?ip=45.72.190.206 to request a delisting.,

Your public IP was blacklisted. Please do a full antivirus/malware scan on exchange server and domain joined desktops to make sure there are no issues with end users PC might have caught spammer attention.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
timgreen7077Exchange EngineerCommented:
yes you could have gave credit to us both. you would need to select both answers. one as best solution and other as assisted. This issue had nothing to do with blacklist. That was display by the failure to successfully telnet, even if it was blacklisted you would have been able to successfully telnet to the MX record, it was really corrected by the change I suggested with DNS.
0
Saif ShaikhServer engineer Commented:
If this issue had nothing to do with blacklist then Tim exchange should be working according to you. Logs don't lie.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Chat / IM

From novice to tech pro — start learning today.