Exchange server cannot send email to a particular email server

John Atkinson
John Atkinson used Ask the Experts™
on
I'll start with the question:  how can I fix my Exchange server's new inability to connect to a remote mail server, to send mail?  Now, the background...

My client's on-premise Microsoft Exchange 2010 server recently lost the ability to send mail to accounts at a domain I'll call pulp.com.  I'm trying to restore this capability.  Specifically, when my client sends an email to, say, john@pulp.com, the message appears to have been sent, but some time later, my client receives a non-delivery report saying

400 4.4.7 Message delayed

Open in new window


My client receives several of these, spaced out by hours, as the Exchange server retries sending the email.  Eventually, an NDR comes in stating

#550 4.4.7 QUEUE.Expired; message expired ##

Open in new window


pulp.com is fine.  (The owner of pulp.com is also a client of mine, as it happens.)  Messages are arriving just fine to john@pulp.com from other senders.  Only my Exchange client is having trouble.

I checked Exchange 2010's SMTP logs for clues.  Here, I found scores of entries that resemble these:  (Note that I'm substituting 23.23.23.23 for pulp.com's real IP address)
2018-10-31T23:19:47.336Z	0	23.23.23.23:25	*	attempting to connect 
2018-10-31T23:20:08.338Z	1	23.23.23.23:25	*	Failed to connect. Error Code: 10060, Error Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 23.23.23.23:25

Open in new window


So, there's a connection attempt, and 20 minutes later, an acknowledgement that it didn't work, due to the remote server not responding.
The SMTP log shows hundreds of successful connection attempts, and message transmissions, all beginning with an attempt to connect on port 25.  Only pulp.com's IP address shows up in these "Failed to connect" scenarios.

The connection only started failing a few days ago.  20 hours before the first failure was the most recent successful connection attempt.  A successful attempt shows up in the log like this:
 Date       Time          Seq. #   LocalEndpoint    RemoteEndpoint
 2018-10-31T03:43:59.225Z  0       23.23.23.23:25                  * attempting to connect 
 2018-10-31T03:43:59.445Z  2       10.10.16.5:56137 23.23.23.23:25 < 220-mail.pulp.com ESMTP Exim 4.91 #1 Tue, 30 Oct 2018 20:43:59 -0700 
 2018-10-31T03:43:59.445Z  3       10.10.16.5:56137 23.23.23.23:25 < 220-We do not authorize the use of this system to transport unsolicited, 
 2018-10-31T03:43:59.445Z  4       10.10.16.5:56137 23.23.23.23:25 < 220 and/or bulk e-mail. 
 2018-10-31T03:43:59.445Z  5       10.10.16.5:56137 23.23.23.23:25 > EHLO mail.onPremExchangeClient.net 
 2018-10-31T03:43:59.506Z  6       10.10.16.5:56137 23.23.23.23:25 < 250-mail.pulp.com Hello mail.onPremExchangeClient.net [209.23.23.23] 
 2018-10-31T03:43:59.506Z  7       10.10.16.5:56137 23.23.23.23:25 < 250-SIZE 52428800 
 2018-10-31T03:43:59.506Z  8       10.10.16.5:56137 23.23.23.23:25 < 250-8BITMIME 
 2018-10-31T03:43:59.506Z  9       10.10.16.5:56137 23.23.23.23:25 < 250-PIPELINING 
 2018-10-31T03:43:59.506Z  10      10.10.16.5:56137 23.23.23.23:25 < 250-AUTH PLAIN LOGIN 
 2018-10-31T03:43:59.506Z  11      10.10.16.5:56137 23.23.23.23:25 < 250-STARTTLS 
 2018-10-31T03:43:59.506Z  12      10.10.16.5:56137 23.23.23.23:25 < 250 HELP 
 2018-10-31T03:43:59.506Z  13      10.10.16.5:56137 23.23.23.23:25 * 385863 sending message 
 2018-10-31T03:43:59.506Z  14      10.10.16.5:56137 23.23.23.23:25 > MAIL FROM:<client@onPremExchangeClient.net> SIZE=7949 
 2018-10-31T03:43:59.506Z  15      10.10.16.5:56137 23.23.23.23:25 > RCPT TO:<john@pulp.com> 
 2018-10-31T03:43:59.568Z  16      10.10.16.5:56137 23.23.23.23:25 < 250 OK 
 2018-10-31T03:43:59.568Z  17      10.10.16.5:56137 23.23.23.23:25 < 250 Accepted 
 2018-10-31T03:43:59.568Z  18      10.10.16.5:56137 23.23.23.23:25 > DATA 
 2018-10-31T03:43:59.629Z  19      10.10.16.5:56137 23.23.23.23:25 < 354 Enter message, ending with "." on a line by itself 
 2018-10-31T03:43:59.761Z  20      10.10.16.5:56137 23.23.23.23:25 < 250 OK id=1gHhQF-007GaQ-Dn 
 2018-10-31T03:43:59.761Z  21      10.10.16.5:56137 23.23.23.23:25 > QUIT 
 2018-10-31T03:43:59.823Z  22      10.10.16.5:56137 23.23.23.23:25 < 221 mail.pulp.com closing connection 
 2018-10-31T03:43:59.823Z  23      10.10.16.5:56137 23.23.23.23:25 - Local 

Open in new window


Just to sanity check, I tried telnet on the Exchange server:
C:\>telnet mail.pulp.com 25 
Connecting To mail.pulp.com...Could not open connection to the host, on port 25: Connect failed

C:\>telnet 23.23.23.23 25 
Connecting To 23.23.23.23...Could not open connection to the host, on port 25: Connect failed

Open in new window


From any offsite location, telnet to these IPs connects fine.   Sending mail to anyaddress@pulp.com works from any other domain I've tried.  

How do I figure out why neither Exchange server's SMTP service, nor telnet, can connect to this mail server, when everyone else can?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Jackson FavreTechnical Consultant

Commented:
Hi John, This is such an interesting behaviour.

Can you telnet 23.23.23.23:25 from any other machine on the same network as your Exchange server?

What firewall are you running in front of the Exchange server?

What relevant logs are present in the firewall?

Cheers

Jackson
John AtkinsonIT Consultant

Author

Commented:
Hi, Jackson.

I cannot telnet to 23.23.23.23.:25 from any computer on the network, neither the servers, nor the workstations.

We had been running a Cisco ASA 5505 until a couple of weeks before this email problem.  It seemed to have failed (it was old) and I removed it, and, for now, built the network using an available Linksys router.   We lost some connectivity when I did that, but I got the organization back onto the Internet, the first priority.  

Thanks,
John
Jackson FavreTechnical Consultant

Commented:
Firstly , does the Linksys router have logging capability?
If it does you can check the logs to see whether it forwards that traffic.

Can you hit 23.23.23.23 on any other port?
If you can't, it's possible that the firewall on the other side is blacklisting or geo-blocking your IP.

If you can hit that IP on other ports, just not SMTP, I would even check that his AntiSpam solution isn't dropping your tcp-25 traffic using an overly-zealous RBL or other pre-connection (SMTP-level) check.

Keen to hear the response so we can look for a next-step.

Cheers

Jackson
Starting with Angular 5

Learn the essential features and functions of the popular JavaScript framework for building mobile, desktop and web applications.

John AtkinsonIT Consultant

Author

Commented:
This router does not offer logging.  

I can't hit 23.23.23.23 on ports 24, 25, 26, or 587, from this network.  

I CAN hit it using port 80, and connect.  

I've been working with the tech support folks whose company hosts pulp.com for two days, and I've learned that our IP address is not blocked at their end, or by any RBL.  It's not an anti-spam issue, since we never actually relay any messages.  Their server logs show absolutely no connection activity from our IP.  It's looking like, we attempt the connection, and pulp.com never sees it, and we time out starving for response.

Thanks,
John
timgreen7077Exchange Engineer
Distinguished Expert 2018

Commented:
this sounds like a firewall issue so review your firewall rules. you may have router, firewall, IDS or anything blocking it. make an allow rule to allow traffic to the IP address
Technical Consultant
Commented:
Was this previously working and did this stop working when you implemented the Linksys firewall?

You said they are looking at their server logs, do their firewall logs show your inbound connection?

Do you have access to any other firewalls that you could use to test?

This looks like active blocking of SMTP ports between your Exchange server and theirs, unless your firewall is doing some GeoBlocking or has a specific deny for that IP range (which I doubt because it's pretty simplistic and you've likely checked that), or their Firewall is doing something similar.

Are there any firewalls between their firewall and their upstream router or their firewall and their Exchange server (up and downstream of the firewall they are looking at) that they either don't control or haven't looked at yet that may be specifically blocking your IP?

My money is still on Antispam or GeoBlocking as it has been the majority of time I've seen this before.

Cheers

Jackson
John AtkinsonIT Consultant

Author

Commented:
pulp.com's hosting company finally looked in the right place and found that the Exchange server's IP address was blacklisted on their firewall, due to "incorrect SMTP authenticate login attempts".  I don't believe server-to-server relays use SMTP authentication, so I assume these attempts were from an email client on the same LAN as the Exchange server, committing these SMTP authentication errors.  Pretty draconian policy, for all the trouble it's caused.  

Thanks, all, for helping to steer me in the right direction as I've dealt with pulp.com's hosting company.  Hope I can return the favor to another user here soon.

John
Jackson FavreTechnical Consultant

Commented:
Glad to hear it's been resolved.

Cheers

Jackson

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial