Exchange server cannot send email to a particular email server

I'll start with the question:  how can I fix my Exchange server's new inability to connect to a remote mail server, to send mail?  Now, the background...

My client's on-premise Microsoft Exchange 2010 server recently lost the ability to send mail to accounts at a domain I'll call pulp.com.  I'm trying to restore this capability.  Specifically, when my client sends an email to, say, john@pulp.com, the message appears to have been sent, but some time later, my client receives a non-delivery report saying

400 4.4.7 Message delayed

Open in new window


My client receives several of these, spaced out by hours, as the Exchange server retries sending the email.  Eventually, an NDR comes in stating

#550 4.4.7 QUEUE.Expired; message expired ##

Open in new window


pulp.com is fine.  (The owner of pulp.com is also a client of mine, as it happens.)  Messages are arriving just fine to john@pulp.com from other senders.  Only my Exchange client is having trouble.

I checked Exchange 2010's SMTP logs for clues.  Here, I found scores of entries that resemble these:  (Note that I'm substituting 23.23.23.23 for pulp.com's real IP address)
2018-10-31T23:19:47.336Z	0	23.23.23.23:25	*	attempting to connect 
2018-10-31T23:20:08.338Z	1	23.23.23.23:25	*	Failed to connect. Error Code: 10060, Error Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 23.23.23.23:25

Open in new window


So, there's a connection attempt, and 20 minutes later, an acknowledgement that it didn't work, due to the remote server not responding.
The SMTP log shows hundreds of successful connection attempts, and message transmissions, all beginning with an attempt to connect on port 25.  Only pulp.com's IP address shows up in these "Failed to connect" scenarios.

The connection only started failing a few days ago.  20 hours before the first failure was the most recent successful connection attempt.  A successful attempt shows up in the log like this:
 Date       Time          Seq. #   LocalEndpoint    RemoteEndpoint
 2018-10-31T03:43:59.225Z  0       23.23.23.23:25                  * attempting to connect 
 2018-10-31T03:43:59.445Z  2       10.10.16.5:56137 23.23.23.23:25 < 220-mail.pulp.com ESMTP Exim 4.91 #1 Tue, 30 Oct 2018 20:43:59 -0700 
 2018-10-31T03:43:59.445Z  3       10.10.16.5:56137 23.23.23.23:25 < 220-We do not authorize the use of this system to transport unsolicited, 
 2018-10-31T03:43:59.445Z  4       10.10.16.5:56137 23.23.23.23:25 < 220 and/or bulk e-mail. 
 2018-10-31T03:43:59.445Z  5       10.10.16.5:56137 23.23.23.23:25 > EHLO mail.onPremExchangeClient.net 
 2018-10-31T03:43:59.506Z  6       10.10.16.5:56137 23.23.23.23:25 < 250-mail.pulp.com Hello mail.onPremExchangeClient.net [209.23.23.23] 
 2018-10-31T03:43:59.506Z  7       10.10.16.5:56137 23.23.23.23:25 < 250-SIZE 52428800 
 2018-10-31T03:43:59.506Z  8       10.10.16.5:56137 23.23.23.23:25 < 250-8BITMIME 
 2018-10-31T03:43:59.506Z  9       10.10.16.5:56137 23.23.23.23:25 < 250-PIPELINING 
 2018-10-31T03:43:59.506Z  10      10.10.16.5:56137 23.23.23.23:25 < 250-AUTH PLAIN LOGIN 
 2018-10-31T03:43:59.506Z  11      10.10.16.5:56137 23.23.23.23:25 < 250-STARTTLS 
 2018-10-31T03:43:59.506Z  12      10.10.16.5:56137 23.23.23.23:25 < 250 HELP 
 2018-10-31T03:43:59.506Z  13      10.10.16.5:56137 23.23.23.23:25 * 385863 sending message 
 2018-10-31T03:43:59.506Z  14      10.10.16.5:56137 23.23.23.23:25 > MAIL FROM:<client@onPremExchangeClient.net> SIZE=7949 
 2018-10-31T03:43:59.506Z  15      10.10.16.5:56137 23.23.23.23:25 > RCPT TO:<john@pulp.com> 
 2018-10-31T03:43:59.568Z  16      10.10.16.5:56137 23.23.23.23:25 < 250 OK 
 2018-10-31T03:43:59.568Z  17      10.10.16.5:56137 23.23.23.23:25 < 250 Accepted 
 2018-10-31T03:43:59.568Z  18      10.10.16.5:56137 23.23.23.23:25 > DATA 
 2018-10-31T03:43:59.629Z  19      10.10.16.5:56137 23.23.23.23:25 < 354 Enter message, ending with "." on a line by itself 
 2018-10-31T03:43:59.761Z  20      10.10.16.5:56137 23.23.23.23:25 < 250 OK id=1gHhQF-007GaQ-Dn 
 2018-10-31T03:43:59.761Z  21      10.10.16.5:56137 23.23.23.23:25 > QUIT 
 2018-10-31T03:43:59.823Z  22      10.10.16.5:56137 23.23.23.23:25 < 221 mail.pulp.com closing connection 
 2018-10-31T03:43:59.823Z  23      10.10.16.5:56137 23.23.23.23:25 - Local 

Open in new window


Just to sanity check, I tried telnet on the Exchange server:
C:\>telnet mail.pulp.com 25 
Connecting To mail.pulp.com...Could not open connection to the host, on port 25: Connect failed

C:\>telnet 23.23.23.23 25 
Connecting To 23.23.23.23...Could not open connection to the host, on port 25: Connect failed

Open in new window


From any offsite location, telnet to these IPs connects fine.   Sending mail to anyaddress@pulp.com works from any other domain I've tried.  

How do I figure out why neither Exchange server's SMTP service, nor telnet, can connect to this mail server, when everyone else can?
John AtkinsonIT ConsultantAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jackson FavreTechnical ConsultantCommented:
Hi John, This is such an interesting behaviour.

Can you telnet 23.23.23.23:25 from any other machine on the same network as your Exchange server?

What firewall are you running in front of the Exchange server?

What relevant logs are present in the firewall?

Cheers

Jackson
John AtkinsonIT ConsultantAuthor Commented:
Hi, Jackson.

I cannot telnet to 23.23.23.23.:25 from any computer on the network, neither the servers, nor the workstations.

We had been running a Cisco ASA 5505 until a couple of weeks before this email problem.  It seemed to have failed (it was old) and I removed it, and, for now, built the network using an available Linksys router.   We lost some connectivity when I did that, but I got the organization back onto the Internet, the first priority.  

Thanks,
John
Jackson FavreTechnical ConsultantCommented:
Firstly , does the Linksys router have logging capability?
If it does you can check the logs to see whether it forwards that traffic.

Can you hit 23.23.23.23 on any other port?
If you can't, it's possible that the firewall on the other side is blacklisting or geo-blocking your IP.

If you can hit that IP on other ports, just not SMTP, I would even check that his AntiSpam solution isn't dropping your tcp-25 traffic using an overly-zealous RBL or other pre-connection (SMTP-level) check.

Keen to hear the response so we can look for a next-step.

Cheers

Jackson
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

John AtkinsonIT ConsultantAuthor Commented:
This router does not offer logging.  

I can't hit 23.23.23.23 on ports 24, 25, 26, or 587, from this network.  

I CAN hit it using port 80, and connect.  

I've been working with the tech support folks whose company hosts pulp.com for two days, and I've learned that our IP address is not blocked at their end, or by any RBL.  It's not an anti-spam issue, since we never actually relay any messages.  Their server logs show absolutely no connection activity from our IP.  It's looking like, we attempt the connection, and pulp.com never sees it, and we time out starving for response.

Thanks,
John
timgreen7077Exchange EngineerCommented:
this sounds like a firewall issue so review your firewall rules. you may have router, firewall, IDS or anything blocking it. make an allow rule to allow traffic to the IP address
Jackson FavreTechnical ConsultantCommented:
Was this previously working and did this stop working when you implemented the Linksys firewall?

You said they are looking at their server logs, do their firewall logs show your inbound connection?

Do you have access to any other firewalls that you could use to test?

This looks like active blocking of SMTP ports between your Exchange server and theirs, unless your firewall is doing some GeoBlocking or has a specific deny for that IP range (which I doubt because it's pretty simplistic and you've likely checked that), or their Firewall is doing something similar.

Are there any firewalls between their firewall and their upstream router or their firewall and their Exchange server (up and downstream of the firewall they are looking at) that they either don't control or haven't looked at yet that may be specifically blocking your IP?

My money is still on Antispam or GeoBlocking as it has been the majority of time I've seen this before.

Cheers

Jackson

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
John AtkinsonIT ConsultantAuthor Commented:
pulp.com's hosting company finally looked in the right place and found that the Exchange server's IP address was blacklisted on their firewall, due to "incorrect SMTP authenticate login attempts".  I don't believe server-to-server relays use SMTP authentication, so I assume these attempts were from an email client on the same LAN as the Exchange server, committing these SMTP authentication errors.  Pretty draconian policy, for all the trouble it's caused.  

Thanks, all, for helping to steer me in the right direction as I've dealt with pulp.com's hosting company.  Hope I can return the favor to another user here soon.

John
Jackson FavreTechnical ConsultantCommented:
Glad to hear it's been resolved.

Cheers

Jackson
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.