Link to home
Start Free TrialLog in
Avatar of Roger Vallee
Roger ValleeFlag for United States of America

asked on

External emails not being received since KB5008207 installed on server

Hello,

After installing KB5008207 external email is not coming in.

I installed KB5008207 on the exchange server on premises. This was the update required for the Windows Server 2016 server on which it is installed.  Normally after I install patches from the Patch Tuesday for each month I do the following:

1. Confirm that the Microsoft Exchange services are started
2. Confirm I can access the EAC
3. Confirm that the database is mounted and is in the healthy state
4. Confirm I can access my outlook client as usual and that it shows "all folders up to date"


However, yesterday after installing the KB5008207 on the server:

* I noticed that I could not open my Outlook client on my machine. It is unclear if this issue is also related to the issue above.  I confirmed that 1-3 above are OK.
* I logged onto another PC and confirmed I could send email internally to staff and externally to Gmail. However when I tested sending a test email from Gmail to my company email it was not received.
* I checked the queue viewer and do not see emails in the queue.

* I received a bounce message when I tried to send my self a message from yahoo but it just states it was unable to deliver after multiple tries.


In the past this happened once and it was due to low disk space on the Exchange server. I confirmed we had 62GB of space available but I increased it to 92GB.  I read it was recommended to restart the Exchange Transport service and did this. However as of yesterday evening external emails are not being received.

This seems to be tied to the KB5008207 unless it is coincidental as there were no issues with Exchange until after the update was installed. No configuration changes of any kind were made to Exchange before this issue started last night.

I appreciate any help that could be provided as I need to get external emails working again.
Thank you.
 

I used the Microsoft Remote Connectivity Analyzer to Test inbound SMTP mail flow for domain (my mailbox) and it shows The specified port is either blocked, not listening, or not producing the expected response .


How do I fix this?



Avatar of Kimputer
Kimputer

The event viewer will probably give you hints why it's not receiving emails. Try to find the part where the server was rebooting, and find if any of the MS Exchange services is started with an error (most likely one of the Transport services). Also check if port 25 is working on the server itself (if it is, it's a firewall issue).
Avatar of Roger Vallee

ASKER

Hi Kimputer,

Thank you. I just tested with telnet on the exchange server and get:
220 ExchangeServer.DomainName Microsoft ESMTP MAIL Service ready at Sat, 18 Dec 2021 10:02:18 -0500

 When you say firewall issue, do you mean the firewall protecting the network or the firewall on the exchange server?  How could this suddenly be a problem if everything was working until last night when the update was installed?

What would need to be done on the firewall to fix this?  Are you saying that something would have changed on the firewall?  I didn't change anything on the firewall and it would only be me that would make changes.  I'm just trying to understand how this issue happened and what needs to be done to fix it.
Thank you.

**I confirmed with the firewall provider that there are no issues as traffic is being forwarded to the exchange server.  So from the firewall side this does not appear to be the problem.
Did you do your telnet test from an outside address? The connectivity analyzer eliminates some of this, but doesn't directly test the ip address you think it should be but the one the dns resolves.
If yes, then your receive connectors are working and it points to either the transport service, the mailbox transport service or the information store service.
Do you get a failure message from gmail?
Are you running internal anti-spam on exchange?
Since you say internal works I think that eliminates mailbox transport.
Also double check your receive connector, and double check that your fqdn resolves to the right address from outside.

Hi Scott,

I tested telnet on the exchange server itself. How do I test telnet from an outside address?  

The failure message I received from Yahoo when trying to send to my company email just "Unable to deliver message after multiple retries, giving up."  No codes in the message itself.

Yes anti-spam is being run on the exchange itself but this didn't change and I haven't made any changes.
Yes, sending emails between staff internally works oK.

You mention the receive connector and to check that the fQDN resolves to the right address from outside.  Can you elaborate?

Right now the FQDN in the scoping section for each receive connector is the servername.domainname.

Note: In October I changed the FQDN of the send connector from the servname.domainname to the mail.domainname. This was done to previous the disclosure of the internal IP in the message header.  

Should the FQDN of the send connectors be the same as the receive connectors?  If so, wouldn't I have had an issue before now?

Thanks.
"
Since telnet is working on the server itself, telnet to the server from another PC INSIDE THE LAN.
If it's not working, it's the firewall on the Exchange server blocking it.
If it IS working, you either have a firewall on the ISP modem/router or somewhere, or your public IP has changed (check with https://whatismyipaddress.com/ from inside the LAN)
Hi Kimputer,

I tested telnet from my pc to the server and it failed  stating "Could not open connection to the host, on port 25: Connect failed"  If the firewall on the exchange server is blocking it is this tied to not receiving external emails?  How do I unblock it? I checked the advanced firewall rules and don't see any exchange rules being blocked for port 25.
In the Advanced Firewall rules, add incoming TCP port 25 and you'll see the PC will be able to telnet to the server. After that, your missing emails will start streaming in (if there are no other issues).
@Kimputer

I added an incoming rule for TCP Port 25.  I tested telnet from my pc on the LAN but still get the "Could not open connection to the host, on port 25: Connect failed" message.

I do not see emails coming in. I sent a test email from Yahoo but nothing has arrived.
I would start by verifying your router has port 25 open and forwarded to the exchange server.
Also, you have a static IP address, correct?  If it's dynamic, it's possible that the IP changed.
Hi Lee,

Port 25 is open and email is forwarded to the exchange server based on packet capture from the network firewall.  Yes, we have a static IP on the server.
Have you reviewed the Event Logs for the server?
Is it possible your antivirus includes a firewall that was turned on?
Telnet from PC in LAN still has to work before you try anything else. Something is still blocking port 25 on the mail server. Can you show screenshots of all the tabs on the TCP port 25 rule in your Windows Advanced Firewall?
I suspect this will fail, but you can also run the Microsoft Remote Connectivity Analyzer: https://testconnectivity.microsoft.com/tests/exchange

As for receive connectors:
Exchange Server 2013 and 2016 uses a concept called ‘Receive Connector.’ You can check for settings of receive connectors and try to remove the error. There is a total of five connectors that are present at mail flow >> receive connectors in Exchange Admin Center. Here are their list and a brief introduction –
Default Front End MBG-EX01: This connector receives all the incoming email messages on port number 25. It further sends the emails to Hub Transport to receive a connector.
Client Front End MBG-EX01: It is a receive connector that established a connection on port 587 and used by IMAP and POP clients. It creates a proxy connection for IMAP and POP program to Hub Transport receive connector.
Default MBG-EX01: It is another hub transport server that takes the emails coming from the front-end transport service. Then it further sends the emails to the mailbox transport service.
Outbound Proxy Front End MBG-EX01: When the user has examined proxy through the client access server option for the send connector, it will receive outgoing emails. It means the transport connector receives from transport service only when the option proxy through client access is examined through a proxy.
Client Proxy MBG-EX01: It is the hub transport connector and connects with port number 465. It receives the IMAP and POP connection from the proxy coming from Client Front End MBG-EX01.
You can check the settings of these connectors and then ask an external user to send an email to your account.
(The above from https://www.kerneldatarecovery.com/blog/fix-exchange-2010-2013-2016-not-receiving-external-emails/)

See also: https://docs.microsoft.com/en-us/exchange/mail-flow/connectors/receive-connectors?view=exchserver-2019



Lee,
I set up the inbound rule based on Kimputer's reply. I selected the default setting and just selected TCP and listed port 25.  Where specifically would I be looking in the event viewer?

I checked the receive connectors and they have not changed and have the ports indicated in the post.  I did not make change changes on the exchange server.

I was upgrading our antivirus for the office but have not updated the client version on the exchange server.

Note: In October I changed the FQDN of the send connector from the servname.domainname to the mail.domainname. This was done to previous the disclosure of the internal IP in the message header.  

Should the FQDN of the send connectors be the same as the receive connectors?  If so, wouldn't I have had an issue before now?
In the Advanced Firewall options, there's the option to log blocked packets. Enable it. See if your telnet attempts show up in the logs.
Kimputer,
Sorry where do I find that settingUser generated image
Just to be sure, set the Profile from Domain, to Domain/Private/Public.
A simple test... Temporarily disable the firewall on the exchange server and quickly test. If you still get a connection error, it isn't the windows firewall... 
The windows firewall settings are managed by the antivirus software. I am reaching out to the vendor so that I can temporarily disable it  and enable the log dropped packets option and to disable it to test telnet from my LAN PC.

If the new inbound rule I added for TCP port 25 is not working,  To be clear on the telnet test I am following https://docs.microsoft.com/en-us/exchange/mail-flow/test-smtp-with-telnet?view=exchserver-2019. Is this correct?

  1. Open a Command Prompt window, type telnet, and then press Enter.
    This command opens the Telnet session.
  2. Type set localecho, and then press Enter.
    This optional command lets you view the characters as you type them, and it might be required for some SMTP servers.
  3. Type set logfile <filename>, and then press Enter.
    This optional command enables logging and specifies the log file for the Telnet session. If you only specify a file name, the log file is located in the current folder. If you specify a path and file name, the path needs to be on the local computer, and you might need to enter the path and file name in the Windows DOS 8.3 format (short name with no spaces). The path needs to exist, but the log file is created automatically.
  4. Type OPEN mail1.fabrikam.com 25, and then press Enter.
  5. Type EHLO contoso.com, and then press Enter.
  6. Type MAIL FROM:<chris@contoso.com>, and then press Enter.
  7. Type RCPT TO:<kate@fabrikam.com> NOTIFY=success,failure, and then press Enter.
    The optional NOTIFY command specifies the particular delivery status notification (DSN) messages (also known as bounce messages, nondelivery reports, or NDRs) that the SMTP is required to provide. In this example, you're requesting a DSN message for successful or failed message delivery.
  8. Type DATA, and then press Enter.
  9. Type Subject: Test from Contoso, and then press Enter.
  10. Press Enter again.
    A blank line is needed between the Subject: field and the message body.
  11. Type This is a test message, and then press Enter.
  12. Type a period ( . ), and then press Enter.
  13. To disconnect from the SMTP server, type QUIT, and then press Enter.
  14. To close the Telnet session, type quit, and then press Enter.

If FIrewall is managed by third party, the steps we went through (Windows Advanced Firewall) will NOT help.
Urgently try to find a way to add port 25 to your current firewall solution.
Please run the step Lee outlined above using the Exchange/Remote Connectivity test:

https://www.experts-exchange.com/questions/29230196/External-emails-not-being-received-since-KB5008207-installed-on-server.html#a43371191 

Restart the NLA service on the Exchange server if the Windows Firewall is set to PUBLIC to get it set to DOMAIN.

Do NOT mess with the Receive Connectors as that will blow things up down the road.

For obvious reasons an anonymous listener is not created on port 25 for internal IPs!

http://blog.mpecsinc.ca/2018/06/exchange-2013-set-up-receive-connector.html
^^^
That's how to set up a scoped anonymous relay for devices like copiers and MFPs.

EDIT: Backpressure warnings will be in the logs if Exchange is refusing because the % of Free Disk Space is too low.
Hi Philip,
Thanks.
Do you mean the inbound SMTP email test on https://testconnectivity.microsoft.com/tests/exchange?  I ran this and it shows:

Testing TCP port 25 on host mail.Domain.com to ensure it's listening and open.
The specified port is either blocked, not listening, or not producing the expected response.

I restarted the Network Location awareness service.  The windows firewall profile is set to Domain.

How do I unblock the port? How would it become blocked in the first place when I didn't make any changes to the server?

The free disk space is 90GB.
First question: Is there a sanitation provider in front of the Exchange server? We set up the router/edge to only allow SMTP 25 and 587 inbound from their subnet.

It sounds like the router/edge may be blocking inbound SMTP?
Or, the Exchange server's IP changed so SMTP port forward rule(s) on the router/edge are no longer working?
@Philip,

Spam filtering is on the exchange server itself. No changes were made to this. I confirmed with our firewall provider that traffic is being forwarded to the exchange server. Traffic is not being blocked at the firewall. Again, no setting changes were made on the network firewall. The exchange IP is static and this has not changed.
You already established there's a third party firewall blocking the traffic on the mail server itself. FOCUS there before focusing on other things. IT's THE MAIN CAUSE you're not receiving mails right now.
Missed that.

We don't put third party anything on servers.

Get rid of it.

If it is a compliance issue, use the built in Windows A/V and tune it for Exchange.
@Philip

The endpoint antivirus has been installed for years without any issue. 
Kimputer,
User generated imageOnce I can get symantec temporarily disabled How do I enable the logging you mentioned?
https://www.experts-exchange.com/questions/29230196/External-emails-not-being-received-since-KB5008207-installed-on-server.html#a43371195 

It appears there was no issue until I installed the updated antivirus client on the Host server.  This should not impact the exchange vm that is on it.  Am I mistaken?


Obviously, it is impacting it. You clearly see that Symantec has taken over control. Figure out to either fully uninstall it, or add the port 25 exception.
If you insist on running third party AV on a server, you at a minimum need to update it when you do server updates as there might be version conflicts...
No guarantee this fixes it, but it could have changed the interactions as many av products link into the kernel level, and big updates can break this.
@Kimputer,

This is not a new setting. This has been on the server since 2019 with the same message and hasn't bee a problem. I would understand if I upgraded the client and the original configuration for the antivirus changed that it would point to that.  In my case I only upgraded the client on the host server not the exchange server.  I'm not seeing how upgrading the client on the host server would impact the exchange server since it has its own client, which has not been updated.

I am at the mercy of symantec as they don't have senior technical support available now and am waiting for someone to get back to me.
Removed my original comment. I'll keep my opinions to myself.

Systems are fluid especially when it comes to updating.

That means that things can be triggered that are unexpected.

Check the A/V firewall logs for DROPPED.
Until Symantec comes through, do research, usually it's remote managed. Therefore, find the Symantec "server" in your server farm, and add a policy for the mail server to allow port 25. Push the policy out ASAP.
@Kimputer

I am still waiting for a reply from Symantec. 
Sorry, when you say add a policy for the mail server to allow port 25., what policy do you mean?
@Kimputer,
I removed the Symantec client from the exchange server so it is no longer in control. I enabled the setting to allow the Log for dropped packets to be Yes.  How do I interpret the log? It is filled with the date -allow-tcp -IP and Send Receive line entries.
If you fully removed Symantec, and gave the server a reboot, the PC in the LAN still can't connect?
@Kimputer

Port 25 still is blocked.
I tried telnet on exchange server itself and get
220 ServerName.DomainName.com Microsoft ESMTP MAIL Service ready at Sun, 19 Dec 2021 10:42:17 -0500
ehlo DomainName.com
501 5.5.4 Invalid domain name

According to https://docs.microsoft.com/en-us/exchange/mail-flow/test-smtp-with-telnet?view=exchserver-2019  it states
Possible reasons for failure:
Invalid characters in the domain name. _(the domain name is correct)Connection restrictions on the destination SMTP server. (This is on the server itself)  How do I remove the block?

when you connected via telnet did you use a valid domain name in your EHLO command and not a made up one?
On exchange server can you do dns lookups that resolve?
IE... In a command prompt if you do something like "nslookup microsoft.com" does it give you an ip address?

First we still have to unblock port 25 before we analyze the other things. If you fully disable the Windows Firewall, you still can't telnet to it from PC in the LAN?
@Scott

The results of the telnet on the server:

220 ServerName.DomainName.com Microsoft ESMTP MAIL Service ready at Sun, 19 Dec 2021 11:08:04 -0500
ehlo pc
250-ServerName.DomainName.com Hello [IP]
250-SIZE 37748736
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-X-ANONYMOUSTLS
250-AUTH NTLM
250-X-EXPS GSSAPI NTLM
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 XRDST

yes DNS is working properly. This wasn't an issue before Friday 12/17. Yes I can run nslookup  google.com and it returns a valid ip.
@Kimputer,

I just tested telnet after fully disabling the windows firewall on the server but still cannot telnet from my PC on the LAN.
I verified with the  firewall vendor that traffic is being forwarded to the internal exchange server. This was seen with a packet capture. I would be the only one to make changes on the Network firewall but didn't make any. 
The issue only appeared Friday night seemingly after the KB installation.  

**The one thing I did change was to install the IP address and Domain Restrictions role on 11/15.  This was based on https://www.alitajran.com/disable-external-access-to-ecp-exchange-2016/ .  I followed those instructions but applied it to EWS so that the prompt to log on would not be accessible when trying to access it from outside the LAN. I added the deny and allow entry as indicated. 

Remove everything done via https://www.alitajran.com/disable-external-access-to-ecp-exchange-2016/

If you want to remove external access use ARR (Application Request Routing) and URLReWrite to set up a rule for mail.domain.com/ecp to redirect to a 404 page.

I suggest never making out of the box changes to Exchange. It does not like that.
@Philip

I changed the setting back to allow and restarted the Transport service. I tested telnet from my PC but still cannot connect.
This is beyond a Q&A forum like this.

Do you have an Exchange expert you can reach out to who can get hand's on?
@Philip.

Understood. We don't have an Exchange expert but contacting one will be my next step. Thanks.
One more thing to try just in case Symantec didn't fully remove as it has been known to do in the past.

  1. Run the following commands at a command prompt that is run as administrator
    • netsh winsock reset
    • netsh int ip reset
    • ipconfig /flushdns
  2. Reboot

Hello all,
Thank you for your help.  I have confirmed the issue is definitely related to installing the new version of Symantec Endpoint Protection 14.3 build 4637 on the host server. I removed that version and confirmed that:

* external inbound email was working fine
* OWA once again working fine- when I first installed the new AV version on the host server and rebooted it, I had an issue opening the OWA on my PC both on and LAN and a personal PC
* outlook client working fine- when I first installed the new AV version on the host server and rebooted it, I had an issue opening the Outlook client on my PC.

I am working with Symantec to try to speak to a senior technician to address exceptions that need to be in placed to prevent port 25 from being blocked.

Thanks again for your time.

ASKER CERTIFIED SOLUTION
Avatar of Roger Vallee
Roger Vallee
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial