asked on
External emails not being received since KB5008207 installed on server
Hello,
After installing KB5008207 external email is not coming in.
I installed KB5008207 on the exchange server on premises. This was the update required for the Windows Server 2016 server on which it is installed. Normally after I install patches from the Patch Tuesday for each month I do the following:
1. Confirm that the Microsoft Exchange services are started
2. Confirm I can access the EAC
3. Confirm that the database is mounted and is in the healthy state
4. Confirm I can access my outlook client as usual and that it shows "all folders up to date"
However, yesterday after installing the KB5008207 on the server:
* I noticed that I could not open my Outlook client on my machine. It is unclear if this issue is also related to the issue above. I confirmed that 1-3 above are OK.
* I logged onto another PC and confirmed I could send email internally to staff and externally to Gmail. However when I tested sending a test email from Gmail to my company email it was not received.
* I checked the queue viewer and do not see emails in the queue.
* I received a bounce message when I tried to send my self a message from yahoo but it just states it was unable to deliver after multiple tries.
In the past this happened once and it was due to low disk space on the Exchange server. I confirmed we had 62GB of space available but I increased it to 92GB. I read it was recommended to restart the Exchange Transport service and did this. However as of yesterday evening external emails are not being received.
This seems to be tied to the KB5008207 unless it is coincidental as there were no issues with Exchange until after the update was installed. No configuration changes of any kind were made to Exchange before this issue started last night.
I appreciate any help that could be provided as I need to get external emails working again.
Thank you.
I used the Microsoft Remote Connectivity Analyzer to Test inbound SMTP mail flow for domain (my mailbox) and it shows The specified port is either blocked, not listening, or not producing the expected response .
How do I fix this?
ASKER
Thank you. I just tested with telnet on the exchange server and get:
220 ExchangeServer.DomainName Microsoft ESMTP MAIL Service ready at Sat, 18 Dec 2021 10:02:18 -0500
When you say firewall issue, do you mean the firewall protecting the network or the firewall on the exchange server? How could this suddenly be a problem if everything was working until last night when the update was installed?
What would need to be done on the firewall to fix this? Are you saying that something would have changed on the firewall? I didn't change anything on the firewall and it would only be me that would make changes. I'm just trying to understand how this issue happened and what needs to be done to fix it.
Thank you.
**I confirmed with the firewall provider that there are no issues as traffic is being forwarded to the exchange server. So from the firewall side this does not appear to be the problem.
If yes, then your receive connectors are working and it points to either the transport service, the mailbox transport service or the information store service.
Do you get a failure message from gmail?
Are you running internal anti-spam on exchange?
Since you say internal works I think that eliminates mailbox transport.
Also double check your receive connector, and double check that your fqdn resolves to the right address from outside.
ASKER
I tested telnet on the exchange server itself. How do I test telnet from an outside address?
The failure message I received from Yahoo when trying to send to my company email just "Unable to deliver message after multiple retries, giving up." No codes in the message itself.
Yes anti-spam is being run on the exchange itself but this didn't change and I haven't made any changes.
Yes, sending emails between staff internally works oK.
You mention the receive connector and to check that the fQDN resolves to the right address from outside. Can you elaborate?
Right now the FQDN in the scoping section for each receive connector is the servername.domainname.
Note: In October I changed the FQDN of the send connector from the servname.domainname to the mail.domainname. This was done to previous the disclosure of the internal IP in the message header.
Should the FQDN of the send connectors be the same as the receive connectors? If so, wouldn't I have had an issue before now?
Thanks.
"
If it's not working, it's the firewall on the Exchange server blocking it.
If it IS working, you either have a firewall on the ISP modem/router or somewhere, or your public IP has changed (check with https://whatismyipaddress.com/ from inside the LAN)
ASKER
I tested telnet from my pc to the server and it failed stating "Could not open connection to the host, on port 25: Connect failed" If the firewall on the exchange server is blocking it is this tied to not receiving external emails? How do I unblock it? I checked the advanced firewall rules and don't see any exchange rules being blocked for port 25.
ASKER
I added an incoming rule for TCP Port 25. I tested telnet from my pc on the LAN but still get the "Could not open connection to the host, on port 25: Connect failed" message.
I do not see emails coming in. I sent a test email from Yahoo but nothing has arrived.
Also, you have a static IP address, correct? If it's dynamic, it's possible that the IP changed.
ASKER
Port 25 is open and email is forwarded to the exchange server based on packet capture from the network firewall. Yes, we have a static IP on the server.
Is it possible your antivirus includes a firewall that was turned on?
As for receive connectors:
Exchange Server 2013 and 2016 uses a concept called ‘Receive Connector.’ You can check for settings of receive connectors and try to remove the error. There is a total of five connectors that are present at mail flow >> receive connectors in Exchange Admin Center. Here are their list and a brief introduction –
Default Front End MBG-EX01: This connector receives all the incoming email messages on port number 25. It further sends the emails to Hub Transport to receive a connector.
Client Front End MBG-EX01: It is a receive connector that established a connection on port 587 and used by IMAP and POP clients. It creates a proxy connection for IMAP and POP program to Hub Transport receive connector.
Default MBG-EX01: It is another hub transport server that takes the emails coming from the front-end transport service. Then it further sends the emails to the mailbox transport service.
Outbound Proxy Front End MBG-EX01: When the user has examined proxy through the client access server option for the send connector, it will receive outgoing emails. It means the transport connector receives from transport service only when the option proxy through client access is examined through a proxy.
Client Proxy MBG-EX01: It is the hub transport connector and connects with port number 465. It receives the IMAP and POP connection from the proxy coming from Client Front End MBG-EX01.
You can check the settings of these connectors and then ask an external user to send an email to your account.
(The above from https://www.kerneldatarecovery.com/blog/fix-exchange-2010-2013-2016-not-receiving-external-emails/)
See also: https://docs.microsoft.com/en-us/exchange/mail-flow/connectors/receive-connectors?view=exchserver-2019
ASKER
I set up the inbound rule based on Kimputer's reply. I selected the default setting and just selected TCP and listed port 25. Where specifically would I be looking in the event viewer?
I checked the receive connectors and they have not changed and have the ports indicated in the post. I did not make change changes on the exchange server.
I was upgrading our antivirus for the office but have not updated the client version on the exchange server.
Note: In October I changed the FQDN of the send connector from the servname.domainname to the mail.domainname. This was done to previous the disclosure of the internal IP in the message header.
Should the FQDN of the send connectors be the same as the receive connectors? If so, wouldn't I have had an issue before now?
ASKER
If the new inbound rule I added for TCP port 25 is not working, To be clear on the telnet test I am following https://docs.microsoft.com/en-us/exchange/mail-flow/test-smtp-with-telnet?view=exchserver-2019. Is this correct?
- Open a Command Prompt window, type telnet, and then press Enter.
This command opens the Telnet session. - Type set localecho, and then press Enter.
This optional command lets you view the characters as you type them, and it might be required for some SMTP servers. - Type set logfile <filename>, and then press Enter.
This optional command enables logging and specifies the log file for the Telnet session. If you only specify a file name, the log file is located in the current folder. If you specify a path and file name, the path needs to be on the local computer, and you might need to enter the path and file name in the Windows DOS 8.3 format (short name with no spaces). The path needs to exist, but the log file is created automatically. - Type OPEN mail1.fabrikam.com 25, and then press Enter.
- Type EHLO contoso.com, and then press Enter.
- Type MAIL FROM:<chris@contoso.com>, and then press Enter.
- Type RCPT TO:<kate@fabrikam.com> NOTIFY=success,failure, and then press Enter.
The optional NOTIFY command specifies the particular delivery status notification (DSN) messages (also known as bounce messages, nondelivery reports, or NDRs) that the SMTP is required to provide. In this example, you're requesting a DSN message for successful or failed message delivery. - Type DATA, and then press Enter.
- Type Subject: Test from Contoso, and then press Enter.
- Press Enter again.
A blank line is needed between the Subject: field and the message body. - Type This is a test message, and then press Enter.
- Type a period ( . ), and then press Enter.
- To disconnect from the SMTP server, type QUIT, and then press Enter.
- To close the Telnet session, type quit, and then press Enter.
Urgently try to find a way to add port 25 to your current firewall solution.
https://www.experts-exchange.com/questions/29230196/External-emails-not-being-received-since-KB5008207-installed-on-server.html#a43371191
Restart the NLA service on the Exchange server if the Windows Firewall is set to PUBLIC to get it set to DOMAIN.
Do NOT mess with the Receive Connectors as that will blow things up down the road.
For obvious reasons an anonymous listener is not created on port 25 for internal IPs!
http://blog.mpecsinc.ca/2018/06/exchange-2013-set-up-receive-connector.html
^^^
That's how to set up a scoped anonymous relay for devices like copiers and MFPs.
EDIT: Backpressure warnings will be in the logs if Exchange is refusing because the % of Free Disk Space is too low.
ASKER
Thanks.
Do you mean the inbound SMTP email test on https://testconnectivity.microsoft.com/tests/exchange? I ran this and it shows:
Testing TCP port 25 on host mail.Domain.com to ensure it's listening and open.
The specified port is either blocked, not listening, or not producing the expected response.
I restarted the Network Location awareness service. The windows firewall profile is set to Domain.
How do I unblock the port? How would it become blocked in the first place when I didn't make any changes to the server?
The free disk space is 90GB.
It sounds like the router/edge may be blocking inbound SMTP?
Or, the Exchange server's IP changed so SMTP port forward rule(s) on the router/edge are no longer working?
ASKER
Spam filtering is on the exchange server itself. No changes were made to this. I confirmed with our firewall provider that traffic is being forwarded to the exchange server. Traffic is not being blocked at the firewall. Again, no setting changes were made on the network firewall. The exchange IP is static and this has not changed.
We don't put third party anything on servers.
Get rid of it.
If it is a compliance issue, use the built in Windows A/V and tune it for Exchange.
ASKER
The endpoint antivirus has been installed for years without any issue.
ASKER
Once I can get symantec temporarily disabled How do I enable the logging you mentioned?
https://www.experts-exchange.com/questions/29230196/External-emails-not-being-received-since-KB5008207-installed-on-server.html#a43371195
It appears there was no issue until I installed the updated antivirus client on the Host server. This should not impact the exchange vm that is on it. Am I mistaken?
No guarantee this fixes it, but it could have changed the interactions as many av products link into the kernel level, and big updates can break this.
ASKER
This is not a new setting. This has been on the server since 2019 with the same message and hasn't bee a problem. I would understand if I upgraded the client and the original configuration for the antivirus changed that it would point to that. In my case I only upgraded the client on the host server not the exchange server. I'm not seeing how upgrading the client on the host server would impact the exchange server since it has its own client, which has not been updated.
I am at the mercy of symantec as they don't have senior technical support available now and am waiting for someone to get back to me.
Systems are fluid especially when it comes to updating.
That means that things can be triggered that are unexpected.
Check the A/V firewall logs for DROPPED.
ASKER
I am still waiting for a reply from Symantec.
Sorry, when you say add a policy for the mail server to allow port 25., what policy do you mean?
ASKER
I removed the Symantec client from the exchange server so it is no longer in control. I enabled the setting to allow the Log for dropped packets to be Yes. How do I interpret the log? It is filled with the date -allow-tcp -IP and Send Receive line entries.
ASKER
Port 25 still is blocked.
I tried telnet on exchange server itself and get
220 ServerName.DomainName.com Microsoft ESMTP MAIL Service ready at Sun, 19 Dec 2021 10:42:17 -0500
ehlo DomainName.com
501 5.5.4 Invalid domain name
According to https://docs.microsoft.com/en-us/exchange/mail-flow/test-smtp-with-telnet?view=exchserver-2019 it states
Possible reasons for failure:
Invalid characters in the domain name. _(the domain name is correct)Connection restrictions on the destination SMTP server. (This is on the server itself) How do I remove the block?
On exchange server can you do dns lookups that resolve?
IE... In a command prompt if you do something like "nslookup microsoft.com" does it give you an ip address?
ASKER
The results of the telnet on the server:
220 ServerName.DomainName.com Microsoft ESMTP MAIL Service ready at Sun, 19 Dec 2021 11:08:04 -0500
ehlo pc
250-ServerName.DomainName.com Hello [IP]
250-SIZE 37748736
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-X-ANONYMOUSTLS
250-AUTH NTLM
250-X-EXPS GSSAPI NTLM
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 XRDST
yes DNS is working properly. This wasn't an issue before Friday 12/17. Yes I can run nslookup google.com and it returns a valid ip.
ASKER
I just tested telnet after fully disabling the windows firewall on the server but still cannot telnet from my PC on the LAN.
ASKER
The issue only appeared Friday night seemingly after the KB installation.
**The one thing I did change was to install the IP address and Domain Restrictions role on 11/15. This was based on https://www.alitajran.com/disable-external-access-to-ecp-exchange-2016/ . I followed those instructions but applied it to EWS so that the prompt to log on would not be accessible when trying to access it from outside the LAN. I added the deny and allow entry as indicated.
If you want to remove external access use ARR (Application Request Routing) and URLReWrite to set up a rule for mail.domain.com/ecp to redirect to a 404 page.
I suggest never making out of the box changes to Exchange. It does not like that.
ASKER
I changed the setting back to allow and restarted the Transport service. I tested telnet from my PC but still cannot connect.
Do you have an Exchange expert you can reach out to who can get hand's on?
ASKER
Understood. We don't have an Exchange expert but contacting one will be my next step. Thanks.
- Run the following commands at a command prompt that is run as administrator
- netsh winsock reset
- netsh int ip reset
- ipconfig /flushdns
- Reboot
ASKER
Thank you for your help. I have confirmed the issue is definitely related to installing the new version of Symantec Endpoint Protection 14.3 build 4637 on the host server. I removed that version and confirmed that:
* external inbound email was working fine
* OWA once again working fine- when I first installed the new AV version on the host server and rebooted it, I had an issue opening the OWA on my PC both on and LAN and a personal PC
* outlook client working fine- when I first installed the new AV version on the host server and rebooted it, I had an issue opening the Outlook client on my PC.
I am working with Symantec to try to speak to a senior technician to address exceptions that need to be in placed to prevent port 25 from being blocked.
Thanks again for your time.
I realize there's a lot of back and forth but that's what I said in the linked comment from above?