Windows 2012R2 SMTP service slow queue

Just a few questions:

1.  Can you post some SMTP logs from around the time when the queue fills up?
1a.  Are there any Application Event Log warnings or errors?
2.  Does the SMTP Server take inbound emails or only relays outbound?
3.  How is the SMTP relay configured?  What servers are allowed to use the SMTP Service as a relay?
4.  What are you running for an internal email server?  (Exchange or some other MAPI/IMAP service)
5.  Is there queuing at the MAPI/IMAP endpoint as well?  (server in point 4)

Dan

Seems that issue is with our Exchange 2013 servers.
Ever since we have upgraded to CU12 we started experiencing this issue.

So internal mail relay servers and external spam provider causing delays to our Exchange 2013 CU12 servers.  

found someone else with same problem with CU12, but no solution:
https://social.technet.microsoft.com/Forums/office/en-US/caf4d3f0-6e06-4c03-86fa-99c39ac83231/external-email-issue-following-exchange-2013-cu12-upgrade?forum=exchangesvrsecuremessaging


To answer your questions, I dont think its Windows 2012R2 SMTP server as I thought previously seems that its Exchange 2013CU12 that gets too busy with accepting mails.

Any suggestions greatly appreciated!

Are you seeing any back pressure on your Exchange servers?

Link:  https://technet.microsoft.com/en-us/library/bb201658(v=exchg.160).aspx

In the event logs, do you see any of these event ids:  15004, 15006 or 15007?

If you run the Powershell command referenced in the above link (Get-ExchangeDiagnosticInfo), what does the results say?

Dan

mail.que file is: 73GB    (seems very large for some reason)    -  we have enough space on disks though.
If I look at Exchange Queue with Exchange Queue Viewer, it is empty all mails flush fine from there.

Get-ExchangeDiagnosticInfo results:

[PS] C:\Windows\system32>Get-ExchangeDiagnosticInfo


RunspaceId  : 78c03cec-7bcf-4fb9-ad8b-f04f6785c5e7
Result      : <Diagnostics>
                <ProcessLocator>
                  <count>31</count>
                  <Process>
                    <guid>7944c6fd-dfe8-4619-b121-ee1035f0fd66</guid>
                    <id>5328</id>
                    <name>MSExchangeDelivery</name>
                  </Process>
                  <Process>
                    <guid>975c3e33-9644-40bc-a08c-cb6e531da5bb</guid>
                    <id>40600</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>885fe659-e2a2-442f-be50-96631f69ac29</guid>
                    <id>37768</id>
                    <name>MSExchangeHMWorker</name>
                  </Process>
                  <Process>
                    <guid>6c15a30f-2b44-4d46-9987-5f283b2a33f6</guid>
                    <id>3916</id>
                    <name>MSExchangeECPAppPool</name>
                  </Process>
                  <Process>
                    <guid>66e35e08-6dfb-439d-884a-6868d06ef733</guid>
                    <id>21856</id>
                    <name>MSExchangeHMHost</name>
                  </Process>
                  <Process>
                    <guid>b390ff6b-f4d9-4739-ba23-ac68c7fe0c53</guid>
                    <id>5380</id>
                    <name>noderunner</name>
                  </Process>
                  <Process>
                    <guid>8ebbf9c6-8342-477d-8fe1-08b729aea4d8</guid>
                    <id>14588</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>6394c786-24be-427e-98df-e973e00937b4</guid>
                    <id>14680</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>df3577c8-ca70-4062-9216-43516965c76f</guid>
                    <id>14924</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>55c2500c-bafa-46b9-9622-7ab2cf25c12f</guid>
                    <id>14580</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>27c100b7-4f44-4cd1-933d-c425acbca47e</guid>
                    <id>14528</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>ecc5160c-3a2e-493a-89c3-a72f122a365d</guid>
                    <id>13348</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>e2c0b7e2-d9f6-4cc1-a5d9-a4bfc4fa8949</guid>
                    <id>4664</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>5dbebd22-b844-448e-9ecb-96ab82fec4e0</guid>
                    <id>12564</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>72afb2e7-82ed-45c2-a0ba-d1eba2c301c3</guid>
                    <id>6240</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>b8b04533-779b-46e9-9fba-de7dab64a1b0</guid>
                    <id>13368</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>a19d10a2-a29c-41e5-b847-cd52574ff225</guid>
                    <id>13444</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>cebdedb8-ba53-49c5-a7c9-041ec3f7eb01</guid>
                    <id>12580</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>6d67dd4e-68c7-4323-a6ab-3cc9ccde9369</guid>
                    <id>12728</id>
                    <name>Microsoft.Exchange.Store.Worker</name>
                  </Process>
                  <Process>
                    <guid>c249ff49-1c42-4123-9b8f-67bb37b526cc</guid>
                    <id>3524</id>
                    <name>EdgeTransport</name>
                  </Process>
                  <Process>
                    <guid>e4fb7fae-48bf-4662-acde-799a75053991</guid>
                    <id>1108</id>
                    <name>MSExchangeSyncAppPool</name>
                  </Process>
                  <Process>
                    <guid>7e35a6fb-5a82-4624-b4df-fc295f1bbc67</guid>
                    <id>9764</id>
                    <name>MSExchangeSubmission</name>
                  </Process>
                  <Process>
                    <guid>51eb74c5-7136-469d-a4b8-b2ee1fd20b34</guid>
                    <id>8700</id>
                    <name>MSExchangeMailboxReplication</name>
                  </Process>
                  <Process>
                    <guid>6d0a41f9-88b6-4777-8fea-d96cb8152fd5</guid>
                    <id>9596</id>
                    <name>Microsoft.Exchange.ServiceHost</name>
                  </Process>
                  <Process>
                    <guid>f32d758c-b909-48f2-973c-475e59aa48fa</guid>
                    <id>9352</id>
                    <name>Microsoft.Exchange.RpcClientAccess.Service</name>
                  </Process>
                  <Process>
                    <guid>26296986-c569-4406-b418-b434d8206ca3</guid>
                    <id>8980</id>
                    <name>Microsoft.Exchange.Store.Service</name>
                  </Process>
                  <Process>
                    <guid>17d99450-1e74-49c4-acdf-c9856ff0c4d8</guid>
                    <id>7816</id>
                    <name>MSExchangeMailboxAssistants</name>
                  </Process>
                  <Process>
                    <guid>0a8f8b49-1757-44a8-8f53-28dc4eef85e2</guid>
                    <id>8308</id>
                    <name>MSExchangeFrontendTransport</name>
                  </Process>
                  <Process>
                    <guid>9f51d229-ae72-4782-8d60-727f097ffb93</guid>
                    <id>7944</id>
                    <name>Microsoft.Exchange.Search.Service</name>
                  </Process>
                  <Process>
                    <guid>15e48764-a434-4717-ab64-ee59792f8720</guid>
                    <id>5616</id>
                    <name>MSExchangeServicesAppPool</name>
                  </Process>
                  <Process>
                    <guid>dcde0603-c283-4c12-8ed9-eec6168ae911</guid>
                    <id>6244</id>
                    <name>MSExchangeAutodiscoverAppPool</name>
                  </Process>
                </ProcessLocator>
              </Diagnostics>
Identity    :
IsValid     : True
ObjectState : New



[PS] C:\Windows\system32>

Select allOpen in new window

RAM in the Server?

73GB for the mail.que file is nothing.  The size of the file is controlled by 2 things:  your Safety Net configuration and the number of emails flowing thru your system in the last 24 hours.

My mail.que files typically sit around 200-300GB.

Are there any event log warnings or errors to be found?

Dan

Last Error: 452 4.3.1 Insufficient system resources  (found this on the delivery transport server)  

so seems that our exchange 2013 accepts whenever,  very intermittently.

48GB memory - used 36GB
4 Sockets - 16 total cores.

cant find any other errors except on other servers trying to submit to Exch2013CU12:
Last Error: 452 4.3.1 Insufficient system resources

Also, is IPv6 enabled but in a default configuration on your servers?

Can you post your Front-End receive connector config?  Are you using the default/generic setting of all addresses or have your set a specific IP address for the binding?

Dan

Can you look into the event logs for the event ids I mentioned above.  If there is back pressure, there are specific steps to take to resolve the resource issues.

Dan

found with 15004

The resource pressure increased from Medium to High.

The following resources are under pressure:
Version buckets = 212 [High] [Normal=80 Medium=120 High=200]
Physical memory load = 89% [limit is 94% to start dehydrating messages.]

The following components are disabled due to back pressure:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from Pickup directory
Mail submission from Replay directory
Mail submission from Mailbox server
Mail delivery to remote domains
Content aggregation
Mail resubmission from the Message Resubmission component.
Mail resubmission from the Shadow Redundancy Component

The following resources are in normal state:
Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 63% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 64% [Normal] [Normal=95% Medium=97% High=99%]
Private bytes = 3% [Normal] [Normal=71% Medium=73% High=75%]
Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
Temporary Storage disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 64% [Normal] [Normal=95% Medium=97% High=99%]

Select allOpen in new window

We have not touched IPv6 configuration since installation of OS, so that is default.

Dan, thanks very much for all your help!

So we have 2 mailbox servers in DAG setup.
Never thought to check passive server, and that server was running less than 10% of space, that was affecting the active server.   ONce i have cleared space on Passive server everything started working smoothly.
Just never thought to check passive, was troubleshooting active and didnt think it would affect primary.

Thanks again for all the help!

Issue was with our Exchange 2013 receiving server.    We have 2 mailbox Exchange2013 servers in DAG, one of them was running low space that affected primary as well not to accept any emails or accept when space frees up to keep it under control.