David Pyman
asked on
Exchange 2010 Transport service - Mail.que growing rapidly to eventual shutdown of service
Exchange 2010 on an SBS 2011 server, starting a few days ago, the transport DB mail.que is growing to ~5GB at which time the Exchange transport moves the Queue folder contents to a subfolder Queue.old and proceeds again with rapid expansion of the mail.que file, along with ~1,000 .log files. The Transport service will steadily increase in CPU and Memory usage. Once the mail.que file hits ~ 5GB the second time, the service stops. The service does however change status in services.msc, to blank. I disable the service and reboot the server, rename the Queue folder, restart the transport service and the cycle starts over. The entire cycle of events takes 1 - 2 hours to complete.
During the time that the Transport services is running, mail moves in and out, running a Get-Queue at various times during the 1-2 hours shows mail moving through without issue the queue is empty most of the time once it processes through the stacked up messages from being down.
Following are the items attempted/addressed so far, in chronological order:
Verified no AV is loaded or scanning now.
Verified Drive space is no issue
Mail.que stops growing when transport service is stopped.
Edited the Edgetransport.config to move the QueueDatabasePath and QueueDatabaseLoggingPath to another partition.
Downloaded and Scanned the system with Malwarebytes, no issues found, uninstalled.
I am familiar in Exchange Admin but not an expert by any means.
Thanks,
Dave
During the time that the Transport services is running, mail moves in and out, running a Get-Queue at various times during the 1-2 hours shows mail moving through without issue the queue is empty most of the time once it processes through the stacked up messages from being down.
Following are the items attempted/addressed so far, in chronological order:
Verified no AV is loaded or scanning now.
Verified Drive space is no issue
Mail.que stops growing when transport service is stopped.
Edited the Edgetransport.config to move the QueueDatabasePath and QueueDatabaseLoggingPath to another partition.
Downloaded and Scanned the system with Malwarebytes, no issues found, uninstalled.
I am familiar in Exchange Admin but not an expert by any means.
Thanks,
Dave
You can enable verbose logging on your send connector and see what is going on.
I advise you to start complete server and check again.
ASKER
I have enabled verbose logging on the send connector.
Amit,
I have restarted the server a few times as the transport process hung and could not be killed via procmon.
THanks,
Amit,
I have restarted the server a few times as the transport process hung and could not be killed via procmon.
THanks,
Receive connector too, I hope.
ASKER
Verbose logging on the send and receive connectors is enabled.
There is a 1024 that occurs after every restart of the transport service:
EVENT 1024 - MailSubmission has detected the Hub Transport server XXXXXXX state change from dd0f88a7c670450ca3b51010dc
Many Event 106 errors for various counters, examples below:
Performance counter updating error. Counter name is Temporary Submission Failures, category name is MSExchange Mail Submission. Optional code: 1. Exception: The exception thrown is : System.InvalidOperationExc
at System.Diagnostics.Perform
at System.Diagnostics.Perform
at Microsoft.Exchange.Diagnos
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
at System.Diagnostics.Process
at Microsoft.Exchange.Diagnos
Performance counter updating error. Counter name is MDB Health: Database Replication Lagging, category name is MSExchange Mailbox Replication Service Per Mdb. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationExc
at System.Diagnostics.Perform
at System.Diagnostics.Perform
at Microsoft.Exchange.Diagnos
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
at System.Diagnostics.Process
at Microsoft.Exchange.Diagnos
Processes running while Performance counter failed to update:
There is a 1024 that occurs after every restart of the transport service:
EVENT 1024 - MailSubmission has detected the Hub Transport server XXXXXXX state change from dd0f88a7c670450ca3b51010dc
Many Event 106 errors for various counters, examples below:
Performance counter updating error. Counter name is Temporary Submission Failures, category name is MSExchange Mail Submission. Optional code: 1. Exception: The exception thrown is : System.InvalidOperationExc
at System.Diagnostics.Perform
at System.Diagnostics.Perform
at Microsoft.Exchange.Diagnos
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
at System.Diagnostics.Process
at Microsoft.Exchange.Diagnos
Performance counter updating error. Counter name is MDB Health: Database Replication Lagging, category name is MSExchange Mailbox Replication Service Per Mdb. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationExc
at System.Diagnostics.Perform
at System.Diagnostics.Perform
at Microsoft.Exchange.Diagnos
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
at System.Diagnostics.Process
at Microsoft.Exchange.Diagnos
Processes running while Performance counter failed to update:
Any July patches installed that could affect mail flow?
ASKER
The patch installation date did not correlate well with the start of the issues, (8-21, 4pm) but I could uninstall to verify.
No need to uninstall just verify if you have the affected patch and load the one to fix it.
The Event 1024 surprises me a little. Since you said SBS 2011, I presumed there is only a single Exchange server. Is that correct?
ASKER
Correct, one SBS box.
ASKER
Here's the latest from the weekend efforts:
- Exchange 2010 was at SP2 R8, I upgraded to SP3, Rollup 19.
- Ran BPA, no issues.
- unplugged ethernet to server, no effect. Que continued to grow.
- dismounted mail stores, no no effect. Que continued to grow.
- Scanned with Malwarebytes and Microsoft Safety Scanner, clean.
- reviewed smtp send and receive logs, look to be expected traffic. Not very large.
- loaded Exchange User Monitor, I have one mailbox with consistent large bytes out. I disabled the user in AD and made sure the PC is turned off. iPhone email profile deleted off the phone. data for this user/mailbox is still showing up on refresh in EUM.
Not sure where to go next, any recommendations would be appreciated.
- Exchange 2010 was at SP2 R8, I upgraded to SP3, Rollup 19.
- Ran BPA, no issues.
- unplugged ethernet to server, no effect. Que continued to grow.
- dismounted mail stores, no no effect. Que continued to grow.
- Scanned with Malwarebytes and Microsoft Safety Scanner, clean.
- reviewed smtp send and receive logs, look to be expected traffic. Not very large.
- loaded Exchange User Monitor, I have one mailbox with consistent large bytes out. I disabled the user in AD and made sure the PC is turned off. iPhone email profile deleted off the phone. data for this user/mailbox is still showing up on refresh in EUM.
Not sure where to go next, any recommendations would be appreciated.
ASKER
Here are results from the Exchange Troubleshooting Assistant. Looks like the one mailbox/user is causing the issue, I have not found a method to resolve.
The user 'Tim' is using 46.48% of the CPU in MAPI operation "ReadStream". The user issued 55698 requests of this type.
The user 'Tim' is using 53.52% of the CPU in MAPI operation "End". The user issued 0 requests of this type.
The user 'Tim' is using 46.48% of the CPU in MAPI operation "ReadStream". The user issued 55698 requests of this type.
The user 'Tim' is using 53.52% of the CPU in MAPI operation "End". The user issued 0 requests of this type.
I'd like to see your Application event log and the send/receive protocol logs. I realize that is not practical here.
I don't have any more good ideas. :-(
I don't have any more good ideas. :-(
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
But generally, this sounds like someone is trying to send a multi-gigabyte file by email. If you have Verbose logging turned on for your receive connectors, you should be able to obtain more details on this in those logs. Also, netstat.exe and TCPView (Sysinternals) can let you look at potential IP addresses.