Exchange 2010 Transport service - Mail.que growing rapidly to eventual shutdown of service

Exchange 2010 on an SBS 2011 server, starting a few days ago, the transport DB mail.que is growing to ~5GB at which time the Exchange transport moves the  Queue folder contents to  a subfolder Queue.old and proceeds again with rapid expansion of the mail.que file, along with ~1,000 .log files.  The Transport service will steadily increase in CPU and Memory usage.  Once the mail.que file hits ~ 5GB the second time, the service stops.   The service does however change status in services.msc, to blank.   I disable the service and reboot the server, rename the Queue folder, restart the transport service and the cycle starts over.  The entire cycle of events takes 1 - 2 hours to complete.

During the time that the Transport services is running, mail moves in and out, running a Get-Queue at various times during the 1-2 hours shows mail moving through without issue the queue is empty most of the time once it processes through the stacked up messages from being down.

Following are the items attempted/addressed so far, in chronological order:

Verified no AV is loaded or scanning now.
Verified Drive space is no issue
Mail.que stops growing when transport service is stopped.
Edited the Edgetransport.config to move the QueueDatabasePath and QueueDatabaseLoggingPath to another partition.
Downloaded and Scanned the system with Malwarebytes, no issues found, uninstalled.

I am familiar in Exchange Admin but not an expert by any means.

Thanks,

Dave
David PymanAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Michael B. SmithExchange & Active Directory ExpertCommented:
There should be a number of events in the Application log that describe what is going on with Transport. I'd spend some time there, first.

But generally, this sounds like someone is trying to send a multi-gigabyte file by email. If you have Verbose logging turned on for your receive connectors, you should be able to obtain more details on this in those logs. Also, netstat.exe and TCPView (Sysinternals) can let you look at potential IP addresses.
0
Edward van BiljonMessaging and Collaboration Technical Lead (Exchange MVP)Commented:
You can enable verbose logging on your send connector and see what is going on.
0
AmitIT ArchitectCommented:
I advise you to start complete server and check again.
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

David PymanAuthor Commented:
I have enabled verbose logging on the send connector.

Amit,

  I have restarted the server a few times as the transport process hung and could not be killed via procmon.

THanks,
0
Michael B. SmithExchange & Active Directory ExpertCommented:
Receive connector too, I hope.
0
David PymanAuthor Commented:
Verbose logging on the send and receive connectors is enabled.  

There is a 1024 that occurs after every restart of the transport service:

EVENT 1024 - MailSubmission has detected the Hub Transport server XXXXXXX state change from dd0f88a7c670450ca3b51010dcd7d985 to 7d66161139384332980cf2961b6918e7, moving 1 shadow messages for resubmission.

Many Event 106 errors for various counters, examples below:
Performance counter updating error. Counter name is Temporary Submission Failures, category name is MSExchange Mail Submission. Optional code: 1. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
   at System.Diagnostics.PerformanceCounter.Initialize()
   at System.Diagnostics.PerformanceCounter.IncrementBy(Int64 value)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.IncrementBy(Int64 incrementValue)
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
   at System.Diagnostics.Process.GetProcessById(Int32 processId, String machineName)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo()


Performance counter updating error. Counter name is MDB Health: Database Replication Lagging, category name is MSExchange Mailbox Replication Service Per Mdb. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
   at System.Diagnostics.PerformanceCounter.Initialize()
   at System.Diagnostics.PerformanceCounter.get_RawValue()
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value)
Last worker process info : System.ArgumentException: Process with an Id of 15860 is not running.
   at System.Diagnostics.Process.GetProcessById(Int32 processId, String machineName)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo()
Processes running while Performance counter failed to update:
0
Edward van BiljonMessaging and Collaboration Technical Lead (Exchange MVP)Commented:
Any July patches installed that could affect mail flow?
0
David PymanAuthor Commented:
The patch installation date did not correlate well with the start of the issues, (8-21, 4pm) but I could uninstall to verify.
0
Edward van BiljonMessaging and Collaboration Technical Lead (Exchange MVP)Commented:
No need to uninstall just verify if you have the affected patch and load the one to fix it.
0
Michael B. SmithExchange & Active Directory ExpertCommented:
The Event 1024 surprises me a little. Since you said SBS 2011, I presumed there is only a single Exchange server. Is that correct?
0
David PymanAuthor Commented:
Correct, one SBS box.
0
David PymanAuthor Commented:
Here's the latest from the weekend efforts:

- Exchange 2010 was at SP2 R8, I upgraded to SP3, Rollup 19.
- Ran BPA, no issues.
- unplugged ethernet to server, no effect.  Que continued to grow.
- dismounted mail stores, no no effect.  Que continued to grow.
- Scanned with Malwarebytes and Microsoft Safety Scanner, clean.
- reviewed smtp send and receive logs, look to be expected traffic.  Not very large.
- loaded Exchange User Monitor, I have one mailbox with consistent large bytes out.  I disabled the user in AD and made sure the PC is turned   off.  iPhone email profile deleted off the phone.  data for this user/mailbox is still showing up on refresh in EUM.

Not sure where to go next, any recommendations would be appreciated.
0
David PymanAuthor Commented:
Here are results from the Exchange Troubleshooting Assistant.  Looks like the one mailbox/user is causing the issue, I have not found a method to resolve.

The user 'Tim' is using 46.48% of the CPU in MAPI operation "ReadStream". The user issued 55698 requests of this type.
The user 'Tim' is using 53.52% of the CPU in MAPI operation "End". The user issued 0 requests of this type.
0
Michael B. SmithExchange & Active Directory ExpertCommented:
I'd like to see your Application event log and the send/receive protocol logs. I realize that is not practical here.

I don't have any more good ideas. :-(
0
David PymanAuthor Commented:
Issue is solved,  BPA confirmed the results of the ETA.  I backed up and deleted the offending mailbox and re-created a new mailbox for the user.  Transport operation returned to normal.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.