David Sowerby
asked on
Exchange 2016 Server Crashing with 100% CPU
Exchange Server 2016, fully patched
Is a virtual server 2012R2 server with 8 processors and 32gb of RAM (totally over-spec'd, I know)
We have 318 mailboxes on the server
Email scenario/issues are as follows:
Since completing the move of all mailboxes in their entirety to Exchange 2016 in June, we have noticed a drastic drop in reliability and performance from the server. Some of our Departmental mailboxes are 5gb+ (the largest being over 8gb). These Departmental mailboxes can have entire departments connecting to them every day. The worst one probably has around 15-20 people connecting to it.
The problem we have is that round about 9am every morning, the Exchange server grinds to a halt, 100% processor usage and will stay like that for up to an hour, or more. We used to have a lot of our Users connect to the Exchange server in an “Online” state and found that they would be disconnected from the Exchange server while the ongoing processor usage was 100% and would not reconnect until the server had sorted itself out again. This caused problems as the Users’ would receive no email for up to and in some instances, more than an hour.
It mainly happens to Users that have multiple mailboxes in Outlook.
We have since moved everyone onto Cached Exchange Mode to try and take some of the stress off the server, it hasn’t. Server CPU still sits at 100% between 9am-10am, not as much user affecting currently as they are now not being disconnected.
Problem we now also have is that with everyone being in cached exchange mode, emails are not automatically being downloaded to the cache to update Inbox and manual folder updates need to be done. This does not happen to everyone, but we have at least a dozen that it affects.
One or two people in particular have issues throughout the day with Outlook freezing when email/diary invitations come through.
Usually, while CPU is being thrashed, it is the Microsoft.Exchange.Store.W
Any help would be much appreciated.
Is a virtual server 2012R2 server with 8 processors and 32gb of RAM (totally over-spec'd, I know)
We have 318 mailboxes on the server
Email scenario/issues are as follows:
Since completing the move of all mailboxes in their entirety to Exchange 2016 in June, we have noticed a drastic drop in reliability and performance from the server. Some of our Departmental mailboxes are 5gb+ (the largest being over 8gb). These Departmental mailboxes can have entire departments connecting to them every day. The worst one probably has around 15-20 people connecting to it.
The problem we have is that round about 9am every morning, the Exchange server grinds to a halt, 100% processor usage and will stay like that for up to an hour, or more. We used to have a lot of our Users connect to the Exchange server in an “Online” state and found that they would be disconnected from the Exchange server while the ongoing processor usage was 100% and would not reconnect until the server had sorted itself out again. This caused problems as the Users’ would receive no email for up to and in some instances, more than an hour.
It mainly happens to Users that have multiple mailboxes in Outlook.
We have since moved everyone onto Cached Exchange Mode to try and take some of the stress off the server, it hasn’t. Server CPU still sits at 100% between 9am-10am, not as much user affecting currently as they are now not being disconnected.
Problem we now also have is that with everyone being in cached exchange mode, emails are not automatically being downloaded to the cache to update Inbox and manual folder updates need to be done. This does not happen to everyone, but we have at least a dozen that it affects.
One or two people in particular have issues throughout the day with Outlook freezing when email/diary invitations come through.
Usually, while CPU is being thrashed, it is the Microsoft.Exchange.Store.W
Any help would be much appreciated.
Alex
Sounds like it's indexing, also given the fact that you've overspec'ed it so much, that could impact the hypervisor level. Run ESXTOP on SSH on the host and then check your CSTP time, likelihood is, it'll be well above the recommended level of 3 and hitting over 20. That'll cause the server to freeze and your symptoms that you've described.
Try disabling hyperthreading which is not recommended on Exchange.
Reference Article: https://blogs.technet.microsoft.com/jcoiffin/2016/11/30/100-cpu-on-exchange-20132016-check-the-number-of-cpus-is-not-too-high/
Reference Article: https://blogs.technet.microsoft.com/jcoiffin/2016/11/30/100-cpu-on-exchange-20132016-check-the-number-of-cpus-is-not-too-high/
It's a virtual machine..... If you disable HT you'll affect all other VM's on that host. I'd NOT recommend doing this. Essentially having exchange on there is fine, the number of cores and amount of memory I think could be the issue.
Actually here
"When it is physical CPU without hyperthreading , there are sometimes hardware options to disable part of the CPUs in order to avoid exceeding max number of CPU recommendation."
This isn't a physical CPU, it's virtual, so don't disable HT.
Actually here
"When it is physical CPU without hyperthreading , there are sometimes hardware options to disable part of the CPUs in order to avoid exceeding max number of CPU recommendation."
This isn't a physical CPU, it's virtual, so don't disable HT.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIALMembers can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.