high cpu on hub mailbox 2010 sp3 ru 10
I have one our of 8 hub/mailbox servers having high CPU split between the Store.exe and msexchangerepl.exe processes.
My topoplogy:
2 AD sites 2 CAS servers in each site 4 hub/mailbox servers in each site the DAG spans the site with each mailbox database having 2 copies in each site. (FSW is in a third site)
All servers are 2008r2 with the latest patches.
These are all virtual machines on vmware.
The server having the high CPU has no active databases on it just copies (4 of them).
Not showing any consistent errors or warnings. it has been doing this for over a week now. It will spike to 100% for a while then operate between 80 and 50% for a while then in the evenings drop to almost normal (5%- 15%) like the other hub/mailbox servers.
Restarting services or rebooting has little to no effect.
This server also houses the public folders for this site. I have dismounted them and it makes no difference. I also dismounted the non DAG replicated database (which has no mailboxes on it) - no change.
and I'm sure I'm forgetting something else I tried....
Thank you in advance for your help!
My topoplogy:
2 AD sites 2 CAS servers in each site 4 hub/mailbox servers in each site the DAG spans the site with each mailbox database having 2 copies in each site. (FSW is in a third site)
All servers are 2008r2 with the latest patches.
These are all virtual machines on vmware.
The server having the high CPU has no active databases on it just copies (4 of them).
Not showing any consistent errors or warnings. it has been doing this for over a week now. It will spike to 100% for a while then operate between 80 and 50% for a while then in the evenings drop to almost normal (5%- 15%) like the other hub/mailbox servers.
Restarting services or rebooting has little to no effect.
This server also houses the public folders for this site. I have dismounted them and it makes no difference. I also dismounted the non DAG replicated database (which has no mailboxes on it) - no change.
and I'm sure I'm forgetting something else I tried....
Thank you in advance for your help!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Amit
VMware tools are up to date on that one server, where you are facing issue. Also, how is load on ESX server itself, do you have enough resource on that server. Finally, I assume CPU and RAM is not allocated dynamically.
Here is the VMWare best practices guide for Exchange 2010
http://www.vmware.com/files/pdf/exchange-2010-on-vmware-best-practices-guide.pdf
Hyper threading needs to be turned off and don't oversubscribe resources.
http://www.vmware.com/files/pdf/exchange-2010-on-vmware-best-practices-guide.pdf
Hyper threading needs to be turned off and don't oversubscribe resources.
I would make sure VMware tools are 100% up to date, although I doubt this is the issue. Furthermore you want to turn off HT and not over subscribe. Hyperthreading is not a big issue for the store processes (e.g: database, which is consuming your CPU cycles) but will cause .NET garbage collection, thus issues for the transport processes eventually.
What you are describing sounds like a sizing issue.. either with the amount of available cores or disk IO. If you dont have enough IOPS for the database operations, you will see a spike in CPU operations (from what I have seen). I would recommend running some sizing tools, and then placing those numbers into the Exchange Calculator for Exchange 2010. Confirm the IOPS and CPU core calculation is correct for what you have deployed.
What you are describing sounds like a sizing issue.. either with the amount of available cores or disk IO. If you dont have enough IOPS for the database operations, you will see a spike in CPU operations (from what I have seen). I would recommend running some sizing tools, and then placing those numbers into the Exchange Calculator for Exchange 2010. Confirm the IOPS and CPU core calculation is correct for what you have deployed.
ASKER
Finaly got to run this on my mailbox server. Running through the PAL right now.
VMware tools are up to date and the other 7 mailbox servers are running fine in the environment.
figuring out how to read LDAP read and search times and RPC average latency.
VMware tools are up to date and the other 7 mailbox servers are running fine in the environment.
figuring out how to read LDAP read and search times and RPC average latency.
ASKER
After looking through the data it was another virtual machine on the same SAN running an engineering calculation (Very long runs) taking up all of the disk I/O.!! Moved those off of that SAN and like magic it returned to normal. Thank you all for your patience and advice!