High CPU usage on Exchange 2013 server

Can I see the VM performance graph please?

What storage is the server based on in VMware? San or Local Storage?

How many exchange servers?

What is the hardware resource allocation for this server?

How many mailbox databases?

What size are your mailbox databases?

The vm is using local storage with a single database that is about 60 Gb.  I noticed that there are over 10,000 text files in the same folder as the database that are all 1 MB as well from the last month.  I'm guessing they are some type of log file?

Hi,

Have a look at this link regarding the logs. https://social.technet.microsoft.com/Forums/exchange/en-US/abfaaeba-cadc-4850-93c1-1c86d22500fd/e000000000xtxt-files-in-mailbox-directory

What version of windows is on this box?

How much ram and cpu are allocated to the server?

Have a look at this in regards to turning off noderunner as a test http://www.techieshelp.com/exchange-2013-noderunner-exe-using-lots-of-memory/

if you right click the vm , edit settings, how many sockets and cores has it got assigned to it.

what is the cpu, ram and disk configuration of the host server?

The host server is a Dell R520
8 CPU X 1.8 GHZ
64 GB RAM
2 local data stores, both RAID 1 - 900 GB SAS 10K drives
One data store is used for OS and Exchange app, other is used for Exchange db

noderunner is the process used for indexing so having high cpu for it is normal
i would, though, add some more memory; 8gb is the bare minimum so it would be a benefit.  i would go with 16
keep in mind the information store (store.exe) memory usage will still be high which is normal as it takes as much memory as it can though it will release some if other processes request memory

weve got 2 sockets assigned to this box this is not really good, your stuck with it now though, reason for this is if you ever migrated to a one processor machine or had to remove a core it can cause serious operating system issues.

as the link i posted says noderunner is an indexing service, its especially ok to see it going nuts if this is a new exchange 2013 server.

id add 8GB more memory, are there any backups running during the slowdown? do you have any snapshots taken in vmware? what is happening in the windows system application logs when this slowdown occurs? is the slowdown constant or comes and goes?

can you also post the vmware disk performance and the exchange server disk performance through resource monitor and process manager(with the io read write columns added and sorted from high to low)

memory usage will always be high on this server as mentioned above.

The server has been active for a bout a year.  There is a single snapshot in VMware that I will delete tonight as well as add the additional memory.  Why is a second socket a problem? The poor performance is consistently bad at all times.  Good thing is the Outlook clients do not notice anything.  It is only when you are on the server.

Log Name:      Application
Source:        MSExchangeIS
Date:          6/25/2015 1:19:54 PM
Event ID:      1012
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      C4HMAIL2.context.local
Description:
Exchange Server Information Store has encountered an error while executing a full-text index query ("and(subject:string("SearchQueryStxProbe*", mode="and"), folderid:string("AD1B6B41848B05448FCE67F7AFA8BD0400000000010D0000"))"). Error information: System.TimeoutException: Failed to open a channel.
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel& serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteSearchFlow(String flowName, Dictionary`2 inputData)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.<ExecuteInternal>d__1f.MoveNext()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.<ExecuteSimple>d__a.MoveNext()
   at Microsoft.Exchange.Server.Storage.FullTextIndex.FullTextIndexQuery.ExecutePagedFullTextIndexQuery(Guid databaseGuid, Guid mailboxGuid, Int32 mailboxNumber, String query, CultureInfo culture, Guid correlationId, Boolean needConversationId, QueryLoggingContext loggingContext, PagedQueryResults pagedQueryResults)
   at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreFullTextIndexQuery.ExecuteOnePage(Context context, MailboxState mailboxState, Boolean needConversationDocumentId, SearchExecutionDiagnostics diagnostics)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeIS" />
    <EventID Qualifiers="49156">1012</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-06-25T18:19:54.000000000Z" />
    <EventRecordID>6055740</EventRecordID>
    <Channel>Application</Channel>
    <Computer>C4HMAIL2.context.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>and(subject:string("SearchQueryStxProbe*", mode="and"), folderid:string("AD1B6B41848B05448FCE67F7AFA8BD0400000000010D0000"))</Data>
    <Data>System.TimeoutException: Failed to open a channel.
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel&amp; serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteSearchFlow(String flowName, Dictionary`2 inputData)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.&lt;ExecuteInternal&gt;d__1f.MoveNext()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.&lt;ExecuteSimple&gt;d__a.MoveNext()
   at Microsoft.Exchange.Server.Storage.FullTextIndex.FullTextIndexQuery.ExecutePagedFullTextIndexQuery(Guid databaseGuid, Guid mailboxGuid, Int32 mailboxNumber, String query, CultureInfo culture, Guid correlationId, Boolean needConversationId, QueryLoggingContext loggingContext, PagedQueryResults pagedQueryResults)
   at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreFullTextIndexQuery.ExecuteOnePage(Context context, MailboxState mailboxState, Boolean needConversationDocumentId, SearchExecutionDiagnostics diagnostics)</Data>
    <Binary>5B444941475F4354585D00006C000000FF0D00000000000000025000000080CA00100000000030FF001000000000309F00100000000030DF001000000000B09D001000000000B0DD001000000000B0ED001000000000B08D001000000000B095001000000000B0A5001000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        Service Control Manager
Date:          6/25/2015 12:44:12 PM
Event ID:      7011
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      C4HMAIL2.context.local
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VMTools service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7011</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-06-25T17:44:12.026522100Z" />
    <EventRecordID>169615</EventRecordID>
    <Correlation />
    <Execution ProcessID="240" ThreadID="18648" />
    <Channel>System</Channel>
    <Computer>C4HMAIL2.context.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">30000</Data>
    <Data Name="param2">VMTools</Data>
  </EventData>
</Event>

Ok, can we verify we have exchange 2013 SP1 installed and all updates after that.

can we also run best practices analyzer and post output.

can we ensure were using the latest version of vmware tools as well pls.

apart from VM performance I would like to point out at

I noticed that there are over 10,000 text files in the same folder as the database that are all 1 MB as well from the last month

Please start backup of your exchange server.

what about backups are you using a backup?

need to know the answers to the last two Q's mate when you get a chance, last 2 posts i made sorry. M

I got a successful backup and the logs are gone!  Thank you for pointing that out.  Apparently the vmProtect is not so compatible with Exchange 2013 so I had to use Veeam(which we recently purchased to replace Acronis).  I am running SP1 but don't think any of the rollups are installed.  VMware tools version 9 build 782409 is installed.

[PS] C:\Windows\system32>get-exchangeserver | fl name, edition, admindisplayversion


Name                : C4HMAIL1
Edition             : Enterprise
AdminDisplayVersion : Version 14.3 (Build 123.4)

Name                : C4HMAIL2
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 847.32)

A recent full backup and upgrade seemed to have helped quite a bit