High CPU usage on Exchange 2013 server

Our Exchange 2013 server is very slow.  The cpu is running at 80-90% with 90% memory utilization all day .  The server is running as a virtual machine in on an ESXi host server with 8 cpu's and 8GB of memory assigned.  There are only about 50 users with a db less than 100 gb.  Looking at the running processes I see several instances of noderunner running in the background and consuming over 50%. any idea what the noderunner process is doing and why there is multiple instances?
NytroZAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Can I see the VM performance graph please?

What storage is the server based on in VMware? San or Local Storage?

How many exchange servers?

What is the hardware resource allocation for this server?

How many mailbox databases?

What size are your mailbox databases?
0
NytroZAuthor Commented:
exchange-perf.jpg
The vm is using local storage with a single database that is about 60 Gb.  I noticed that there are over 10,000 text files in the same folder as the database that are all 1 MB as well from the last month.  I'm guessing they are some type of log file?
1
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Hi,

Have a look at this link regarding the logs. https://social.technet.microsoft.com/Forums/exchange/en-US/abfaaeba-cadc-4850-93c1-1c86d22500fd/e000000000xtxt-files-in-mailbox-directory

What version of windows is on this box?

How much ram and cpu are allocated to the server?
0
NytroZAuthor Commented:
esx-resources.png
1
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Have a look at this in regards to turning off noderunner as a test http://www.techieshelp.com/exchange-2013-noderunner-exe-using-lots-of-memory/

if you right click the vm , edit settings, how many sockets and cores has it got assigned to it.

what is the cpu, ram and disk configuration of the host server?
1
NytroZAuthor Commented:
esx-resources2.png
The host server is a Dell R520
8 CPU X 1.8 GHZ
64 GB RAM
2 local data stores, both RAID 1 - 900 GB SAS 10K drives
One data store is used for OS and Exchange app, other is used for Exchange db
1
Seth SimmonsSr. Systems AdministratorCommented:
noderunner is the process used for indexing so having high cpu for it is normal
i would, though, add some more memory; 8gb is the bare minimum so it would be a benefit.  i would go with 16
keep in mind the information store (store.exe) memory usage will still be high which is normal as it takes as much memory as it can though it will release some if other processes request memory
1
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
weve got 2 sockets assigned to this box this is not really good, your stuck with it now though, reason for this is if you ever migrated to a one processor machine or had to remove a core it can cause serious operating system issues.

as the link i posted says noderunner is an indexing service, its especially ok to see it going nuts if this is a new exchange 2013 server.

id add 8GB more memory, are there any backups running during the slowdown? do you have any snapshots taken in vmware? what is happening in the windows system application logs when this slowdown occurs? is the slowdown constant or comes and goes?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
can you also post the vmware disk performance and the exchange server disk performance through resource monitor and process manager(with the io read write columns added and sorted from high to low)

memory usage will always be high on this server as mentioned above.
0
NytroZAuthor Commented:
The server has been active for a bout a year.  There is a single snapshot in VMware that I will delete tonight as well as add the additional memory.  Why is a second socket a problem? The poor performance is consistently bad at all times.  Good thing is the Outlook clients do not notice anything.  It is only when you are on the server.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
can you look through the event logs system and application logs and filter for errors, criticals and warnings and post them up pls.

what about backups are you using a backup?

You should only have one socket and not two because if you were to migrate to a new server that had one processor it may cause operating system instability and if you were to remove one of the two it could cause the same, not the end of the world just keep it in mind in future. 1 socket 4 cores is more than enough for this server imo.

can you also run exchange best practices analyzer its within the EMC.
1
NytroZAuthor Commented:
Log Name:      Application
Source:        MSExchangeIS
Date:          6/25/2015 1:19:54 PM
Event ID:      1012
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      C4HMAIL2.context.local
Description:
Exchange Server Information Store has encountered an error while executing a full-text index query ("and(subject:string("SearchQueryStxProbe*", mode="and"), folderid:string("AD1B6B41848B05448FCE67F7AFA8BD0400000000010D0000"))"). Error information: System.TimeoutException: Failed to open a channel.
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel& serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteSearchFlow(String flowName, Dictionary`2 inputData)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.<ExecuteInternal>d__1f.MoveNext()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.<ExecuteSimple>d__a.MoveNext()
   at Microsoft.Exchange.Server.Storage.FullTextIndex.FullTextIndexQuery.ExecutePagedFullTextIndexQuery(Guid databaseGuid, Guid mailboxGuid, Int32 mailboxNumber, String query, CultureInfo culture, Guid correlationId, Boolean needConversationId, QueryLoggingContext loggingContext, PagedQueryResults pagedQueryResults)
   at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreFullTextIndexQuery.ExecuteOnePage(Context context, MailboxState mailboxState, Boolean needConversationDocumentId, SearchExecutionDiagnostics diagnostics)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeIS" />
    <EventID Qualifiers="49156">1012</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-06-25T18:19:54.000000000Z" />
    <EventRecordID>6055740</EventRecordID>
    <Channel>Application</Channel>
    <Computer>C4HMAIL2.context.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>and(subject:string("SearchQueryStxProbe*", mode="and"), folderid:string("AD1B6B41848B05448FCE67F7AFA8BD0400000000010D0000"))</Data>
    <Data>System.TimeoutException: Failed to open a channel.
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel&amp; serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteSearchFlow(String flowName, Dictionary`2 inputData)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.&lt;ExecuteInternal&gt;d__1f.MoveNext()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.&lt;ExecuteSimple&gt;d__a.MoveNext()
   at Microsoft.Exchange.Server.Storage.FullTextIndex.FullTextIndexQuery.ExecutePagedFullTextIndexQuery(Guid databaseGuid, Guid mailboxGuid, Int32 mailboxNumber, String query, CultureInfo culture, Guid correlationId, Boolean needConversationId, QueryLoggingContext loggingContext, PagedQueryResults pagedQueryResults)
   at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreFullTextIndexQuery.ExecuteOnePage(Context context, MailboxState mailboxState, Boolean needConversationDocumentId, SearchExecutionDiagnostics diagnostics)</Data>
    <Binary>5B444941475F4354585D00006C000000FF0D00000000000000025000000080CA00100000000030FF001000000000309F00100000000030DF001000000000B09D001000000000B0DD001000000000B0ED001000000000B08D001000000000B095001000000000B0A5001000000000</Binary>
  </EventData>
</Event>
0
NytroZAuthor Commented:
Log Name:      System
Source:        Service Control Manager
Date:          6/25/2015 12:44:12 PM
Event ID:      7011
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      C4HMAIL2.context.local
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VMTools service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7011</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-06-25T17:44:12.026522100Z" />
    <EventRecordID>169615</EventRecordID>
    <Correlation />
    <Execution ProcessID="240" ThreadID="18648" />
    <Channel>System</Channel>
    <Computer>C4HMAIL2.context.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">30000</Data>
    <Data Name="param2">VMTools</Data>
  </EventData>
</Event>
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Ok, can we verify we have exchange 2013 SP1 installed and all updates after that.

can we also run best practices analyzer and post output.

can we ensure were using the latest version of vmware tools as well pls.
0
Vaseem MohammedCommented:
apart from VM performance I would like to point out at
I noticed that there are over 10,000 text files in the same folder as the database that are all 1 MB as well from the last month
Please start backup of your exchange server.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
what about backups are you using a backup?

need to know the answers to the last two Q's mate when you get a chance, last 2 posts i made sorry. M
0
NytroZAuthor Commented:
I got a successful backup and the logs are gone!  Thank you for pointing that out.  Apparently the vmProtect is not so compatible with Exchange 2013 so I had to use Veeam(which we recently purchased to replace Acronis).  I am running SP1 but don't think any of the rollups are installed.  VMware tools version 9 build 782409 is installed.

[PS] C:\Windows\system32>get-exchangeserver | fl name, edition, admindisplayversion


Name                : C4HMAIL1
Edition             : Enterprise
AdminDisplayVersion : Version 14.3 (Build 123.4)

Name                : C4HMAIL2
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 847.32)
0
Seth SimmonsSr. Systems AdministratorCommented:
the latest is CU9; SP1 is the same as CU4
since they are cumulative, you don't need to install any previous ones

Cumulative Update 9 for Exchange Server 2013 (KB3049849)
https://www.microsoft.com/en-us/download/details.aspx?id=47679
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
NytroZAuthor Commented:
A recent full backup and upgrade seemed to have helped quite a bit
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.