After starting Microsoft exchange search services, CPU utilization is going full high

I am using Microsoft exchange 2010. We have 3000 Mailboxes. Have assigned 8 cores to both of 2 mailboxes server. Recently I was facing issue that CPU utilization was going full high but we stopped Microsoft exchange search service, CPU utilization was  normal. Today again we started Microsoft exchange search service, CPU is utilizing full. How to solve it?
LVL 1
Arif KhanSystem AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

8046586Commented:
0
David Johnson, CD, MVPOwnerCommented:
Always having low cpu usage means that you've overbought your compute power. The search service runs at a low priority and until things are indexed it will want any available cpu/disk  The service again is designed to release cpu cycles to processes that have a higher priority
0
Arif KhanSystem AdministratorAuthor Commented:
Can you please elaborate the reason? This is serious problem.
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

David Johnson, CD, MVPOwnerCommented:
How is it a serious problem? High Disk Queue's? Please elaborate
0
Arif KhanSystem AdministratorAuthor Commented:
When CPU utilization goes high i.e. 100%. You know back pressure. And mails are in queue. I do not want to disable back pressure. Maximum CPU utilization is causing mail flow problem delay delivery, high mail queue and exchange DB crash.
0
8046586Commented:
Did you try re-creating the index folder?
0
Arif KhanSystem AdministratorAuthor Commented:
Not yet, I am just waiting a reply from management. Will do it in evening cause during rebuilding time, it will again utilize high CPU so night is better for this. Well, any other step can you suggest a apart from rebuilding index?
0
Arif KhanSystem AdministratorAuthor Commented:
I have both mailboxes in DAG. How to rebuild the Index in DAG environment? Should I remove Copy of databse then stop Exchange search service and rename index folder then start it and add DB copy in DAG.? Or without remving from DAG. Need you step by step suggest.
0
8046586Commented:
1. Stop the service
2. Move the folder to another location or rename it if you have enough free space
3. Start the service

When service will start should create a new folder. f you have issues you can always rename back the original folder.
0
Arif KhanSystem AdministratorAuthor Commented:
But it is in DAG in environment. Should I follow this step on both servers OR sync only catalog from reubuilt server??
0
8046586Commented:
Follow the manual steps of this document https://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx on both servers.

Stop the service on both, rename the Catalog data folder on both. run the commands on the first server, start the service on the first server.
If you get positive response start the service on the second server to sync the data.
0
Adam BrownSr Solutions ArchitectCommented:
You have too many mailboxes on a single server. 3000 is quite a lot. If you can, you should be able to distribute the mailboxes across DAG members by splitting the mailboxes into multiple databases, then having some active on one dag member and others active on a different dag member.

Search will use a lot of CPU while it crawls through mail. Once the initial crawl is complete, though, the CPU utilization should decrease.
1
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
3000 is way to much for a single cas if that is what this is. can you be more descriptive of the exchange setup?
also is this physical or virtual? I disagree with some experts that it is ok for Exchange to operate at such high performance levels.

Have a look at this, a sizing chart for Exchange by MS https://technet.microsoft.com/en-us/library/dn879075(v=exchg.150).aspx
0
Arif KhanSystem AdministratorAuthor Commented:
Yes, now  I have moved Two DBs on other mailbox server. But still CPU utilization was 100%. But After stopping Microsoft search (exchange) service, CPU utilization became very normal. I am just getting confuse what is happening. I will recreate Index tomorrow cause the day after tomorrow is Sunday.
0
Arif KhanSystem AdministratorAuthor Commented:
Just to inform you that we have  two more servers on which HUB/CAS is installed so we have two hub and two CAS attached in CAS array. Mailbox servers are separate.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Recreating the index is what I would do here too, reevaluate until then. Assuming there is still indexing going on which is why such high CPU.

Have a look at your AV too, ensure you have proper exclusions in place, on access scanning can make anything slow. I would even consider removing your AV for troubleshooting. http://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx shell steps to reindex same applies to EX 2016.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Arif KhanSystem AdministratorAuthor Commented:
I am recreating index. I started it  on Saturday night but still it is running. I see status as crawling. No percentage is available. How can I estimate the approx time that how much will this take to recreate index. I started it on 16th DEC 2017 around 7 PM  and still  nothing is indexed and all are just showing crawl. I am just shocked, why is this taking too long time. Can someone answer to following quesstion please.

1. How can see progress of indexing? I following but it did not help me to watch the status. I dont see anything under Resource Monitor in exchnge server. https://practical365.com/exchange-server/monitor-rebuilding-content-indexes-exchange-databases/

2. How can I calculate the approx time of indexing?


Database size  is as below..........
295 GB

341 GB

412 GB

345 GB

370 GB

15.6 GB

506 GB

92.6 GB

145 GB
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
In the link you posted Paul Cunningham is saying to use performance monitor, not resource monitor to check this.
start -> run -> perfmon

The comments at the bottom of the page also give some indication.

"
Once I had to run this rebuilding process for some bigger databases in a customers Exchange environment. The rebuilding took about 1-1.5 days per database.
If you don’t need to fully rebuild the index, you can update it from a non-corrupted database copy by:
Update-MailboxDatabaseCopy -Identity DATABASE-NAMEDATABASE-NAME -CatalogOnly
That’s way faster. Sadly I could use that command only for a few databases, because in most of the cases all database copies had damaged search indexes.

I also wonder why Microsoft removed the “ResetSearchIndex.ps1” which still existed in Exchange 2010.
"
0
Arif KhanSystem AdministratorAuthor Commented:
I just checked there, but nothing found as shown in practical365. I have uploaded the screenshot. No indication about process. yes, I know we can reseed catalog only. But in my case, Since CPU was going high on both exchange servers so I planned to rebuild CATALOG folder indexing again from scratch. I did following.

1. Stopped Exchange search server on Active and Passive MBX server
2. Renamed the catalog
3. Start exchange search service
4. Rebuild of index is started and status is crawling.

Still, I am having old catalog folder. According to your expertise experience can you suggest me how much time it might take to complete it as I have given my DB size. So that i can inform same to management team.
exchange-error.jpg
exchange-error.jpg
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
I actually dont know but it is unlikely to take more than 1.5 days as per the comment I posted, so I would wait for the process to complete or error out.

Paul Cunningham wrote that article, so I would retrace your steps, his articles have got me through some of the biggest jobs I have done. He is really really good.
0
Arif KhanSystem AdministratorAuthor Commented:
So, i have total 9 DB , you mean i should wait 9×1.5=14 says?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
If thats what it wants yes. Take it slowly. Is RAM usage high? if so add more RAM will speed it up.
Do not add more cores unless planning on keeping them. Hot adding and removing CPU is not good.

Let the application do its work, wait for 1 DB to complete and then time it from there.

One time I had to wait 17 hours for a 1.3MB windows update to install. LOL. still fricking hilarious. I am a wizard for waiting that one out, so close to calling it quits.
It was a bug in windows, one of the windows winsxs folders was doing a massive update in the background, it was a bug. Just saying patience....
0
Arif KhanSystem AdministratorAuthor Commented:
Thanks, RAM utilization is usual so dar now. Only CPU is sucking my time. I will wait some more says, will share update tomorrow.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Cheers, the perfmon monitor will be available, check over your steps from that link.
That will give you the speed indication/timing.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Pause your nightly backup if you have one, that will hardly help.
Yes please let us know of update.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.