What is the root cause of corrupt exchange 2010 database search

Do you know what would cause the searches to fail on Exchange 2010 databases ?
Users are reporting that they are unable to find all items during a search.  Helpdesk knows how to fix the problem by recreating the search files or to go around it by building new a database and copying the mailboxes. However we are trying to find out what is causing the frequent corruption.  The Exchange server is Exchange  2010, Trend Micro Scan Mail for exchange mail filter,  code two pro signature and Symantec Backup Exec 2015 remote agent also run on the server.

This link is a discussion of similar problems with DAGs and one user traced the problem to an anti virus product.

https://social.technet.microsoft.com/Forums/exchange/en-US/d0d84126-2b83-48bf-8eed-240465900430/repeatedly-corrupted-search-catalogs-on-multiple-dag-members?forum=exchange2010
LVL 1
masterofallAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Guy LidbetterCommented:
Hi masterofnone ;-p

I previously had a very similar issue a while back and stumbled across a post which resolved the issue for us. It came down to the Index DB being corrupt so a CatalogOnly reseed just rebuilt a corrupt DB that failed at a later stage once again on a DB move.

I don't recall where I got it from but I've had this as a sticky in my one note:

1. Logon to the Exchange 2010 server that has the problem database and index.

2. Open the Exchange Management Shell.

3. Change to the following directory where the ResetSearchIndex.ps1 script is located: <drive> :\Program Files\Microsoft\Exchange Server\Scripts

4. Suspend the MailboxDatabaseCopy process from the active node MBX2 to the target passive node MBX1:

Suspend-MailboxDatabaseCopy -Identity DB1\MBX1

5. Run the following script to remove the index directories that are associated with the specified mailbox database with the source MBX2.

ResetSearchIndex.ps1 -force MBX2

How to Rebuild the Full-Text Index Catalog

http://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx 

Note: it also applies to Exchange 2010.

6. Verify that the rebuilding of the index has completed:

a. Before doing the ResetSearchIndex - add this counter to a live Perfmon: MSExchange Search Indices\Full Crawl Mode Status

b. This counter will be 0 before running ResetSearchIndex and go to 1 during the full crawl and then back to 0 after ResetSearchIndex finishes.

c. You will receive MSExchange Search Indexer Event ID 109 when the full crawl begins.

d. You will receive MSExchange Search Indexer Event ID 110 when the full crawl ends.

7. Reseed the content index catalog from a specific source (source node is the active node MBX2 and the passive node is MBX1):

Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2 -CatalogOnly

Reseed the Search Catalog: Exchange 2010 Help

http://technet.microsoft.com/en-us/library/ee633475.aspx 

8. Resume the MailboxDatabaseCopy from the active node MBX2 to the passive node MBX1:

Resume-MailboxDatabaseCopy -Identity DB1\MBX1


Give it a go and let me know how you get on....
Guy LidbetterCommented:
In saying that, it probably doesn't help with the initial cause... a few questions:

Are the Exchange Servers Virtual?
Have you checked the AV exclusion lists?
Are you running a file level backup of the index DB directory?
Have you run any disk IO (read\write Delays, queues etc) monitors?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Will SzymkowskiSenior Solution ArchitectCommented:
If you are getting consistent indexing corruption i would look in the logs to check and see if anything in there is standing out in regards to your database. Also what SP version and RU do you have installed for your Exchange 2010 environment? If you are way out I would consider upgrading to the latest version.

Will.
masterofallAuthor Commented:
Guy Questions

Are the Exchange Servers Virtual? ==> No

Have you checked the AV exclusion lists?  ==> Yes, we did not find any scans.

Are you running a file level backup of the index DB directory?  ==> Yes and my backup admin suspect that it is the problem.

 Have you run any disk IO (read\write Delays, queues etc) monitors ==> Just ran Dell DPACK this weekend.
masterofallAuthor Commented:
The problem was caused by backup exec's SDR backup.  It was backing up the directory with the search file.  The SDR backup does not nee that drive so we unchecked it.
Guy LidbetterCommented:
Glad you got that sorted!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.