Link to home
Start Free TrialLog in
Avatar of First Last
First LastFlag for United States of America

asked on

Keep getting search related index errors on Exchange 2010 server

I've been having problems with Outlook searching over the last few weeks.  With help from some experts here and on technet I was able to repair the problem by rebuilding the index and re-applying the filter packs for Exchange 2010.  This lasted for about a week before the same errors started appearing in the event log and searching for email in outlook started failing to run.  I've been looking at this closely and it looks to me like the problem starts when the databases at my primary site attempt to fail over to the other DAG member.  The DAG contains just two Exchange servers with the primary hosting mailboxes, hub transport, and client access and the secondary (offsite) just hosting a passive copy of the databases.  I have the failover blocked because of high latency and just use the DAG to keep current database copies offsite.  

Here is a sequence of errors I saw on the exchange server beginning at 11pm and through the next day on 7/19 just prior to the index problem starting again:

Event ID 14003
Unable to create Group Metrics distribution share.
Share: GroupMetrics
Directory: C:\Program Files\Microsoft\Exchange Server\V14\GroupMetrics
Message: 00000842

(THERE IS ONE OF THESE ERRORS FOR EACH OF MY 4 DATABASES)
Event ID 2060
The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for 2010_DB1\EXCHANGE. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.

Event ID 7004
The activation of all modules took longer than expected to complete. Total Load Time: 00:00:23.1855780 Total Start Time: 00:00:00.3990425 Load Time Breakdown: (goes on to list verbose info)

Event ID 1033
The setting MaxNoOfItemsMove in the Web.Config file was not valid. The previous value was null and has been changed to 1000.

Event ID 5502
MSExchangeTransport Component Latency for external servers has detected negative latency exceeding 2 seconds.  All servers specified in the InternalSmtpServers parameter should use Network Time Protocol to ensure clock synchronization.

Event ID 107
Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database 2010_DB3 (GUID = 9c6e1d04-78c4-43bf-8065-d018c6b6427e) due to an error (Microsoft.Exchange.Msfte.BatchTimeoutException: Timeout detected for batch Database Guid:9c6e1d04-78c4-43bf-8065-d018c6b6427e; NumberOfFailedDocuments:0; ShouldIsolateError: False; SubmitTickCount:17831928; BatchKind:RetryMailbox; Number of entries:1).

Event ID 1022
The connection between the Client Access server and Mailbox server "EXCHANGE2.domain.local" failed.

Event ID 1016
Exchange ActiveSync has encountered repeated failures when it tries to access data on Mailbox server [EXCHANGE2.domain.local].

(THERE IS ONE EVENT FOR EACH OF MY 4 DATABASES)
Event ID 2091
Database: 2010_DB3\EXCHANGE
Mailbox Server: exchange.domain.local

Database 2010_DB3\CUNJ-EXCHANGE will be mounted with the following loss information:
* The last log generated (known to the server) before the switchover or failover was: 773134
* The last log successfully replicated to the passive copy was: 773134
* AutoDatabaseMountDial is set to: GoodAvailability

Attempts to copy the last logs from the active database copy weren't successful. Error: The log copier was unable to continue processing for database '2010_DB3\CUNJ-EXCHANGE' because the source server 'CUNJ-EXCHANGE2.cunj.local' returned an error: Cannot access file, the file is locked or in use (-1032) [HResult: 0x80131501]. The copier will automatically retry after a short delay.

(THERE IS ONE OF THESE EVENTS FOR EACH OF MY 4 DATABASES)
Event ID 9796
Database "2010_DB2" has been subject to a lossy failover. The database may be patched if the Information Store detects it is necessary.

Once the index repair is run on both servers (takes about 3-4 hours) I immediately see approximately 50 of these errors:

Event ID 9875
Unexpected error "DOC_TOO_HUGE: There are not enough resources to process the document or row" occurred while indexing document.
Mailbox Database: 2010_DB3
Folder ID: 1-83B6
Message ID: 1-1A0F3A34
Document ID: 2879205
Error Code: 0x8004364a

and about half as many of this error:

Event ID 9646
Mapi session "f3541631-a414-4a86-878b-88269ef3d55c: /O=PATH/OU=PATH/cn=Recipients/cn=userid" exceeded the maximum of 250 objects of type "objtMessage".


Events 9875 and 9846 have been going on since day one and don't seem to cause problems in and of themselves but show up en mass when the index is rebuilt.  This seems to occur whenever the databases try to fail over so I'm assuming I have something setup incorrectly there.  I'd very much appreciate any help with this, its been nagging me for a while now.
Avatar of Vijaya Babu Sekar
Vijaya Babu Sekar
Flag of India image

could you please run the below cmdlets in powershell

Get-MailboxDatabaseCopyStatus –Server <server name> | fl Name,*Index*

If you getting result failed.

You can run below cmdlets

Update-MailboxDatabaseCopy -Identity <mailbox database name>\<Server name> -SourceServer <Source server name> -CatalogOnly
Avatar of First Last

ASKER

Hi vijayhakcers!

I ran the first command and got a successful result:
Name                     : 2010_DB1\EXCHANGE
ContentIndexState        : Healthy
ContentIndexErrorMessage :

Name                     : 2010_DB2\EXCHANGE
ContentIndexState        : Healthy
ContentIndexErrorMessage :

Name                     : 2010_DB3\EXCHANGE
ContentIndexState        : Healthy
ContentIndexErrorMessage :

Name                     : 2010_DB4\EXCHANGE
ContentIndexState        : Healthy
ContentIndexErrorMessage :

Search is working at the moment, I'm very interested in how I can prevent the situation from occuring again when the DAG attempts to fail over.
ASKER CERTIFIED SOLUTION
Avatar of Vijaya Babu Sekar
Vijaya Babu Sekar
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial