We help IT Professionals succeed at work.

Exchange DAG will all Content index state for DBs failed and suspended

Ehab Salem
Ehab Salem asked
on
I have an Exchange 2016 DAG setup with 2 servers having the "Content index state" failed and suspended.
Actually on server1 it is failed, and on server2 it is suspended.
Using Exchange powershell commands to catalog the DB did not work most likely because no valid index is available.

What should I do to fix this?
Comment
Watch Question

Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
Ensure your Active Directory is up and restart the Exchange servers one by one and check the status..
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
Active Directory is up for sure, and I have already tried that. didn't work.
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

what is the message when you failover from failed database server to other ?
Sajid Shaik MSystem Admin
CERTIFIED EXPERT

Commented:
context index failed please check the following link for the solution

https://practical365.com/exchange-server/exchange-2016-failed-content-index/

all the best
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
Error: An error occurred while trying to validate the specified database copy for possible activation. Error: EX-01: Database copy 'IT' on server 'EX-01' has content index catalog files in the following state: 'Suspended'. If you need to activate this database copy, you can use the Move-ActiveMailboxDatabase cmdlet with the -SkipClientExperienceChecks parameter to forcibly activate the database.
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
@Sajid: This is For an Exchange 2016 server that is not a member of a DAG.
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
You have 2 options now..

Activate the db on second server skipping the content index skip

Or fix the catalog index and then activate.. what is your opinion?
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
My main interest is to have content indexes on both servers ok.
Would that be second option?
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
Please follow the steps like below..

1. Update the Suspended Catalog on Server 2 with the below command

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly
Update-MailboxDatabaseCopy -Identity DB2\Server2 -CatalogOnly
etc

2. Wait for the Catalog Index to show as healthy

Get-MailboxDatabaseCopyStatus -Server Server2

3. Activate the Database on second server..

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

4. Wait for sometime and see whether the server 1 database goes to healthy status automatically

Get-MailboxDatabaseCopyStatus -Server Server1

If any issues.. Please revert with the error messages..

Note: Updating the catalog index will take some time to index based on the size of the database..

If you don't want to wait.. follow the below steps

1. Activate the Database in Second Server (ignoring the catalog validation in Second Server)

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2 -SkipClientExperienceChecks

2. Once the Database activated in second server.. rebuild the index for database mounted in Second Server

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly

3.wait for some time later  if the server 1 not automatically goes to healthy state.. run the below command

Update-MailboxDatabaseCopy -Identity DB1\Server1 -Databaseonly

Before executing.. if you have any queries.. please clarify and then follow the steps..
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Run get-mailboxdatabasecopystatus *

To check whether the CI is is healthy or failed state on the active node.

If it is in failed state on the active node then you will have to stop the search services and rename the catalog folder on the active node.

This will rebuild the index for the active node databases.

Once the CI is in healthy state on active node, you can then run command: Update-MailboxDatabaseCopy -Identity DB1\MBX1 -CatalogOnly

This will update the catalog folder only on the passive node. Then try to resume the copy and see of that helps.

If resuming does not help then you need to run the complete command to update  the database copy as well as catalag:

Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly

Doesn't this need a valid index to seed from?
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
what is the status of Index on the other server?
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Yes it needs valid index from the active database copy.

Run command: get-mailboxdatabasecopystatus *

to see whether the index is in healthy on the active database copy. Then only the catalog index reseed will work cause it reads from the active node index.
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
what is the status of Index on the other server?
Failed
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus it | sort name | Select name,status,contentindexstate

Name                                                                     Status                       ContentIndexState
IT\MAIL-01                                                             Mounted                  Suspended
IT\MAIL-02                                                             Healthy                     Failed
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Run command: Get-MailboxDatabaseCopyStatus | fl *err*

and post the results

Active database copy shows suspended but we don't know the reason.

Run command: get-mailboxdatabasecopystatus *

Star or asterik will show you index and status of all databases and copies, please run the above commands and paste the output.
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate

Check if the above command helps.
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
Get-MailboxDatabaseCopyStatus | fl *err*

ContentIndexErrorMessage    : An internal error occurred for the database or its index.
ContentIndexErrorCode       : 2
OutstandingDumpsterRequests : {}
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
@Saif: As you see above, I already ran this command and posted the outcome.
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
Please do the following in second server

1. Disable and Stop the below 2 services

Microsoft Exchange Search (MSExchangeFastSearch)
Microsoft Exchange Search Host Controller (HostControllerService)

Rename the folder that looks like below on the fodler in which the database .edb file located.

FolderName - 82575D33-9921-44F0-9FCD-05D7BA30178112.19.Single as 82575D33-9921-44F0-9FCD-05D7BA30178112.19.Single.old

Set those 2 services as automatic and start those 2 services..

2. Wait for the Catalog Index to show as healthy

Get-MailboxDatabaseCopyStatus -Server Server2

3. Activate the Database on second server..

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

4. Wait for sometime and see whether the server 1 database goes to healthy status automatically

Get-MailboxDatabaseCopyStatus -Server Server1
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Create a new Active Directory group that is named "ContentSubmitters," and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.

Force or wait for Active Directory replication.
Restart the following services:
Microsoft Exchange Search
Microsoft Exchange Search Host Controller

Reference Article- https://ril3y.wordpress.com/2013/07/01/exchange-2013-content-index-failure-causes-stalled-mailbox-migration/

I know this is old and is for CU 6, but just try and see if this rings the CI in healthy state.

OR else you will have to rebuild CI in the active node by following below steps:

on the active node then you will have to stop the search services and rename the catalog folder on the active node.

This will rebuild the index for the active node databases.

Once the CI is in healthy state on active node, you can then run command: Update-MailboxDatabaseCopy -Identity DB1\MBX1 -CatalogOnly

This will update the catalog folder only on the passive node. Then try to resume the copy and see of that helps.

If resuming does not help then you need to run the complete command to update  the database copy as well as catalag:

Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
@Rajkumar: I already tried that... this doesn't work for a DAG.
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
did you tried the reseeding of second server DB using -BeginSeed as well ?

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly -BeginSeed
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
I did... not solving my issue.
Rajkumar DuraisamyRajkumar Duraisamy (Rajkumar-MCITP)
Top Expert 2012

Commented:
Try to activate the DB on Server 2 ignoring the index for now and then try to rebuild the index once the DB mounted
MASEE Solution Guide - Technical Dept Head
CERTIFIED EXPERT
Most Valuable Expert 2017

Commented:
Saif ShaikhServer engineer
CERTIFIED EXPERT

Commented:
Try command:
Move-ActiveMailboxDatabase DB1 servername -SkipLagChecks -SkipClientExperienceChecks

The above command will not look for any CI errors and will activate the DB on another server.

Check if this works other wise rebuilding the index is the only option and then running update-mailboxdatabasecopy -catalogonly

By using -catalogonly if this does not work and the DB is in failed and suspended then you will have to update the complete DB copy by using the -deleteexistingfiles
Murat ElmasGeneral Manager, Strategic Planning Director - Computer Engineer, MBA
CERTIFIED EXPERT

Commented:
Hello,

This is actually a common behavior of the exchange server.
My suggestion to you about this is;
First, restart Microsoft Exchange Search and wait a moment.
If there is no improvement, delete the unhealthy secondary copies and make sure that "mount database" is not checked when rebuilding.
If you wait a short time, you will see that the database you just added has failed again.
Press Resume and wait, probably the process will begin and complete after a while, but you will not see healthy again.
After a while, only the index will remain failed. Then restart the search service and wait for a short time.
It may not seem meaningful, but it will be a solution.
It takes a little patience.
Do not forget to mount second database copy after these :)

Have a trouble free day;-)
Murat
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
@MAS: this looks promising...
I created a new Database on the DAG an hour ago and the content index state is still showing "Unknown".
On a new, non-DAG member Server I created a new DB and moved some test mailboxes. This one is showing "Healthy".
MASEE Solution Guide - Technical Dept Head
CERTIFIED EXPERT
Most Valuable Expert 2017

Commented:
Please restart Information store after creating new DB.
MASEE Solution Guide - Technical Dept Head
CERTIFIED EXPERT
Most Valuable Expert 2017

Commented:
Try to set mtu
netsh interface ipv4 set subinterface 13 mtu=1400 store=persistent. Explained here.
https://www.experts-exchange.com/articles/28546/How-to-resolve-Exchange-2013-DR-server-Database-Copy-Status-Displays-unknown-in-the-ECP-console.html
Ehab SalemIT Manager
CERTIFIED EXPERT

Author

Commented:
Set mtu did not solve the issue.
IT Manager
CERTIFIED EXPERT
Commented:
I installed new servers, moved all mailboxes, created a new DAG.
I know it is not the optimum solution, but the DB sizes are not that big, and it worked.