Exchange DAG will all Content index state for DBs failed and suspended

Ensure your Active Directory is up and restart the Exchange servers one by one and check the status..

Active Directory is up for sure, and I have already tried that. didn't work.

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

what is the message when you failover from failed database server to other ?

context index failed please check the following link for the solution

https://practical365.com/exchange-server/exchange-2016-failed-content-index/

all the best

Error: An error occurred while trying to validate the specified database copy for possible activation. Error: EX-01: Database copy 'IT' on server 'EX-01' has content index catalog files in the following state: 'Suspended'. If you need to activate this database copy, you can use the Move-ActiveMailboxDatabase cmdlet with the -SkipClientExperienceChecks parameter to forcibly activate the database.

@Sajid: This is For an Exchange 2016 server that is not a member of a DAG.

You have 2 options now..

Activate the db on second server skipping the content index skip

Or fix the catalog index and then activate.. what is your opinion?

My main interest is to have content indexes on both servers ok.
Would that be second option?

Please follow the steps like below..

1. Update the Suspended Catalog on Server 2 with the below command

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly
Update-MailboxDatabaseCopy -Identity DB2\Server2 -CatalogOnly
etc

2. Wait for the Catalog Index to show as healthy

Get-MailboxDatabaseCopyStatus -Server Server2

3. Activate the Database on second server..

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

4. Wait for sometime and see whether the server 1 database goes to healthy status automatically

Get-MailboxDatabaseCopyStatus -Server Server1

If any issues.. Please revert with the error messages..

Note: Updating the catalog index will take some time to index based on the size of the database..

If you don't want to wait.. follow the below steps

1. Activate the Database in Second Server (ignoring the catalog validation in Second Server)

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2 -SkipClientExperienceChecks

2. Once the Database activated in second server.. rebuild the index for database mounted in Second Server

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly

3.wait for some time later  if the server 1 not automatically goes to healthy state.. run the below command

Update-MailboxDatabaseCopy -Identity DB1\Server1 -Databaseonly

Before executing.. if you have any queries.. please clarify and then follow the steps..

Run get-mailboxdatabasecopystatus *

To check whether the CI is is healthy or failed state on the active node.

If it is in failed state on the active node then you will have to stop the search services and rename the catalog folder on the active node.

This will rebuild the index for the active node databases.

Once the CI is in healthy state on active node, you can then run command: Update-MailboxDatabaseCopy -Identity DB1\MBX1 -CatalogOnly

This will update the catalog folder only on the passive node. Then try to resume the copy and see of that helps.

If resuming does not help then you need to run the complete command to update  the database copy as well as catalag:

Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly

Doesn't this need a valid index to seed from?

what is the status of Index on the other server?

Yes it needs valid index from the active database copy.

Run command: get-mailboxdatabasecopystatus *

to see whether the index is in healthy on the active database copy. Then only the catalog index reseed will work cause it reads from the active node index.

what is the status of Index on the other server?
Failed

[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus it | sort name | Select name,status,contentindexstate

Name                                                                     Status                       ContentIndexState
IT\MAIL-01                                                             Mounted                  Suspended
IT\MAIL-02                                                             Healthy                     Failed

Run command: Get-MailboxDatabaseCopyStatus | fl *err*

and post the results

Active database copy shows suspended but we don't know the reason.

Run command: get-mailboxdatabasecopystatus *

Star or asterik will show you index and status of all databases and copies, please run the above commands and paste the output.

Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate

Check if the above command helps.

Get-MailboxDatabaseCopyStatus | fl *err*

ContentIndexErrorMessage    : An internal error occurred for the database or its index.
ContentIndexErrorCode       : 2
OutstandingDumpsterRequests : {}

@Saif: As you see above, I already ran this command and posted the outcome.

Please do the following in second server

1. Disable and Stop the below 2 services

Microsoft Exchange Search (MSExchangeFastSearch)
Microsoft Exchange Search Host Controller (HostControllerService)

Rename the folder that looks like below on the fodler in which the database .edb file located.

FolderName - 82575D33-9921-44F0-9FCD-05D7BA30178112.19.Single as 82575D33-9921-44F0-9FCD-05D7BA30178112.19.Single.old

Set those 2 services as automatic and start those 2 services..

2. Wait for the Catalog Index to show as healthy

Get-MailboxDatabaseCopyStatus -Server Server2

3. Activate the Database on second server..

Move-ActiveMailboxDatabase -Server EX1 -ActivateOnServer EX2

4. Wait for sometime and see whether the server 1 database goes to healthy status automatically

Get-MailboxDatabaseCopyStatus -Server Server1

Create a new Active Directory group that is named "ContentSubmitters," and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.

Force or wait for Active Directory replication.
Restart the following services:
Microsoft Exchange Search
Microsoft Exchange Search Host Controller

Reference Article- https://ril3y.wordpress.com/2013/07/01/exchange-2013-content-index-failure-causes-stalled-mailbox-migration/

I know this is old and is for CU 6, but just try and see if this rings the CI in healthy state.

OR else you will have to rebuild CI in the active node by following below steps:

on the active node then you will have to stop the search services and rename the catalog folder on the active node.

This will rebuild the index for the active node databases.

Once the CI is in healthy state on active node, you can then run command: Update-MailboxDatabaseCopy -Identity DB1\MBX1 -CatalogOnly

This will update the catalog folder only on the passive node. Then try to resume the copy and see of that helps.

If resuming does not help then you need to run the complete command to update  the database copy as well as catalag:

Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2

@Rajkumar: I already tried that... this doesn't work for a DAG.

did you tried the reseeding of second server DB using -BeginSeed as well ?

Update-MailboxDatabaseCopy -Identity DB1\Server2 -CatalogOnly -BeginSeed

I did... not solving my issue.

Try to activate the DB on Server 2 ignoring the index for now and then try to rebuild the index once the DB mounted

Did you cry creating a new Database (in DAG) and move all your mailboxes to  the new one?
https://www.experts-exchange.com/articles/31741/Move-mailboxes-in-bulk-or-one-by-one-to-another-database-in-Exchange-2010-2013-2016.html

Get a count of mailboxes
https://www.experts-exchange.com/articles/31850/Count-Mailboxes-in-your-Exchange-Organization.html

Try command:
Move-ActiveMailboxDatabase DB1 servername -SkipLagChecks -SkipClientExperienceChecks

The above command will not look for any CI errors and will activate the DB on another server.

Check if this works other wise rebuilding the index is the only option and then running update-mailboxdatabasecopy -catalogonly

By using -catalogonly if this does not work and the DB is in failed and suspended then you will have to update the complete DB copy by using the -deleteexistingfiles

Hello,

This is actually a common behavior of the exchange server.
My suggestion to you about this is;
First, restart Microsoft Exchange Search and wait a moment.
If there is no improvement, delete the unhealthy secondary copies and make sure that "mount database" is not checked when rebuilding.
If you wait a short time, you will see that the database you just added has failed again.
Press Resume and wait, probably the process will begin and complete after a while, but you will not see healthy again.
After a while, only the index will remain failed. Then restart the search service and wait for a short time.
It may not seem meaningful, but it will be a solution.
It takes a little patience.
Do not forget to mount second database copy after these :)

Have a trouble free day;-)
Murat

@MAS: this looks promising...
I created a new Database on the DAG an hour ago and the content index state is still showing "Unknown".
On a new, non-DAG member Server I created a new DB and moved some test mailboxes. This one is showing "Healthy".

Please restart Information store after creating new DB.

Try to set mtu
netsh interface ipv4 set subinterface 13 mtu=1400 store=persistent. Explained here.
https://www.experts-exchange.com/articles/28546/How-to-resolve-Exchange-2013-DR-server-Database-Copy-Status-Displays-unknown-in-the-ECP-console.html

Set mtu did not solve the issue.