Exchange 2013 DAG Sync fails due to constant configuration changes.

We are in the process of migrating from Exchange 2010 to 2013 and have the co-existence up and running correctly.

Our setup is fairly simple. We have two sites ("COL" and "HAV") connected by a point to point line. Each site has one Exchange 2013 server running on Windows Server 2012 R2 (MAIL-COL-2 and MAIL-HAV-2) with one active mail DB on it and a passive copy of the DB from the other site.The HAV DB is just over 9Gb and the COL DB is about 200Mb, as we've not migrated many mailboxes yet.

The problem is that the nodes are not synchronising the passive DB copies correctly. When I update using Exchange Management Shell (Update-MailboxDatabaseCopy -identity db2013-hav-1\mail-col-2 -DeleteExistingFiles), it transfers between 1Gb and 1.6Gb (it varies between attempts, presumably time specific) before failing with the following error:

The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Communication was
terminated by server 'MAIL-HAV-2': Data could not be read because the communication channel was closed. [Database:
DB2013-HAV-1, Server: MAIL-COL-2.titon.co.uk]

Open in new window


The event log on MAIL-HAV-2 elaborates on this with an MsExchangeRepl Warning Id 4136:

Seed-from-active for database 'DB2013-HAV-1' to the passive copy on server MAIL-COL-2 was cancelled. Error: The seeding
was cancelled for database 'DB2013-HAV-1' because the copy configuration changed on the seeding source 'MAIL-HAV-2' ( a
failover could have happened ) or replay service is shutting down on the seeding source. ReasonCode=ConfigChanged.

Open in new window


This is backed up by several Informational event ID 16028s for various Exchange services. For example:

A forced configuration update for Microsoft.Exchange.Transport.ReceiveConnectorConfiguration has successfully completed.
Object details from  the last notification-based reload: . New details: 

Open in new window


According to Microsoft's documentation, this behaviour is by design, but it seems to be interfering with our DB copy process. There is no physical update of the configuration by any Exchange admin, but these are being recorded every couple of minutes in batches of 7 or 8.

Does anyone have any advice for how to proceed? Thanks in advance.
TitonhwAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Valentina PerezExchange ServersCommented:
Hi Tito,

So the seeding is failing. Could you run the following command in order to check the status database and index:

Get-MailboxdatabaseCopyStatus *

Do you have all the index healthy?

Regards
Valentina
0
TitonhwAuthor Commented:
Hi Valentina,

The active copy of the database has an index status of "Healthy", but the passive copy is in a Failed and Suspended state and has a Content Index Status of "Suspended".

Thanks,
Phil
0
Valentina PerezExchange ServersCommented:
Hi Phil,

Could you try to reseed the database with the following command:

Update-MailboxDatabaseCopy "Name of the database" -DeleteExistingFiles -BeginSeed -SourceServer "source server"

Do you have all the pasive databases with index in failed and suspend?

Regards
Valentina
0
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

timgreen7077Exchange EngineerCommented:
See the below link to assist with seeding your DBs, also consider it there are network interruptions causing the failures.

https://practical365.com/exchange-server/reseeding-a-failed-database-copy-in-an-exchange-server-2016-database-availability-group/
0
TitonhwAuthor Commented:
@Valentina
The smaller COL database has seeded correctly and the indexes are healthy, but it is only 200Mb, so only took about 2 minutes to sync completely. I suspect that if I moved a larger mailbox into it, it would demonstrate the same behaviour as the HAV database.

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
----                                          ------          --------- ----------- --------------------   ------------
DB2013-HAV-1\MAIL-HAV-2                       Mounted         0         0                                  Healthy
DB2013-COL-1\MAIL-HAV-2                       Mounted         0         0                                  Healthy
DB2013-COL-1\MAIL-COL-2                       Healthy         0         0           12/03/2018 15:29:19    Healthy
DB2013-HAV-1\MAIL-COL-2                       Seeding         10953     0                                  FailedAnd...

Open in new window


Running Update-MailboxDatabaseCopy DB2013-HAV-1\MAIL-COL-2 -DeleteExistingFiles -BeginSeed -SourceServer MAIL-HAV-2 also fails after 1 minute and 1 second exactly with the same set of errors, although instead of a warning event ID 4136, it now throws an Error 4026:

[Seed Manager] The seed request for database 'DB2013-HAV-1' encountered an error during seeding. Error: Communication was terminated by server 'MAIL-HAV-2': Data could not be read because the communication channel was closed.

Open in new window


@Tim
Unfortunately I've found and followed that article already. It still throws the same set of errors as I listed in my original post.

I can see no other evidence to suggest that there are network interruptions. While attempting to seed, I copied a 4Gb ISO from one server to another and it was successful and uninterrupted, although the seeding process still failed as before.
0
timgreen7077Exchange EngineerCommented:
Are you attempting to seed from a healthy DB in the same site or are you seeding across the WAN. I would suggest attempting to seed from the same sites if you have already.
Have you also tried to reboot the servers and then open the EAC and resume the DB copies.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Valentina PerezExchange ServersCommented:
Hi,

Have you run the test validation for the cluster? In order to be sure that your network is configured correctly and all the other settings.

When did you start to see this behaviour?

Please verify the version of both exchange. Do you have Exchange 2010 Sp3?

Regards
Valentina
0
TitonhwAuthor Commented:
Hi Tim,

We are seeding from a healthy DB across the WAN, but we have a 100Mbps uncontended line with sub 10ms latency, so it's effectively a low bandwidth LAN connection. It is cross AD site as well.

To be fair, I haven't rebooted both servers. IT 101 etc. That will have to wait until out of office hours.

Thanks,
Phil
0
TitonhwAuthor Commented:
Hi Valentina,

We have seen this behaviour since the DAG was first set up. It has never worked correctly.

The cluster passes all validation tests, although it is suggesting that one of the nodes has some Windows updates outstanding compared to the other node. I will install these immediately.

Our organisation has one Exchange 2010 SP3 (v14.3, build 123.4) and one Exchange 2013 (v15.0, build 1365.1) server at each site.

Thanks,
Phil
0
timgreen7077Exchange EngineerCommented:
I would suggest try reseeding locally instead of cross site even though it low usage bandwidth. if that fails then I would reboot after business hours and try again. After reboot you could possibly resume in EAC or shell whichever you choose.
0
TitonhwAuthor Commented:
Well this is embarrassing. I've restarted both servers and it's now syncing perfectly.

It's possible that it was caused by the disparity of the Windows Updates between the nodes, but realistically I've missed IT 101 here!
0
TitonhwAuthor Commented:
I don't think I can show my face again. Thanks for your help.
0
timgreen7077Exchange EngineerCommented:
Good deal. Happy the reboot helped. I have definitely been there before and a reboot fixed my issues hahaha i know the feeling :)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Database Availability Group (DAG)

From novice to tech pro — start learning today.