DFSRMIG migration state inconsistent

Setup

5 2008 R2 Domain Controllers across 4 sites
Functional Level 2008 R2
All Global Catalog Servers

I recently had one DC die and had to seize the FSMO roles on another DC then build a new DC to replace the failed one. The new DC has a new name and IP address assigned. Running dcdiag I can confirm that replication is working across all 5 DC's except for one error on the new DC

Starting test: VerifyReferences
   Some objects relating to the DC DC3 have problems:
      [1] Problem: Missing Expected Value
       Base Object:
      CN=C3,OU=Domain Controllers,DC=domain,DC=com
       Base Object Description: "DC Account Object"
       Value Object Attribute Name: frsComputerReferenceBL
       Value Object Description: "SYSVOL FRS Member Object"
       Recommended Action: See Knowledge Base Article: Q312862

   ......................... DC3 failed test VerifyReferences


The article referenced does not apply as the domain was built at 2008 R2 level and DFRS has been in place as far as I know since the beginning (I started with the company after this was in place). No other DC's provide this error. I began digging deeper and upon checking the DFSR Migration state found that DC3 is not in sync. Here are the results from some dfsrmig commands

C:\Windows\system32>dfsrmig.exe /getglobalstate

Current DFSR global state: 'Eliminated'
Succeeded.

C:\Windows\system32>dfsrmig.exe /getmigrationstate

The following Domain Controllers are not in sync with Global state ('Eliminated'):

Domain Controller (Local Migration State) - DC Type
===================================================

dc3 ('Start') - Writable DC

Migration has not yet reached a consistent state on all Domain Controllers.
State information might be stale due to AD latency.


C:\Windows\system32>repadmin /replsum
Replication Summary Start Time: 2015-04-01 10:10:41

Beginning data collection for replication summary, this may take awhile:
  ........


Source DSA                largest delta    fails/total %%   error
 dcremoteofc2              06m:43s    0 /   5    0
 dcremoteofc1              14m:30s    0 /   5    0
 dc1                                 21m:43s    0 /  10    0
 dc2                                14m:30s    0 /  15    0
 dc3                                 06m:41s    0 /   5    0


Destination DSA     largest delta    fails/total %%   error
 dcremoteofc1              13m:57s    0 /   5    0
 dcremoteofc2              04m:25s    0 /   5    0
 dc1                                 14m:31s    0 /  10    0
 dc2                                 21m:45s    0 /  15    0
 dc3                                 03m:38s    0 /   5    0



I found this excellent write up which describes how to migrate

http://blogs.technet.com/b/filecab/archive/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process.aspx

but as I said earlier this is not a migration. DC3 was built to replace a failed dc. DC3 has been in place for about a month now. Has anyone seen this before? Would it be safe to modify the registry key of DC3 to which controls this to '3' which is "Eliminated" state? Should I start the migration process over from the FSMO role holder?

In addition I do not have a share of 'sysvol_DFRS' but only 'sysvol' exists.

I have also attempted to force this by issuing

repadmin/syncall /aed
LVL 1
Snagajob ITAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

MaheshArchitectCommented:
Since you have directly installed Domain at 2008 R2 functional level, you won't find Sysvol_DFSR folder
U will find it only if your sysvol is migrated from FRS Sysvol (Original domain DC is installed with windows 2003 DC OR with 2003 domain \ forest functional level)
Also you can install DFSR management tools from windows features on DC and verifiy if DFSR propagation test running successfully
 
As far as I suspect, your DC promotion is not happened correctly

My suggestion is just forcefully remove AD from faulty DC by running dcpromo /forceremoval
Then make metadata cleanup
https://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx

After that force AD replication and check AD health again by running
dcdiag /v
repadmin /showreps
 Then build new DC again with new name
Ensure that sysvol and netlogon are shared and you will get event id 1394 in directory service logs
Finally run dfsrmig /GetGlobalState again on that DC to verify if it gets eliminated state
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Snagajob ITAuthor Commented:
This DC is located at our remote data center with our Exchange server. Would it be better to build a new DC at that location before forcing removal of the out of sync one? I think Exchange would perform slowly having to perform topology look up across a VPN to a remote DC. I don't recall any errors during DC promotion and sysvol replication is working along with AD replication. I'll do some further research. Thanks for the reply.
0
Snagajob ITAuthor Commented:
when running the command

repladmin /showrepl servername


against any of the DC's from DC3 I get results that show successful. I am trying to avoid a forceful demotion and metadata cleanup for the second time in a month.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

MaheshArchitectCommented:
The problem here is dfsrmig not showing you "Eliminated" state which means there is some problem with that particular DC

Unless you get "Eliminated" state that DC will not work correctly

Unfortunately I don't see any way other than decommission DC and promote it with new name

Have you checked that AD ports are opened as appropriate between local and remote site?
Check with PortQueryUI tool

Also install DFSR tools from windows server features on DC and check DFSR status there
0
Snagajob ITAuthor Commented:
I have disabled the Windows firewall for testing purposes with the same result.
I have also installed the DFSR tools and there are no reported errors from the servers I have checked on. Based on the lack of community feedback and the rarity of the issue I will proceed to decommission the current server and setup a new one.
0
Snagajob ITAuthor Commented:
I built a new DC with a different name and forcefully removed the one that was not in sync. DFSRMIG is now showing all DC's are in global state 'eliminate', no errors with DCDIAG or REAPADMIN.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.