Active Directory Operations Master down at primary site. No replication to remote site

Experts,

Recently I had an Operations Master go down at my primary site. With your help, I seized the roles of the server that was down. I then did some metadata cleanup, and everything seems to be working just fine at my primary site.

DC1 and DC2 are at the primary site and DC1 went down.
DC3 and DC4 are at the secondary site

My secondary site is no longer replicating to my primary site, and I noticed that the operations master is still showing up (under domain controllers) at the second site.


This is a dcdiag at the remote site:

\>dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SchillerPark\BRETDC3SP
      Starting test: Connectivity
         ......................... BRETDC3SP passed test Connectivity

Doing primary tests

   Testing server: SchillerPark\BRETDC3SP
      Starting test: Replications
         REPLICATION-RECEIVED LATENCY WARNING
         BRETDC3SP:  Current time is 2013-09-08 16:28:15.
            DC=ForestDnsZones,DC=bretfordhq,DC=local
               Last replication recieved from BRETDC1FP at 2013-09-06 16:46:25.
               Last replication recieved from BRETDC2FP at 2013-09-06 15:53:11.
            DC=DomainDnsZones,DC=bretfordhq,DC=local
               Last replication recieved from BRETDC1FP at 2013-09-06 16:46:25.
               Last replication recieved from BRETDC2FP at 2013-09-06 16:04:23.
            CN=Schema,CN=Configuration,DC=bretfordhq,DC=local
               Last replication recieved from BRETDC1FP at 2013-09-06 16:46:24.
               Last replication recieved from BRETDC2FP at 2013-09-06 15:53:11.
            CN=Configuration,DC=bretfordhq,DC=local
               Last replication recieved from BRETDC1FP at 2013-09-06 16:46:24.
               Last replication recieved from BRETDC2FP at 2013-09-06 16:11:24.
            DC=bretfordhq,DC=local
               Last replication recieved from BRETDC1FP at 2013-09-06 16:46:23.
               Last replication recieved from BRETDC2FP at 2013-09-06 16:46:08.
         REPLICATION-RECEIVED LATENCY WARNING
          Source site:
         CN=NTDS Site Settings,CN=BretfordHQ,CN=Sites,CN=Configuration,DC=bretfo
rdhq,DC=local
          Current time: 2013-09-08 16:28:15
          Last update time: 2013-09-06 16:11:09
          Check if source site has an elected ISTG running.
          Check replication from source site to this server.
         ......................... BRETDC3SP passed test Replications
      Starting test: NCSecDesc
         ......................... BRETDC3SP passed test NCSecDesc
      Starting test: NetLogons
         ......................... BRETDC3SP passed test NetLogons
      Starting test: Advertising
         ......................... BRETDC3SP passed test Advertising
      Starting test: KnowsOfRoleHolders
         [BRETDC1FP] DsBindWithSpnEx() failed with error 1722,
         The RPC server is unavailable..
         Warning: BRETDC1FP is the Schema Owner, but is not responding to DS RPC
 Bind.
         [BRETDC1FP] LDAP search failed with error 58,
         The specified server cannot perform the requested operation..
         Warning: BRETDC1FP is the Schema Owner, but is not responding to LDAP B
ind.
         Warning: BRETDC1FP is the Domain Owner, but is not responding to DS RPC
 Bind.
         Warning: BRETDC1FP is the Domain Owner, but is not responding to LDAP B
ind.
         Warning: BRETDC1FP is the PDC Owner, but is not responding to DS RPC Bi
nd.
         Warning: BRETDC1FP is the PDC Owner, but is not responding to LDAP Bind
.
         Warning: BRETDC1FP is the Rid Owner, but is not responding to DS RPC Bi
nd.
         Warning: BRETDC1FP is the Rid Owner, but is not responding to LDAP Bind
.
         Warning: BRETDC1FP is the Infrastructure Update Owner, but is not respo
nding to DS RPC Bind.
         Warning: BRETDC1FP is the Infrastructure Update Owner, but is not respo
nding to LDAP Bind.
         ......................... BRETDC3SP failed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... BRETDC3SP failed test RidManager
      Starting test: MachineAccount
         ......................... BRETDC3SP passed test MachineAccount
      Starting test: Services
         ......................... BRETDC3SP passed test Services
      Starting test: ObjectsReplicated
         ......................... BRETDC3SP passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... BRETDC3SP passed test frssysvol
      Starting test: frsevent
         ......................... BRETDC3SP passed test frsevent
      Starting test: kccevent
         ......................... BRETDC3SP passed test kccevent
      Starting test: systemlog
         An Error Event occured.  EventID: 0x40011006
            Time Generated: 09/08/2013   15:49:33
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x40011006
            Time Generated: 09/08/2013   16:19:33
            (Event String could not be retrieved)
         ......................... BRETDC3SP failed test systemlog
      Starting test: VerifyReferences
         ......................... BRETDC3SP passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : bretfordhq
      Starting test: CrossRefValidation
         ......................... bretfordhq passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... bretfordhq passed test CheckSDRefDom

   Running enterprise tests on : bretfordhq.local
      Starting test: Intersite
         ......................... bretfordhq.local passed test Intersite
      Starting test: FsmoCheck
         Error: The server returned by DsGetDcName() did not match DsListRoles()
 for the PDC
         ......................... bretfordhq.local passed test FsmoCheck




Here is the repadmin command



>repadmin /showconn

repadmin running command /showconn against server localhost

Base DN: CN=SchillerPark,CN=Sites,CN=Configuration,DC=bretfordhq,DC=local
==== KCC CONNECTION OBJECTS ============================================
Connection --
    Connection name : 90494e96-933e-4c68-84f2-63b957969fc1
    Server DNS name : bretdc4sp.bretfordhq.local
    Server DN  name : CN=NTDS Settings,CN=BRETDC4SP,CN=Servers,CN=SchillerPark,C
N=Sites,CN=Configuration,DC=bretfordhq,DC=local
        Source: BretfordHQ\BRETDC1FP
******* 15 CONSECUTIVE FAILURES since 2013-09-06 16:46:25
Last error: 1256 (0x4e8):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
        TransportType: IP
        options:  isGenerated
        ReplicatesNC: CN=Configuration,DC=bretfordhq,DC=local
        Reason:  IntersiteTopology
                Replica link has been added.
        ReplicatesNC: DC=DomainDnsZones,DC=bretfordhq,DC=local
        Reason:  IntersiteTopology
                Replica link has been added.
        ReplicatesNC: DC=bretfordhq,DC=local
        Reason:  IntersiteTopology
                Replica link has been added.
        ReplicatesNC: DC=ForestDnsZones,DC=bretfordhq,DC=local
        Reason:  IntersiteTopology
                Replica link has been added.
Connection --
    Connection name : b3a9b20b-3447-43e7-9b13-421bd1261399
    Server DNS name : bretdc4sp.bretfordhq.local
    Server DN  name : CN=NTDS Settings,CN=BRETDC4SP,CN=Servers,CN=SchillerPark,C
N=Sites,CN=Configuration,DC=bretfordhq,DC=local
        Source: SchillerPark\BRETDC3SP
                No Failures.
        TransportType: intrasite RPC
        options:  isGenerated
        ReplicatesNC: CN=Configuration,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: CN=Schema,CN=Configuration,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=DomainDnsZones,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=ForestDnsZones,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
Connection --
    Connection name : b7d30839-1bed-4e18-8387-291fec4091d4
    Server DNS name : bretdc3sp.bretfordhq.local
    Server DN  name : CN=NTDS Settings,CN=BRETDC3SP,CN=Servers,CN=SchillerPark,C
N=Sites,CN=Configuration,DC=bretfordhq,DC=local
        Source: SchillerPark\BRETDC4SP
                No Failures.
        TransportType: intrasite RPC
        options:  isGenerated
        ReplicatesNC: CN=Configuration,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: CN=Schema,CN=Configuration,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=DomainDnsZones,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
        ReplicatesNC: DC=ForestDnsZones,DC=bretfordhq,DC=local
        Reason:  RingTopology
                Replica link has been added.
3 connections found.


Looks like the servers (DC3 and DC4) are still looking for DC1
Should I go through the same metadata cleanup at the secondary site?
What are the recommended transports to be used between sites?

Any suggestions would be appreciated.

Thank you in advance

Don
dwesolowiczAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

N-WCommented:
Have you removed DC1 from AD Sites and Services, when connected to either DC3 or DC4?
It looks as though your secondary site is setup to only replicate with DC1.

To resolve this, first remove any connections from the other DCs to DC1 (under Sites --> [your_site] --> Servers --> [your_server] --> NTDS Settings), then delete DC1 completely (right click DC1 and select delete).

Once DC1 is removed, you may need to setup a new connection from DC3 or DC4 to replicate with DC2. I.e. under Sites --> [your_site] --> Servers --> DC3 --> NTDS Settings, create a new connection to DC2.

Then run this command to replicate the changes: "repadmin /replicate DC2.yourdomain.com DC3.yourdomain.com dc=yourdomain,dc=com".

Note: the above should only be run from either DC3 or DC4, not both. When you make these changes, DC3 and DC4 should automatically replicate the changes.

Once the new connections are setup to DC2 and replication is occurring, DC3 and DC4 will become aware of the new FSMO changes. Just remember to be patient with the replication, it can take a few hours for everything to sync up fully.
0
dwesolowiczAuthor Commented:
Thank you for your reply! I did not remove DC1 from sites and services at the remote site. Im not near a PC at the moment, but will give this a shot tomorrow an let you know how things go.
0
jmanishbabuCommented:
Create a new site link with DC2 to other DC's that should resolve replication issues.
0
jmanishbabuCommented:
Connection objects will be automatically created .
0
SandeshdubeySenior Server EngineerCommented:
As you have mentioned that metadata cleanup is perfromed.Ensure that dns setting of other DC and clienrts is pointing to online DC.Also ensure correct dns setting on DC as this:http://abhijitw.wordpress.com/2012/03/03/best-practices-for-dns-client-settings-on-domain-controller/

Dont create manaul connection,if create delete the same and lett kcc design the AD replication topology:http://blogs.technet.com/b/markmoro/archive/2012/10/26/3445433.aspx

Wiat for replication to complete and then perform diagnosis test by dcdiag and repadmin /replsum.

If still issue pertsist post the dcdiag /q and repadmin /replsum form all DCs.Also ensure that instances of faulty DC is remove from ADdatabase,dns,AD sites and service ,DC OU.http://sandeshdubey.wordpress.com/2011/10/12/metadata-cleanup-of-a-domain-controller/
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.