Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

Troubleshooting
Research
Professional Opinions
Ask a Question
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

troubleshooting Question

Active Directory not replicating between sites

Avatar of jakert50
jakert50Flag for United States of America asked on
Windows Server 2003Active DirectoryWindows Server 2008
13 Comments1 Solution3495 ViewsLast Modified:
I am running into problems with our two sites not replicating with each other. Here's a quick background story:
SITE1 has 2 domains with 2 domain controllers each (DOMAIN.COM and SITE1.DOMAIN.COM), each part of the same site in AD Sites and Services
SITE2 has 1 domain controller (SITE2.DOMAIN.COM) in it's own site in AD Sites and Services.

The DC at SITE2 was having issues booting from time to time, so a temporary DC was set up to transfer roles to. Once the roles were transferred, the original DC was reformatted/reinstalled and rejoined the domain. The roles were transferred back from the temp DC to the original DC (now with a different name, and now running server 2008 instead of server 2003 R2).
(This was also the first 2008 DC in the forest. All adprep commands were run successfully)

About a week later, I checked the logs on all the servers. I noticed several alarming events:

On the DC on SITE2, the following entrie was found:
Log Name:      Application
Source:        Microsoft-Windows-CertificateServicesClient-CertEnroll
Date:          1/26/2009 10:11:39 AM
Event ID:      13
Task Category: None
Level:         Error
Keywords:      Classic
User:          SYSTEM
Computer:      DC1.SITE2.DOMAIN.COM
Description:
Certificate enrollment for Local system failed to enroll for a DomainController certificate from PDC1.FORESTDOMAIN.COM\DC1 (The RPC server is unavailable. 0x800706ba (WIN32: 1722)).

The following is the contents of dcdiag on this server:
Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = DC1
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE2\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests

   Testing server: SITE2\DC1
      Starting test: Advertising
         ......................... DC1 passed test Advertising
      Starting test: FrsEvent
         ......................... DC1 passed test FrsEvent
      Starting test: DFSREvent
         ......................... DC1 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... DC1 passed test SysVolCheck
      Starting test: KccEvent
         ......................... DC1 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... DC1 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... DC1 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... DC1 passed test NCSecDesc
      Starting test: NetLogons
         ......................... DC1 passed test NetLogons
      Starting test: ObjectsReplicated
         ......................... DC1 passed test ObjectsReplicated
      Starting test: Replications
         ......................... DC1 passed test Replications
      Starting test: RidManager
         ......................... DC1 passed test RidManager
      Starting test: Services
         ......................... DC1 passed test Services
      Starting test: SystemLog
         ......................... DC1 passed test SystemLog
      Starting test: VerifyReferences
         ......................... DC1 passed test VerifyReferences


   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : SITE2
      Starting test: CheckSDRefDom
         ......................... SITE2 passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... SITE2 passed test CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running enterprise tests on : DOMAIN.com
      Starting test: LocatorCheck
         ......................... DOMAIN.com passed test LocatorCheck
      Starting test: Intersite
         ......................... DOMAIN.com passed test Intersite


This error was reported on the forest domain controller (also the CA) :
Event Type:      Error
Event Source:      NTDS Replication
Event Category:      Replication
Event ID:      1864
Date:            1/25/2009
Time:            11:22:27 PM
User:            NT AUTHORITY\ANONYMOUS LOGON
Computer:      PDC1
Description:
This is the replication status for the following directory partition on the local domain controller.
 
Directory partition:
DC=PDC1,DC=DOMAIN,DC=com
 
The local domain controller has not recently received replication information from a number of domain controllers.   The count of domain controllers is shown, divided into the following intervals.
 
More than 24 hours:
1
More than a week:
1
More than one month:
0
More than two months:
0
More than a tombstone lifetime:
0
Tombstone lifetime (days):
60
 Domain controllers that do not replicate in a timely manner may encounter errors. It may miss password changes and be unable to authenticate. A DC that has not replicated in a tombstone lifetime may have missed the deletion of some objects, and may be automatically blocked from future replication until it is reconciled.
 
To identify the domain controllers by name, install the support tools included on the installation  CD and run dcdiag.exe.
You can also use the support tool repadmin.exe to display the replication latencies of the domain controllers in the forest.   The command is "repadmin /showvector /latency <partition-dn>".

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

...And this is the contents of dcdiag on this server:
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE1\PDC1
      Starting test: Connectivity
         ......................... PDC1 passed test Connectivity

Doing primary tests

   Testing server: SITE1\PDC1
      Starting test: Replications
         REPLICATION-RECEIVED LATENCY WARNING
         PDC1:  Current time is 2009-01-26 11:04:42.
            DC=ForestDnsZones,DC=DOMAIN,DC=com
               Last replication recieved from FURY at 2009-01-03 11:15:28.
            CN=Schema,CN=Configuration,DC=DOMAIN,DC=com
               Last replication recieved from FURY at 2009-01-03 11:15:28.
            CN=Configuration,DC=DOMAIN,DC=com
               Last replication recieved from FURY at 2009-01-03 11:15:28.
            DC=SITE2,DC=DOMAIN,DC=com
               Last replication recieved from FURY at 2009-01-03 11:15:28.
         ......................... PDC1 passed test Replications
      Starting test: NCSecDesc
         ......................... PDC1 passed test NCSecDesc
      Starting test: NetLogons
         ......................... PDC1 passed test NetLogons
      Starting test: Advertising
         ......................... PDC1 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... PDC1 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... PDC1 passed test RidManager
      Starting test: MachineAccount
         ......................... PDC1 passed test MachineAccount
      Starting test: Services
         ......................... PDC1 passed test Services
      Starting test: ObjectsReplicated
         ......................... PDC1 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... PDC1 passed test frssysvol
      Starting test: frsevent
         ......................... PDC1 passed test frsevent
      Starting test: kccevent
         ......................... PDC1 passed test kccevent
      Starting test: systemlog
         ......................... PDC1 passed test systemlog
      Starting test: VerifyReferences
         ......................... PDC1 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : domain
      Starting test: CrossRefValidation
         ......................... domain passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... domain passed test CheckSDRefDom

   Running enterprise tests on : domain.com
      Starting test: Intersite
         ......................... domain.com passed test Intersite
      Starting test: FsmoCheck
         ......................... domain.com passed test FsmoCheck

(Note that FURY is the name of the original DC in SITE2 that was reinstalled and renamed.)
I've also noticed that once a week for the past two weeks, DC1 in SITE2 has DNS errors with SITE1 and DOMAIN.COM (I need to manually reload from master, then DNS is OK for another week).

To test that the two sites weren't talking to each other, I added a new domain controller to SITE2.DOMAIN.COM. Within SITE2, AD Sites and Services seemed to look normal--it found the new server, and NTDS settings were automatically generated.
When I switched back to the PDC1 in SITE1, Sites and Services still showed the original FURY server--not showing the replaced DC1 server, nor the new temporary DC that I added to test this.

My original assumption was FSMO roles weren't transferred correctly. However, I checked DC1 via ntdsutil and it looks like everything is in order:
Server "DC1" knows about 5 roles
Schema - CN=NTDS Settings,CN=PDC1,CN=Servers,CN=Site1,CN=Sites,CN=Configuration,DC=domain,DC=com
Naming Master - CN=NTDS Settings,CN=PDC1,CN=Servers,CN=Site1,CN=Sites,CN=Configuration,DC=domain,DC=com
PDC - CN=NTDS Settings,CN=DC1,CN=Servers,CN=Site2,CN=Sites,CN=Configuration,DC=domain,DC=com
RID - CN=NTDS Settings,CN=DC1,CN=Servers,CN=Site2,CN=Sites,CN=Configuration,DC=domain,DC=com
Infrastructure - CN=NTDS Settings,CN=DC1,CN=Servers,CN=Site2,CN=Sites,CN=Configuration,DC=domain,DC=com

I had read online somewhere that the certenroll may be a problem with the correct users not being added to the CERTSVC_DCOM_ACCESS group. I since added Domain Computers, Domain Controllers, and Domain Users for all domains to this group.

What else am I missing that I should check into?
ASKER CERTIFIED SOLUTION
Avatar of jakert50
jakert50Flag of United States of America image

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Commented:
This problem has been solved!
Unlock 1 Answer and 13 Comments.
See Answers