• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1301
  • Last Modified:

DC replication failing... need to troubleshoot

We have two domain controllers in our environment.  Replication doesn't appear to be working and apparently hasn't been working since the beginning of February.

The problem originally manifested itself when I tried to join a computer to the domain and it failed. The event log stated:

The session setup from computer 'MYCOMPANY056-PC' failed because the security database does not contain a trust account 'MYCOMPANY056-PC$' referenced by the specified computer.  

USER ACTION  
If this is the first occurrence of this event for the specified computer and account, this may be a transient issue that doesn't require any action at this time.  If this is a Read-Only Domain Controller and 'MYCOMPANY056-PC$' is a legitimate machine account for the computer 'MYCOMPANY056-PC' then 'MYCOMPANY056-PC' should be marked cacheable for this location if appropriate or otherwise ensure connectivity to a domain controller  capable of servicing the request (for example a writable domain controller).  Otherwise, the following steps may be taken to resolve this problem:  

If 'MYCOMPANY056-PC$' is a legitimate machine account for the computer 'MYCOMPANY056-PC', then 'MYCOMPANY056-PC' should be rejoined to the domain.  

If 'MYCOMPANY056-PC$' is a legitimate interdomain trust account, then the trust should be recreated.  

Otherwise, assuming that 'MYCOMPANY056-PC$' is not a legitimate account, the following action should be taken on 'MYCOMPANY056-PC':  

If 'MYCOMPANY056-PC' is a Domain Controller, then the trust associated with 'MYCOMPANY056-PC$' should be deleted.  

If 'MYCOMPANY056-PC' is not a Domain Controller, it should be disjoined from the domain.

Open in new window



On the problematic domain controller I'm also seeing troubling events like this:

The DNS server could not bind a User Datagram Protocol (UDP) socket to 198.249.244.253. The event data is the error code. Restart the DNS server or reboot your computer.   (DATA WAS: 2741)

Open in new window


The DNS server could not open socket for address 198.249.244.253. 
Verify that this is a valid IP address for the server computer.  If it is NOT valid use the Interfaces dialog under Server Properties in the DNS Manager to remove it from the list of IP interfaces.  Then stop and restart the DNS server. (If this was the only IP interface on this machine and the DNS server may not have started as a result of this error.  In that case remove the DNS\Parameters\ ListenAddress value in the services section of the registry and restart.) 
 
If this is a valid IP address for this machine, make sure that no other application (e.g. another DNS server) is running that would attempt to use the DNS port. 
 
For more information, see "DNS server log reference" in the online Help

Open in new window


The DNS server has encountered a critical error from the Active Directory. Check that the Active Directory is functioning properly. The extended error debug information (which may be empty) is "". The event data contains the error.

Open in new window


The DFS Replication service failed to contact domain controller  to access configuration information. Replication is stopped. The service will try again during the next configuration polling cycle, which will occur in 60 minutes. This event can be caused by TCP/IP connectivity, firewall, Active Directory Domain Services, or DNS issues. 
 
Additional Information: 
Error: 160 (One or more arguments are not correct.)

Open in new window


NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)

Open in new window


Active Directory Domain Services could not resolve the following DNS host name of the source domain controller to an IP address. This error prevents additions, deletions and changes in Active Directory Domain Services from replicating between one or more domain controllers in the forest. Security groups, group policy, users and computers and their passwords will be inconsistent between domain controllers until this error is resolved, potentially affecting logon authentication and access to network resources. 
 
Source domain controller: 
 MYCOMPANYDC 
Failing DNS host name: 
 1c5c3ac5-xxxx-xxxx-xxxx-1db0a998a98a._msdcs.MYCOMPANY.com 
 
NOTE: By default, only up to 10 DNS failures are shown for any given 12 hour period, even if more than 10 failures occur.  To log all individual failure events, set the following diagnostics registry value to 1: 
 
Registry Path: 
HKLM\System\CurrentControlSet\Services\NTDS\Diagnostics\22 DS RPC Client 
 
User Action: 
 
 1) If the source domain controller is no longer functioning or its operating system has been reinstalled with a different computer name or NTDSDSA object GUID, remove the source domain controller's metadata with ntdsutil.exe, using the steps outlined in MSKB article 216498. 
 
 2) Confirm that the source domain controller is running Active Directory Domain Services and is accessible on the network by typing "net view \\<source DC name>" or "ping <source DC name>". 
 
 3) Verify that the source domain controller is using a valid DNS server for DNS services, and that the source domain controller's host record and CNAME record are correctly registered, using the DNS Enhanced version of DCDIAG.EXE available on http://www.microsoft.com/dns 
 
  dcdiag /test:dns 
 
 4) Verify that this destination domain controller is using a valid DNS server for DNS services, by running the DNS Enhanced version of DCDIAG.EXE command on the console of the destination domain controller, as follows: 
 
  dcdiag /test:dns 
 
 5) For further analysis of DNS error failures see KB 824449: 
   http://support.microsoft.com/?kbid=824449 
 
Additional Data 
Error value: 
 11004 The requested name is valid, but no data of the requested type was found. 

Open in new window



Dynamic registration or deregistration of one or more DNS records failed with the following error: 
No DNS servers configured for local system.

Open in new window



The Intersite Messaging service terminated with the following error: 
The specified server cannot perform the requested operation.

Open in new window



The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed. 
 
Attempts:
725 
Directory service:
CN=NTDS Settings,CN=MYCOMPANYDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=MYCOMPANY,DC=com 
Period of time (minutes):
41161 
 
The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed. 
 
Additional Data 
Error value:
8524 The DSA operation is unable to proceed because of a DNS lookup failure.

Open in new window



Running DCDIAG on my GOOD domain controller (MYCOMPANYDC) appears to return successful results.

Running DCDIAG on the BAD domain controller (MYCOMPANYDC2) returns a lot of errors:

...

      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems. 

...

      Starting test: KccEvent
         A warning event occurred.  EventID: 0x8000051C
            Time Generated: 03/11/2015   18:30:09
            Event String:
            The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed. 

...

      Starting test: NCSecDesc
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have 
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=ForestDnsZones,DC=MYCOMPANY,DC=com
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have 
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=DomainDnsZones,DC=MYCOMPANY,DC=com
         ......................... MYCOMPANYSERVER2 failed test NCSecDesc

...

      Starting test: Replications
         [Replications Check,MYCOMPANYSERVER2] A recent replication attempt
         failed:
            From MYCOMPANYDC to MYCOMPANYSERVER2
            Naming Context: DC=ForestDnsZones,DC=MYCOMPANY,DC=com
            The replication generated an error (8524):
            The DSA operation is unable to proceed because of a DNS lookup failure.
            
            The failure occurred at 2015-03-11 18:25:39.
            The last success occurred at 2015-02-09 21:50:29.
            725 failures have occurred since the last success.
            The guid-based DNS name
            1c5c3ac5-3823-453a-aec3-1db0a998a98a._msdcs.MYCOMPANY.com
            is not registered on one or more DNS servers.

...

      Starting test: SystemLog
         A warning event occurred.  EventID: 0x80040020
            Time Generated: 03/11/2015   18:10:31
            Event String:
            The driver detected that the device \Device\Harddisk0\DR0 has its write cache enabled. Data corruption may occur.
         A warning event occurred.  EventID: 0x80040020
            Time Generated: 03/11/2015   18:10:31
            Event String:
            The driver detected that the device \Device\Harddisk0\DR0 has its write cache enabled. Data orruption may occur.
         A warning event occurred.  EventID: 0x80040020
            Time Generated: 03/11/2015   18:10:31
            Event String:
            The driver detected that the device \Device\Harddisk0\DR0 has its write cache enabled. Data corruption may occur.
         A warning event occurred.  EventID: 0x00000079
            Time Generated: 03/11/2015   18:10:42
            Event String:
            The firewall exception to allow Internet Storage Name Server (iSNS) client functionality is not enabled. iSNS client functionality is not available.
         A warning event occurred.  EventID: 0x8000001D
            Time Generated: 03/11/2015   18:10:44
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons, or the KDC certificate could not be verified. Smart card logon may not function correctly if this problem is not resolved. To correct this problem, either verify the existing KDC certificate using certutil.exe or enroll for a new KDC certificate.
         A warning event occurred.  EventID: 0x00000C18
            Time Generated: 03/11/2015   18:10:46
            Event String:
            The primary Domain Controller for this domain could not be located.
         An error event occurred.  EventID: 0xC0001B6F
            Time Generated: 03/11/2015   18:10:56
            Event String:
            The Intersite Messaging service terminated with the following error: 

        A warning event occurred.  EventID: 0x00001696
            Time Generated: 03/11/2015   18:11:15
            Event String:
            Dynamic registration or deregistration of one or more DNS records failed with the following error: 

         A warning event occurred.  EventID: 0x00000081
            Time Generated: 03/11/2015   18:11:17
            Event String:
            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)
         A warning event occurred.  EventID: 0x00000081
            Time Generated: 03/11/2015   18:11:18
            Event String:
            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)
         ......................... MYCOMPANYSERVER2 failed test SystemLog


...

Open in new window




So.... obviously something is broken on this DC. How do I go about troubleshooting it?



A couple things that I wonder if they might be contributing to the issue:

1) It's possible there was a power outage on this server resulting in a hard shutdown

2) The server that is having trouble does have an IPv6 address, obtained automatically via dhcp, but I notice that it is not able to ping any other servers via their IPv6 address. I get request timed out. On other Windows 7 machines on my network I can ping via the ipv6 address and it works.

3) I notice that connecting to servers via their hostname takes a long time
0
Frosty555
Asked:
Frosty555
1 Solution
 
David Johnson, CD, MVPOwnerCommented:
MYCOMPANY056-PC  did you follow the instructions and disjoin and rejoin this pc to the domain?
0
 
Frosty555Author Commented:
I did and it joined the domain, but had various issues trying to access file shares, apply group policies etc. saying that the "Target account name is incorrect".

These are new PCs I was setting up, and the issue with joining them to the domain lead me to discover this replication issue, which is now my main priority. The MYCOMPANY056-PC has been shelved until this issue is solved.
0
 
Frosty555Author Commented:
I suspect this is an IPv6 issue... because MYCOMPANYserver2, the DC I am having trouble with, is having problems with that. See my related question if it helps:

http://www.experts-exchange.com/Networking/Protocols/DNS/Q_28634392.html
0
How do you know if your security is working?

Protecting your business doesn’t have to mean sifting through endless alerts and notifications. With WatchGuard Total Security Suite, you can feel confident that your business is secure, meaning you can get back to the things that have been sitting on your to-do list.

 
Mohammed KhawajaManager - Infrastructure:  Information TechnologyCommented:
It might be easier to demote the server and promote it again.  Is there any good reason for enabling IPv6?
0
 
Josh RoweCommented:
I agree with Mohammed. If you have at least one good domain controller to replicate from, then I would be demoting the bad one, and then promoting it again back to a domain controller. Keep in mind though you may not be able to cleanly demote it either. In which case it could easier again to down it, forcibly remove it, and run up a new DC. I would also be disabling IPv6.
This might just save you the headache of hours or even days trying to solve replication issues.
0
 
Zacharia KurianAdministrator- Data Center & NetworkCommented:
Things you can try before demoting the server are;

1. Make sure the IPv4 address are correct and you can ping to your PDC. Remove IPV6.
2. Disable the windows firewall/ Anti virus for test purposes.
3. Make sure the are no hard ware related issues.
4. Make sure the DNS Ip points to itself and you have added the PDC IPs too. Test the DNS (remove all event logs before you run any DCDIAG test).
5. From the command line, execute repadmin /showrepl * (see when it started to give replications issues.)
6. Further, run repadmin /syncall and see if it throws any errors.
7.  It is most likely an issue with  DNS /corrupted entry in sysvol. But you need to carefully check it out. Also make sure that all the AD related services are up & running.

At the mean time create another additional healthy DC and then if the above steps do not help you, then you start thinking of  demoting the troubled one.
0
 
Frosty555Author Commented:
Turned out it indeed was an ipv6 issue. See my other question here: http://www.experts-exchange.com/Networking/Protocols/DNS/Q_28634392.html

The local IP addresses on this particular network start with 198, which means that the 6to4 tunneling adapter considers it to be a "public" IP address and kicks in assigning 2002:: addresses to everything. I'm not sure why this became a problem now as opposed to earlier, but once I disabled ipv6 tunnel adapters and all systems started using the fe80:: link-local ipv6 addresses, and once the DNS server was updated to reflect that, everything sprang back to life and it all seems to be working now.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now