DNS/Domain Contoller issues - can't ping domain, no replication

I've been noticing some strange things on the Primary DC and Secondary DC as well (Primary is 2008 R2, Secondary is 2016 - both VMs and on hyper-v) and i'm not sure how to resolve the issue(s)

troubleshooting.PNG
I see the above on the Primary - the Secondary and additional client server I have on the domain just show as being in a private network

I cannot ping the domain from any server

Opening DNS on either DC gives me an access denied

Looking at the event viewer on the primary DC these are the relevant Event IDs

DNS:
Errors  4000 , 4007
Information 408

Active Directory Domain Services:
Error 1126
Warn 2092, 1655

ADUC opens on primary and ADSS opens on primary without any errors

Currently using host files w/hardcoded IPs to sidestep the issue

Was thinking about blowing out the virtual switch, removing the NIC team, then re-doing the NIC team and setting up the vSwitch again on the physical host that serves up the VMs but am waiting for a proper maintenance window to give that a shot and honestly not sure if it would even resolve it. It's almost like the primary will no longer talk to itself.  Please let me know if you need any additional information to resolve, thanks
nflynn85Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Justin EvansCommented:
from the primary Domain Controller can you run DCDIAG /fix

and then send the results of  dcdiag test dns

paste them here and remove the sensitive information.
0
nflynn85Author Commented:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>dcdiag /fix

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = abc
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\abc
      Starting test: Connectivity
         The host 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local could
         not be resolved to an IP address. Check the DNS server, DHCP, server
         name, etc.
         Got error while checking LDAP and RPC connectivity. Please check your
         firewall settings.
         ......................... abc failed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\abc
      Skipping all tests, because server abc is not responding to directory
      service requests.


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : 123
      Starting test: CheckSDRefDom
         ......................... 123 passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... 123 passed test CrossRefValidation

   Running enterprise tests on : 123.local
      Starting test: LocatorCheck
         Warning: DcGetDcName(TIME_SERVER) call failed, error 1355
         A Time Server could not be located.
         The server holding the PDC role is down.
         Warning: DcGetDcName(KDC_REQUIRED) call failed, error 1355
         A KDC could not be located - All the KDCs are down.
         ......................... 123.local failed test LocatorCheck
      Starting test: Intersite
         ......................... 123.local passed test Intersite

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>dcdiag /test:DNS

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = abc
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\abc
      Starting test: Connectivity
         The host 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local could
         not be resolved to an IP address. Check the DNS server, DHCP, server
         name, etc.
         Got error while checking LDAP and RPC connectivity. Please check your
         firewall settings.
         ......................... abc failed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\abc

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...
         Invalid service startup type: kdc on abc, current value DEMAND_START,
         expected value AUTO_START
         kdc Service is stopped on [abc]
         ......................... abc failed test DNS

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : 123

   Running enterprise tests on : 123.local
      Starting test: DNS
         Test results for domain controllers:

            DC: abc.123.local
            Domain: 123.local


               TEST: Basic (Basc)
                  Error: No LDAP connectivity
                  Error: kdc service is not running
                  Warning: adapter [00000007] Microsoft Hyper-V Network Adapter
                  has invalid DNS server: 101.50 (abc)
                  Warning: adapter [00000007] Microsoft Hyper-V Network Adapter
                  has invalid DNS server: 101.55 (<name unavailable>)
                  Warning: adapter [00000007] Microsoft Hyper-V Network Adapter
                  has invalid DNS server: 127.0.0.1 (abc)
                  Error: all DNS servers are invalid
                  No host records (A or AAAA) were found for this DC
                  Warning: no DNS RPC connectivity (error or non Microsoft DNS s
erver is running)

         Summary of test results for DNS servers used by the above domain
         controllers:

            DNS server: 101.50 (abc)
               2 test failure on this DNS server
               Name resolution is not functional. _ldap._tcp.123.local. failed o
n the DNS server 101.50

            DNS server: 101.55 (<name unavailable>)
               1 test failure on this DNS server
               Name resolution is not functional. _ldap._tcp.123.local. failed o
n the DNS server 101.55

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
            _________________________________________________________________
            Domain: 123.local
               abc                          PASS FAIL n/a  n/a  n/a  n/a  n/a

         ......................... 123.local failed test DNS

C:\Windows\system32>
0
Justin EvansCommented:
you can try this

this looks like when the DC/DNS Server has lost its secure channel with itself or a PDC

> Stop the KDC service on the DC experiencing the issue.
> Run the following command with elevated rights: netdom resetpwd /server:<PDC.domain.com> /userd:<Domain\domain_admin> /passwordd:*
>  It will prompt for the password of the Domain Admin account that you used, enter that.
> Once the command executes, reboot the server.
> DNS zones should load now.
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

nflynn85Author Commented:
So this is a production server and not something I can do on the DC at this time (rebooting a no go during core hours), will have to wait until tonight or tomorrow night. Let me tell you what i'm seeing and ask a few questions....

The KDC on the Primary DC/DNS Server is currently stopped and set to manual (should it be set to automatic, what about starting it before running the command?)

And the command you're asking me to run is just resetting the domain admin password and I can use any domain admin account correct?

Do I actually need to reboot the whole server or are there any service related items I can restart without having to do a full reboot?
0
Justin EvansCommented:
The KDC Service should be started probably why you are getting access denied when trying to open DNS,  what your doing in resetting the domain account password to re-establish the secure channel.  only follow these steps after starting the KDC service and checking if service is resumed.  

Keep me informed on your progress.  

Kind Regards

Justin
0
nflynn85Author Commented:
Sounds good, will update later tonight or tomorrow
0
Justin EvansCommented:
The KDC service should be set to automatic,  can you try and start the service?  If it fails can you look in the event logs and see why and paste the details?
0
DrDave242Commented:
And the command you're asking me to run is just resetting the domain admin password and I can use any domain admin account correct?

Just FYI, that command resets the computer account password, not the password of your domain admin user account. The credentials you supply to the command have nothing to do with the password that's being reset; you're simply supplying credentials of a user account with permission to run the command.
0
nflynn85Author Commented:
The KDC service should be set to automatic,  can you try and start the service?  If it fails can you look in the event logs and see why and paste the details?

I was able to successfully start the service

Just FYI, that command resets the computer account password, not the password of your domain admin user account. The credentials you supply to the command have nothing to do with the password that's being reset; you're simply supplying credentials of a user account with permission to run the command.

Thank you for the clarification
0
Justin EvansCommented:
I believe so yes,  can you restart the DNS Services now,   and see if you can get load DNS Management and see the zone files ?
0
nflynn85Author Commented:
After starting the KDC service, I was able to open DNS Management and see the zone files

"The DNS server has finished the background loading of zones. All zones are now available for DNS updates and zone transfers, as allowed by their individual zone configuration."

I restarted the same services on my secondary DC and now see that zones are loading over there as well

However I am not seeing replication between the two, I tried kicking off replication between the two and saw Event ID Errors 4612 and 5002 jump up in the log
0
nflynn85Author Commented:
Actually I take that back, I'm now seeing replication "The DFS Replication service successfully established an inbound connection with partner for replication group Domain System Volume. "
0
Justin EvansCommented:
ok,  we are nearly there,  can you run DCDIAG /TEST:DNS again  and see if we have full connectivity and all those PASSES at the end?

Then we can start troubleshooting the replication.
0
Justin EvansCommented:
please can you accept this answer as the resolution unless there is anything else? ;)
0
nflynn85Author Commented:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\nflynn>dcdiag /test:dns

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = abc
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\abc
      Starting test: Connectivity
         ......................... abc passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\abc

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...
         ......................... SAS passed test DNS

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : 123

   Running enterprise tests on : 123.local
      Starting test: DNS
         Test results for domain controllers:

            DC: abc.123.local
            Domain: 123.local


               TEST: Records registration (RReg)
                  Network Adapter [00000007] Microsoft Hyper-V Network Adapter:
                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kerberos._tcp.dc._msdcs.123.local

                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kerberos._tcp.123.local

                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kerberos._udp.123.local

                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kpasswd._tcp.123.local

                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.bsr
.local

                     Warning:
                     Missing SRV record at DNS server 101.55:
                     _kerberos._tcp.Default-First-Site-Name._sites.bsr.local

               Error: Record registrations cannot be found for all the network
               adapters

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
            _________________________________________________________________
            Domain: 123.local
               abc                          PASS PASS PASS PASS PASS FAIL n/a

         ......................... 123.local failed test DNS
0
nflynn85Author Commented:
I'm still seeing DFSR Errors 4612 and 5002 pop up in the secondary
0
Justin EvansCommented:
try dcdiag /FIX from  the broken SVR host (sorry I don't know which server this is).    then run netdiag /fix and finally run ipconfig /flushdns ipconfig /registerdns
0
Justin EvansCommented:
I think the broken SVR records are coming from the secondary am I correct?  and how are we looking now after those commands have been run?  DCDIAG /TEST:DNS again please.
0
nflynn85Author Commented:
dcdiag /fix from which server? Primary or secondary?  Also netdiag /fix doesn't work, but restarted netlogon instead if that works

The broken SVR records are from the secondary DC
0
Justin EvansCommented:
from the secondary server that has the SVR errors,    so run ipconfig /flushdns , ipconfig /registerdns restart netlogon service.
0
Justin EvansCommented:
what version of windows is the secondary domain controller running? (with the SVR record errors)
0
Justin EvansCommented:
restart dns on the secondary DC as well.
0
Justin EvansCommented:
where are we at?
0
nflynn85Author Commented:
Did all of that on the secondary, went back to the primary and ran dcdiag /test dns and i'm still seeing the missing SRV records for the secondary DNS server
0
Justin EvansCommented:
what version of windows are you running on the secondary,  the reason why your having those replication errors is that your having connectivity issues.
0
nflynn85Author Commented:
Primary 2008 R2
Secondary 2016

Recall that both of my adapters are showing this, not sure if it's relevant any longer

network connection
0
nflynn85Author Commented:
also, I tried forcing replication from primary to secondary via AD Sites and Services, source server is currently rejecting replication requests
0
MaheshArchitectCommented:
when you first time ran dcdiag, at that moment test warned that KDC service startup is manual and not automatic and as a result KDC service stopped after DC reboot probably
have you made service startup automatic?
after you started service manually, things are started moving
now actually you need to simply restart netlogon service on all DCs one by one so that all Dc records will get reregistered in DNS and then try replication attempt manually from elevated command prompt
repadmin /syncall
repadmin /showrepl
0
Justin EvansCommented:
hopefully after 60 minutes by default the SVR records should replicate,  if not your having a problem with your DNS replication

where you can do the following. clear the cache and update Server Data files.  check the dns logs for errors.
0
nflynn85Author Commented:
Here's the result from the primary DC, I have not ran the repadmin on the secondary

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>repadmin /syncall
CALLBACK MESSAGE: The following replication is in progress:
    From: e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local
    To  : 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local
CALLBACK MESSAGE: The following replication completed successfully:
    From: e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local
    To  : 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.


C:\Windows\system32>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\SAS
DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL
Site Options: (none)
DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
DSA invocationID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

==== INBOUND NEIGHBORS ======================================

DC=bsr,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 12:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        4551 consecutive failure(s).
        Last success @ 2017-12-13 10:15:15.

CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:06:20 was successful.

CN=Schema,CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 12:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1153 consecutive failure(s).
        Last success @ 2017-12-13 09:51:02.

DC=DomainDnsZones,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 12:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1223 consecutive failure(s).
        Last success @ 2017-12-13 09:51:02.

DC=ForestDnsZones,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 12:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1158 consecutive failure(s).
        Last success @ 2017-12-13 10:08:08.

Source: Default-First-Site-Name\dce
******* 4551 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


C:\Windows\system32>
0
Justin EvansCommented:
Welcome to the party Mahesh the problem is the SVR records are missing in DNS on the Secondary Domain Controller,  once this is solved then we can replicate the directory.  I will be back in 30 minutes time.
1
MaheshArchitectCommented:
the servers have stopped replication after 13th December 2017
Today its 1st February 2018, its more than 49 days DCs are not replicated
you need to run repadmin /syncall command from other DCs as well and after that again run "repadmin /shwrepl" and let us know what is last replication time now its showing?
0
nflynn85Author Commented:
The above output is from the primary, this is from the secondary


C:\Windows\system32>repadmin /syncall
CALLBACK MESSAGE: Error contacting server 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local (network error): -2146893022 (0x80090322):
    The target principal name is incorrect.
CALLBACK MESSAGE: Error contacting server e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local (network error): -2146893022 (0x80090322):
    The target principal name is incorrect.

SyncAll exited with fatal Win32 error: 8440 (0x20f8):
    The naming context specified for this replication operation is invalid.

C:\Windows\system32>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\dce
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
DSA invocationID: 8ee8afc9-2272-4418-aa77-cf14ccf40f7b

==== INBOUND NEIGHBORS ======================================

DC=123,DC=local
    Default-First-Site-Name\abc via RPC
        DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
        Last attempt @ 2018-02-01 13:00:30 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        6331 consecutive failure(s).
        Last success @ 2017-12-15 16:59:50.

CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\abc via RPC
        DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
        Last attempt @ 2018-02-01 13:04:01 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        941 consecutive failure(s).
        Last success @ 2017-12-15 16:59:50.

CN=Schema,CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\abc via RPC
        DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
        Last attempt @ 2018-02-01 12:54:06 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        938 consecutive failure(s).
        Last success @ 2017-12-15 16:59:50.

DC=ForestDnsZones,DC=123,DC=local
    Default-First-Site-Name\abc via RPC
        DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
        Last attempt @ 2018-02-01 12:54:06 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        944 consecutive failure(s).
        Last success @ 2017-12-15 16:59:50.

DC=DomainDnsZones,DC=123,DC=local
    Default-First-Site-Name\abc via RPC
        DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
        Last attempt @ 2018-02-01 12:54:06 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        1011 consecutive failure(s).
        Last success @ 2017-12-15 16:59:50.






Source: Default-First-Site-Name\abc
******* 6330 CONSECUTIVE FAILURES since 2017-12-15 16:59:50
Last error: 8456 (0x2108):
            The source server is currently rejecting replication requests.


C:\Windows\system32>
0
MaheshArchitectCommented:
the only valid solution to your problem is to reset domain controller computer account password for every DC one by one

you need to follow steps carefully in below post and reset DC account password  one by one.
https://support.microsoft.com/en-in/help/325850/how-to-use-netdom-exe-to-reset-machine-account-passwords-of-a-windows

This will hopefully resolve replication issue
0
DrDave242Commented:
It appears that you have a broken secure channel ("The target principal name is incorrect" is commonly associated with that), and you may have something else. One of your DCs has inbound and outbound replication disabled (DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL).

Check the Netlogon service on both of your DCs. Is it in a "Paused" state on either one?
0
nflynn85Author Commented:
Netlogon is running on both DCs
0
DrDave242Commented:
Netlogon is running on both DCs

OK, good. There's no USN rollback situation, then. (That's one cause for replication being disabled, and it can be an ugly one.) A broken secure channel won't disable replication like that, so things may have happened the other way around: replication was disabled by something (can't say what at this point), and that in turn broke the secure channel.

Your "primary" DC is the one with replication disabled, according to the output you posted. On that DC, run the following two commands to re-enable it:

repadmin /options <ServerName> -DISABLE_INBOUND_REPL
repadmin /options <ServerName> -DISABLE_OUTBOUND_REPL


(Yes, the switches are correct. This command's syntax is a little counter-intuitive, as shown here.)

The netdom command that was posted some time ago (which is also in the link Mahesh posted above) will reset a particular machine's secure channel. You will probably only have to run that command on the "secondary" DC, but you may have to run it on both.
0
MaheshArchitectCommented:
yes, commands are correct, however I doubt command will successful as command would work if manually replication is disabled by same command
0
Justin EvansCommented:
Are you running IPV6 on the Secondary Domain Controller?    here is something else I have found  

https://support.microsoft.com/en-us/help/321046/how-to-use-dnslint-to-troubleshoot-active-directory-replication-issues

can you download dnslint and then run the following :  

http://download.microsoft.com/download/2/7/2/27252452-e530-4455-846a-dd68fc020e16/dnslint.v204.exe

dnslint /ad 192.168.1.1 /s 192.168.1.2

where 1.1 = Windows 2008 DC
where 1.2 = Windows 2016 DC
0
nflynn85Author Commented:
repadmin /options <ServerName> -DISABLE_INBOUND_REPL
repadmin /options <ServerName> -DISABLE_OUTBOUND_REPL

I did this on the primary

The netdom command that was posted some time ago (which is also in the link Mahesh posted above) will reset a particular machine's secure channel. You will probably only have to run that command on the "secondary" DC, but you may have to run it on both.

I did that on the secondary DC and rebooted, I cannot reboot the primary during core business hours

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\abc
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
DSA invocationID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

==== INBOUND NEIGHBORS ======================================

DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        4552 consecutive failure(s).
        Last success @ 2017-12-13 10:15:15.

CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:09:20 was successful.

CN=Schema,CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1154 consecutive failure(s).
        Last success @ 2017-12-13 09:51:02.

DC=DomainDnsZones,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:10:11 was successful.

DC=ForestDnsZones,DC=123,DC=local
    Default-First-Site-Name\dce via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:10:08 was successful.

Source: Default-First-Site-Name\dce
******* 4552 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


C:\Windows\system32>
0
nflynn85Author Commented:
IPV6 is not enabled on the secondary or primary, it's unchecked on the NIC
0
Justin EvansCommented:
good
0
DrDave242Commented:
OK, it looks like some condition has disabled replication on the primary DC again. I believe lingering objects (objects which have been deleted and garbage-collected on one DC but still exist on another) can cause this. Check the Directory Service event log on the primary DC for errors and feel free to post any that you find.
0
nflynn85Author Commented:
A repadmin /syncall and /showrepl went off with zero errors on the secondary
0
DrDave242Commented:
A repadmin /syncall and /showrepl went off with zero errors on the secondary

Wait...it did? In that case, ignore my previous comment. Does repadmin /showrepl on both DCs report success for all five directory partitions now?
0
Justin EvansCommented:
if you can download dnslint and run  

dnslint /ad 192.168.1.1 /s 192.168.1.2

where 1.1 = Windows 2008 DC
where 1.2 = Windows 2016 DC

paste the results and maybe the error might bring us the answer to this problem.
0
nflynn85Author Commented:
I'm seeing 100% success on the secondary, but the primary is still failing with what I posted above

DNSLint Report

System Date: Thu Feb 01 14:25:03 2018

Command run:

 Root of Active Directory Forest:

    123.local
Active Directory Forest Replication GUIDs Found:
 
DC: abc
GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

DC: cde
GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136


Total GUIDs found: 2


The following 3 DNS servers were checked for records related to AD forest replication:

DNS server: User Specified DNS Server
IP Address:
 UDP port 53 responding to queries: YES
TCP port 53 responding to queries: Not tested
Answering authoritatively for domain: Unknown

SOA record data from server:
 Authoritative name server: dce.123.local
Hostmaster: hostmaster.123.local
Zone serial number: 3315
Zone expires in: 1.00 day(s)
Refresh period: 900 seconds
Retry delay: 600 seconds
Default (minimum) TTL: 3600 seconds


Additional authoritative (NS) records from server:
 dce.123.local Unknown
 abc.123.local Unknown




Alias (CNAME) and glue (A) records for forest GUIDs from server:
 CNAME: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local
Alias: abc.123.local
Glue:

CNAME: e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local
Alias: cde.123.local
Glue:


Total number of CNAME records found on this server: 2

Total number of CNAME records missing on this server: 0

Total number of glue (A) records this server could not find: 0




DNS server: cde.123.local
IP Address:
 UDP port 53 responding to queries: YES
TCP port 53 responding to queries: Not tested
Answering authoritatively for domain: YES

SOA record data from server:
 Authoritative name server: dce.123.local
Hostmaster: hostmaster.123.local
Zone serial number: 3315
Zone expires in: 1.00 day(s)
Refresh period: 900 seconds
Retry delay: 600 seconds
Default (minimum) TTL: 3600 seconds


Additional authoritative (NS) records from server:
 abc.123.local Unknown
 dce.123.local Unknown




Alias (CNAME) and glue (A) records for forest GUIDs from server:
 CNAME: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local
Alias: abc.123.local
Glue:

CNAME: e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local
Alias: dce.123.local
Glue:


Total number of CNAME records found on this server: 2

Total number of CNAME records missing on this server: 0

Total number of glue (A) records this server could not find: 0




DNS server: abc.123.local
IP Address:
 UDP port 53 responding to queries: YES
TCP port 53 responding to queries: Not tested
Answering authoritatively for domain: YES

SOA record data from server:
 Authoritative name server: abc.123.local
Hostmaster: hostmaster.123.local
Zone serial number: 3315
Zone expires in: 1.00 day(s)
Refresh period: 900 seconds
Retry delay: 600 seconds
Default (minimum) TTL: 3600 seconds


Additional authoritative (NS) records from server:
 dce.123.local Unknown
 abc.123.local Unknown




Alias (CNAME) and glue (A) records for forest GUIDs from server:
 CNAME: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8._msdcs.123.local
Alias: abc.123.local
Glue:

CNAME: e3ccb41a-74f8-448b-bc56-a7862c1ff136._msdcs.123.local
Alias: dce.123.local
Glue:


Total number of CNAME records found on this server: 2

Total number of CNAME records missing on this server: 0

Total number of glue (A) records this server could not find: 0




Notes:
One or more DNS servers may not be authoritative for the domain




Legend: warning, error

DNSLint developed by Tim Rains
0
DrDave242Commented:
I'm seeing 100% success on the secondary, but the primary is still failing with what I posted above

Sorry, there's quite a lot posted above. Which specific error are you seeing on the primary now? You don't have to post the complete repadmin results, just the error, unless you want to post it all.
0
nflynn85Author Commented:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\SAS
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
DSA invocationID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

==== INBOUND NEIGHBORS ======================================

DC=bsr,DC=local
    Default-First-Site-Name\DC2BSR via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        4552 consecutive failure(s).
        Last success @ 2017-12-13 10:15:15.

CN=Configuration,DC=bsr,DC=local
    Default-First-Site-Name\DC2BSR via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:09:20 was successful.

CN=Schema,CN=Configuration,DC=bsr,DC=local
    Default-First-Site-Name\DC2BSR via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1154 consecutive failure(s).
        Last success @ 2017-12-13 09:51:02.

DC=DomainDnsZones,DC=bsr,DC=local
    Default-First-Site-Name\DC2BSR via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:19:06 was successful.

DC=ForestDnsZones,DC=bsr,DC=local
    Default-First-Site-Name\DC2BSR via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:19:03 was successful.

Source: Default-First-Site-Name\DC2BSR
******* 4552 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


C:\Windows\system32>
0
Justin EvansCommented:
ok,  can you paste the results of this command from the primary running Windows 2008

dnslint /ad /s localhost
0
nflynn85Author Commented:
the output was exactly the same as the previous out I provided to you
0
DrDave242Commented:
Well, that's a little odd. The domain partition and schema partition are getting "The destination server is currently rejecting replication requests," but the other three partitions are replicating successfully.

Ah, hold on a sec. The two failed partitions show that the last attempt was at 13:58:36, while the three successful partitions show more recent timestamps. This may clear up on its own, but if you want to speed it up, try forcing replication from the secondary to the primary. (Make sure you do it in that direction.)
0
Justin EvansCommented:
fine,  I am trying to find an error for the SVC record which I can search apon,  what I would like to do at this point is give the primary a reboot. when can we do that?
0
nflynn85Author Commented:
It's currently 3pm EST - I will not be able to reboot for several hours so later this evening at some point

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\abc
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
DSA invocationID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

==== INBOUND NEIGHBORS ======================================

DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        4552 consecutive failure(s).
        Last success @ 2017-12-13 10:15:15.

CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:09:20 was successful.

CN=Schema,CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 13:58:36 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        1154 consecutive failure(s).
        Last success @ 2017-12-13 09:51:02.

DC=DomainDnsZones,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:19:06 was successful.

DC=ForestDnsZones,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:19:03 was successful.

Source: Default-First-Site-Name\dce
******* 4552 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


C:\Windows\system32>
0
nflynn85Author Commented:
I tried forcing the connection above in ADSS but no luck clearly
0
nflynn85Author Commented:
Sorry, that was incorrect, this is the current state of repadmin /showrepl on the primary

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\abc
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8
DSA invocationID: 2feb9c4c-53f5-4d75-8c17-624714bfa2c8

==== INBOUND NEIGHBORS ======================================

DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:54:58 was successful.

CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:54:58 was successful.

CN=Schema,CN=Configuration,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:54:58 was successful.

DC=DomainDnsZones,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:54:58 was successful.

DC=ForestDnsZones,DC=123,DC=local
    Default-First-Site-Name\cde via RPC
        DSA object GUID: e3ccb41a-74f8-448b-bc56-a7862c1ff136
        Last attempt @ 2018-02-01 14:54:58 was successful.

Source: Default-First-Site-Name\cde
******* 4552 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


C:\Windows\system32>
0
DrDave242Commented:
That's 100% success. (The error at the bottom simply references the last error message, so it can be ignored.)
0
nflynn85Author Commented:
WOOP - Thanks guys!
0
nflynn85Author Commented:
Going to do a reboot off hours just to make myself feel better
0
Justin EvansCommented:
what is the windows 2008 domain functional level ?
0
nflynn85Author Commented:
the domain level and forest level are both 2008 on the 2008 DC
0
Justin EvansCommented:
I am looking at this article now

https://support.microsoft.com/en-gb/help/2023007/troubleshooting-ad-replication-error-8456-or-8457-the-source-destinati

Can you give the values on the 2008 DC registry of

HKLM\System\CurrentControlSet\Services\NTDS\Parameters

Setting

DSA not writable

Type

 (Reg_dword)

save the reg file and paste.
0
nflynn85Author Commented:
Why would I need to do that? Everything is replicating - all 5 show success, we've already established the last error is the prior error message
0
Justin EvansCommented:
sorry I missed that,  so if everything is replicating what is the error now,   is it the missing SVR records still?  

Source: Default-First-Site-Name\cde
******* 4552 CONSECUTIVE FAILURES since 2017-12-13 10:15:15
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.

just to confirm have you only two domain controllers in your environment,  problem is when you blank out your server information we lose the ability to see the differences between the server,  so when you paste the result could you type in server1 for the 2008 DC and server2 for the secondary 2012 DC.  this would help a lot.
0
nflynn85Author Commented:
I think we're done here - Primary DC shows passing dcdiag /test:dns
0
Justin EvansCommented:
brilliant!  Its all fixed then?  as I said it took 60 minutes for the replication to correct the missing SVR records,  your replication should kick in.  if not write up another case and we will look at that separately.  

please accept the solution.  Enjoyed working with you.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
nflynn85Author Commented:
Will do that first thing in the morning, thanks to you and Dr. Dave for the help
0
nflynn85Author Commented:
I don't think we are out of the woods just yet

I shut down the secondary, the client server, and the primary DC in that order

I then brought them up in reverse order (primary, client, and secondary)

The Primary DC came up, but netlogon was became paused and NIC card is showing internal network only (private) and isn't connected to outside internet. It is not showing as being on 123.local

The secondary shows as being on the domain and connected to external internet

The client server shows as internal network (private) and connected to external internet

DNS tests are passing with flying colors and replication is still successful

What gives? Am I supposed to start secondary, then primary, then client?
0
nflynn85Author Commented:
Nevermind, I got it,

Had to reset the machine account with netdom on the primary
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking

From novice to tech pro — start learning today.