Link to home
Start Free TrialLog in
Avatar of ITPIP
ITPIP

asked on

Active Directory FRS error 13508 in FRS Event Log

Well I have been working on this problem for awhile now and I just can't seem to figure it out.

Here is what is going on:

A couple of weeks ago as I was doing some research into replacing some of my aging DCs I came across an error that I can't seem to remedy.  I was running DCDIAG just to check and make sure there wasn't any errors and I found one.  All tests would pass except the Netlogons test and the FRSEvent test when running DCDIAG.

I get the following output for this test:
Starting test: NetLogons
         * Network Logons Privileges Check
         Unable to connect to the NETLOGON share! (\\SERVER\netlogon)
         [SERVER] An net use or LsaPolicy operation failed with error 1203, No network provider accepted the given network path..
         ......................... SERVER failed test NetLogons


I also fail the FRSEVENT test with the error:
Starting test: frsevent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 08/23/2010   22:49:31
            (Event String could not be retrieved)
         ......................... SERVER failed test frsevent


So I did some research and was directed towards the FRS Event Logs.  Once I looked in there I saw the following error:
The File Replication Service is having trouble enabling replication from <server> to <server> for <path> using the DNS name <name>. FRS will keep retrying.
Following are some of the reasons you would see this warning.

[1] FRS can not correctly resolve the DNS name <name> from this computer.
[2] FRS is not running on <name>.
[3] The topology information in the Active Directory for this replica has not yet replicated to all the Domain Controllers.

This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.


I found the following ideas to fix this:
https://www.experts-exchange.com/questions/21740439/FILE-REPLICATION-SERVICE-is-having-trouble-Event-ID-13508.html

I have tried the steps outlined towards the bottom of that thread from malboteju and iistech.  I tried changing that value to D2 and D4 and each time for just a couple minutes I almost thought that it worked.  I recieved the following event in my FRS log immediately after restarting the FRS service:
The File Replication Service is no longer preventing the computer <SERVER> from becoming a domain controller.  The system volume has been successfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL.

Type "net share" to check for the SYSVOL share.


Then about 2 minutes later that 13508 error comes back.

Looking at the Shares in Computer Management I only see SYSVOL and not NETLOGON.

So obviously something still isn't working with FRS but I haven't been able to find what it is.

I have this same problem on my other DC as well.  So 2 of my 3 DCs have this exact same problem and I have followed the same troubleshooting steps for both.

The FRS log for the 1 "working" server doesn't have any errors listed and has both a SYSVOL and NETLOGON share listed.

I also ran FRSDiag and it came back with the following:
------------------------------------------------------------
FRSDiag v1.7 on 8/23/2010 10:38:11 AM
.\SERVER on 2010-08-23 at 10.38.11 AM
------------------------------------------------------------

Checking for errors/warnings in FRS Event Log .... &#9;
NtFrs&#9;8/22/2010 9:45:03 PM&#9;Warning&#9;13508&#9;The File Replication Service is having trouble enabling replication  from SERVER to SERVER for c:\windows\sysvol\domain using the DNS name SERVER.DOMAIN.com. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name SERVER.DOMAIN.com from this computer.     [2] FRS is not running on SERVER.DOMAIN.com.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.&#9;
NtFrs&#9;8/21/2010 6:39:37 PM&#9;Warning&#9;13508&#9;The File Replication Service is having trouble enabling replication  from SERVER to SERVER for c:\windows\sysvol\domain using the DNS name SERVER.DOMAIN.com. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name SERVER.DOMAIN.com from this computer.     [2] FRS is not running on SERVER.DOMAIN.com.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.&#9;
NtFrs&#9;8/20/2010 5:22:41 PM&#9;Warning&#9;13508&#9;The File Replication Service is having trouble enabling replication  from SERVER to SERVER for c:\windows\sysvol\domain using the DNS name SERVER.DOMAIN.com. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name SERVER.DOMAIN.com from this computer.     [2] FRS is not running on SERVER.DOMAIN.com.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.
&#9;WARNING: Found Event ID 13508 errors without trailing 13509 ... see above for (up to) the 3 latest entries!

 ......... failed 1
Checking for errors in Directory Service Event Log .... passed
Checking for minimum FRS version requirement ... passed
Checking for errors/warnings in ntfrsutl ds ... passed
Checking for Replica Set configuration triggers... passed
Checking for suspicious file Backlog size... passed
Checking Overall Disk Space and SYSVOL structure (note: integrity is not checked)... passed
Checking for suspicious inlog entries ... passed
Checking for suspicious outlog entries ... passed
Checking for appropriate staging area size ... passed
Checking for errors in debug logs ...
&#9;ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     5244:   904: S0: 10:31:27> :SR: Cmd 01744da0, CxtG 6f08502c, WS ERROR_ACCESS_DENIED, To   SERVER.DOMAIN.com Len:  (362) [SndFail - Send Penalty]
&#9;ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     5244:   877: S0: 10:36:27> :SR: Cmd 017484b8, CxtG 6f08502c, WS ERROR_ACCESS_DENIED, To   SERVER.DOMAIN.com Len:  (362) [SndFail - rpc call]
&#9;ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     5244:   904: S0: 10:36:27> :SR: Cmd 017484b8, CxtG 6f08502c, WS ERROR_ACCESS_DENIED, To   SERVER.DOMAIN.com Len:  (362) [SndFail - Send Penalty]
&#9;ERROR on NtFrs_0003.log : "RPC_S_CALL_FAILED_DNE(Indicates RPC Session was established to target, but there was a failure to send RPC call package. Check for Networking problems!)" : <SndCsMain:                     5512:   884: S0: 16:28:20> :SR: Cmd 0143ff68, CxtG 075b2240, WS RPC_S_CALL_FAILED_DNE, To   dc-1.DOMAIN.com Len:  (362) [SndFail - rpc exception]
&#9;ERROR on NtFrs_0003.log : "RPC_S_CALL_FAILED_DNE(Indicates RPC Session was established to target, but there was a failure to send RPC call package. Check for Networking problems!)" : <SndCsMain:                     5512:   883: S0: 16:28:36> ++ ERROR - EXCEPTION (000006bf) :  WStatus: RPC_S_CALL_FAILED_DNE
&#9;ERROR on NtFrs_0003.log : "RPC_S_CALL_FAILED_DNE(Indicates RPC Session was established to target, but there was a failure to send RPC call package. Check for Networking problems!)" : <SndCsMain:                     5512:   884: S0: 16:28:36> :SR: Cmd 01454a00, CxtG 075b2240, WS RPC_S_CALL_FAILED_DNE, To   dc-1.DOMAIN.com Len:  (462) [SndFail - rpc exception]

&#9;Found 7906 ERROR_ACCESS_DENIED error(s)! Latest ones (up to 3) listed above
&#9;Found 4 RPC_S_CALL_FAILED_DNE error(s)! Latest ones (up to 3) listed above

 ......... failed with 7910 error entries
Checking NtFrs Service (and dependent services) state...
&#9;ERROR : Cannot access NETLOGON share on SERVER
 ......... failed 1
Checking NtFrs related Registry Keys for possible problems...passed
Checking Repadmin Showreps for errors...passed


I also tried something regarding a JOURNAL WRAP registry key but FRSDIAG came back and said that shouldn't be used anymore.

The ERROR_ACCESS_DENIED error from the FRSDIAG leads me to believe that this could be a permissions issue either on a share or directory but if it is I am not sure what permissions I am missing.

I inherited this domain and I have not added any new domain controllers or removed any old ones since I have been working with it, not sure if that information is needed but it might answer some questions before someone asks them.

Also, regarding the D2 and D4 registry entries.  Should I be doing this on the working server too?  I only tried changing those keys on the servers that have the FRS errors in their event logs but I haven't tried it on the "working server".  From the information I have read so far it doesn't seem like I would need to but I just want to make sure.

Thanks in advance for your help.  Let me know if you need more information and I will try and provide as much of it as I can.
Avatar of ChiefIT
ChiefIT
Flag of United States of America image

go to the faild DC and type:

DCdiag /test:DNS

Please provide this data

You should *not* set the burflags to D4 on the DC that is missing the Netlogon share. You'll then say that this DC holds the authoritative SYSVOL even if it's missing the Netlogon.
Avatar of ITPIP
ITPIP

ASKER

Attached is the output from the DCDIAG /test:DNS.  Obviously there are some failures there.  10.10.0.11 is the address of my other secondary DNS server.  That server however is not a DC.  From previous reading I understand that a DNS server that is not a DC will not replicate the primary DNS servers records.  This hasn't shown itself to be a problem in the day to day but could this be an underlying cause of some of these issues?

snusgubben:
Should I set the burflags on the "working" server to D4?  Will this tell the other DCs to look at that server for the autoritative SYSVOL?

Thanks,
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\SERVER
      Starting test: Connectivity
         ......................... SERVER passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\SERVER

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: SERVER.domain.com
            Domain: domain.com

                  
               TEST: Forwarders/Root hints (Forw)
                  Error: Forwarders list has invalid forwarder: 24.149.0.24 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.6 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.7 (<name unavailable>)
                  Error: Root hints list has invalid root hint server: a.root-servers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-servers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-servers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-servers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-servers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-servers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-servers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-servers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-servers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-servers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-servers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (198.32.64.12)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-servers.net. (202.12.27.33)
                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
                  
               TEST: Records registration (RReg)
                  Network Adapter [00000007] Broadcom BCM5708C NetXtreme II GigE (NDIS VBD Client):
                     Error: Missing CNAME record at DNS server 10.10.0.11 :
                     c94a0f86-3f87-4ca7-af40-2faca3b85381._msdcs.domain.com
                     
                     Warning: Missing DC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.dc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

                     Warning: Missing GC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.gc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

                     Warning: Missing PDC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.pdc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

                  Network Adapter [00000008] Broadcom BCM5708C NetXtreme II GigE (NDIS VBD Client):
                     Error: Missing CNAME record at DNS server 10.10.0.11 :
                     c94a0f86-3f87-4ca7-af40-2faca3b85381._msdcs.domain.com
                     
                     Warning: Missing DC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.dc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

                     Warning: Missing GC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.gc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

                     Warning: Missing PDC SRV record at DNS server 10.10.0.11 :
                     _ldap._tcp.pdc._msdcs.domain.com
                     (Ignore the error if DNSAvoidRegisterRecord registry key or its Group Policy
                     has been configured to prevent registration of this Record.)

               Error: Record registrations cannot be found for all the network adapters
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 24.149.0.7 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.7
               
            DNS server: 24.149.0.6 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.6
               
            DNS server: 24.149.0.24 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.24
               
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33
               
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42
               
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4
               
            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.32.64.12
               
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129
               
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30
               
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241
               
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17
               
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12
               
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201
               
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10
               
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4
               
            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.8.10.90
               
            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               SERVER                      PASS PASS FAIL PASS WARN FAIL n/a  
         
         ......................... domain.com failed test DNS

Open in new window

You should not set the D4 flag at all since you have DNS problems. The D4 flag will not say to the other DC's come and get my SYSVOL content. If you add a new DC it will get the content from this DC.

The D4 flag should be used on the DC that holds a healthy SYSVOL, and the D2 flag should be set on the DC(s) with a broken SYSVOL. This should only be used as a last resort if your SYSVOL is burned and you want to bulk reset your SYSVOL in the hole domain. You don't want that in your case.

I would set all DC's to point to your DNS that is held by a DC and remove the "none-DC-DNS" from the NIC (and maybe from the name server tab on your DNS).

Run "ipconfig /flushdns", "ipconfig /registerdns" and restart the netlogon service.

Then re-run the command ChiefIT provided.
Avatar of ITPIP

ASKER

Snusgubben:

Just completed those tasks.  I posted the new results of the DCDIAG /TEST:DNS.

Thanks.

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\SERVER
      Starting test: Connectivity
         ......................... SERVER passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\SERVER

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: SERVER.domain.com
            Domain: domain.com

                  
               TEST: Forwarders/Root hints (Forw)
                  Error: Forwarders list has invalid forwarder: 24.149.0.24 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.6 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.7 (<name unavailable>)
                  Error: Root hints list has invalid root hint server: a.root-servers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-servers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-servers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-servers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-servers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-servers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-servers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-servers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-servers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-servers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-servers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (198.32.64.12)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-servers.net. (202.12.27.33)
                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 24.149.0.7 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.7
               
            DNS server: 24.149.0.6 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.6
               
            DNS server: 24.149.0.24 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.24
               
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33
               
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42
               
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4
               
            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.32.64.12
               
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129
               
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30
               
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241
               
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17
               
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12
               
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201
               
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10
               
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4
               
            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.8.10.90
               
            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               SERVER                      PASS PASS FAIL PASS WARN PASS n/a  
         
         ......................... domain.com failed test DNS

Open in new window

Please give us an overview of your servers:

Which ones are AD servers (meaning domain controllers)?
Which onese are DNS servers?
Which ones are member servers and what functions do they perform?
Avatar of ITPIP

ASKER

I have three DCs:

We will call them:
DC 1
DC 2
DC 3


DC 1 doesn't have the FRS errors and holds the roles: Schema Owner, Domain Owner

DC 2 has the errors but doesn't hold any roles that I can see

DC 3 has the errors and is the server I have been running the DCDIAG tests on.  It is a DNS and DHCP Server.  It holds the roles: PDC Owner, RID Owner, and Infrastructure Update Owner


Thats it for the DCs.

I have a quite a few other servers that are not DCs.  One of them, previously mentioned to hold the address 10.10.0.11 is a secondary DNS server.  The rest are various web, email, SQL, and other application servers.

I think DC 1 used to be a DNS server but the service is not running on that server anymore.  If you need more info let me know but that seems like it would be the important stuff.


Thanks.
I am confused. Only one DC should hold the Flexible SINGLE Master Operations FSMO Rolls. It appears like you have DC1 and DC3 holding the roles. Is DC1 and DC3 on two separate domains? If not, you have two domains on your network with the same domain name, (IF AND ONLY IF you have two dcs with rolls)

Please correct the confusion by telling me what's up with DC1 and DC3, both having FSMO roles, and we will come up with a recovery plan for you.
Avatar of ITPIP

ASKER

Hmmm..

Well the only reason I listed those FSMO roles was because of the output of DCDIAG run from DC3.  I attached that below.  To me that doesn't appear as though they are on two seperate domains.  Also if they were on two seperate domains wouldn't I only see one or two DCs in the AD Sites and Services Snap in under Sites->Default-First-Site-Name->Servers?  Right now all three servers are listed in this snap-in.  Wouldn't I be able to move these roles to which ever server I wanted anyways?

Also, to explain why two hold FSMO roles, it could have been because of an issue I had quite a few months back where I came into the office and DC1 was down.  This caused quite a few issues on the domain.  Hardware was to blame and as soon as I replaced the hardware and booted that DC up all issues went away.  In my cleanup effort and debrief of that failure I had convinced myself that I needed to transfer important roles off of DC1 as that was the oldest server and therefore more prone to failure.  I wanted to eventually phase that DC out but I haven't had a chance to do that.  So I probably moved those roles myself and that is why there are two different servers holding FSMO roles.  There was some rules that I had researched about which DCs should hold which roles but I will need more time to dig that information up.

Thanks.
Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         ......................... DC3 passed test KnowsOfRoleHolders

Open in new window

Well there is nothing wrong with splitting up the FSMO roles as long as they are  consistant between them.

The main DNS problem that is causing problems with your File replications is the delegation records appear to be outdated. This happened to me as well, and I would like to introduce you to what I did.

Now, if you follow Dariusq and Chris Dent's advice by deleting the Delegation and SRV records and re-registering them on all servers, you must do so by going to each Server and deleting both MSDCS file folders and after that, you must reregister them on each server by going to the command prompt and typing:

IPconfig /flushdns
IPconfig /registerdns
Net Stop Netlogon
Net Start Netlogon.

As a safety precaustion add in this additional command line:
DCdiag /fix:DNS

Here is an exampe of DNS delegation records expired:
https://www.experts-exchange.com/questions/24349599/URGENT-MSDCS-records-registering-directly-under-FWD-lookup-zone-not-under-FQDN-name-space.html
__________________________________

Once done fixing your SRV records get back with me and we should fix your Root hint servers,,, or forwarders to make sure DNS resolution to the outside world goes smoothly.

I also see some reverse lookups that are incorrect. They also appear to be root hint servers. I am thinking it best to use the ISPs servers as forwarders.....

___________________________________________________

Furthermore, after fixing DNS, you will want to reset your replication set. You can attempt to do this by a force replication, restarting FRS, and then doing the Burflag method. The authoritative Burflag is used for the PDCe (the role holder). In your case the FSMO roles are split between two DCs. This isn't the standard setup, but we can work with it. In your case the authoritative Burflag will be the PDC Owner, (meaning DC3)..

I would also make sure that all three servers are GC servers.

RECOMMENDED:
I might also be so bold to recommend you take all FSMO roles onto DC1 and then fix your replication set.

BOTTOM LINE:
-FIX DNS DELEGATION RECORDS PROBLEMS
-FIX FORWARD/ROOT HINTS LOOKUPS
-RESET YOUR REPLICATION SET BY:
----1) force replication
---2) reset the FRS service
---3) Burflag method (authoritative on the PDCe, non authoritative otherwise)

Avatar of ITPIP

ASKER

Chief,

Sorry for not getting back to you till now.  I was just looking at the first step of deleting the _mcds folder on my DNS server.  You mentioned I should do this on both servers but only one is AD integrated.  The other is not.  So I can only do this on the AD integrated DNS server, correct?  If I am not right about that then which other server should I do this on?

Just want to clarify before I start deleting.

Thanks.
All DNS replicated partners should hold the SRV records within DNS. We have to fix these SRV records. Your problem right now is your delegation record is expired and that is causing your DNS server not to see the SeRVice records. This stopps replications between domain controllers. We will have to fix DNS on each DC that has problems. You can monitor progress using DCdiag /test:DNS on each DNS server. Once the SRV records are changed on a Global catalog server with DNS, then you should be able to replicate the changes out to all other DNS servers.
Avatar of ITPIP

ASKER

Just deleted _msdcs and ran the commands you listed except when I try running dcdiag /fix:DNS I get an error: Invalid Syntax.  Not sure which command you wanted me to run in place of that.  It looks like it recreated the _msdcs folder on my AD integrated DNS server.  This is the only AD integrated DNS server I have on my domain right now.  The other is a non AD integrated DNS server and isn't really being used right now.

So now that that is done, where do we go from here?

Thanks.
Let's confirm the SRV records exist:

DCdiag /test:DNS
Once done, we will need to reset replication between the replicating partners.

I like to try least invasive approach first.There are three methods that can reset replication.

1) Go in and try to force replicate.

How, you ask:  http://www.windowsnetworking.com/kbase/WindowsTips/Windows2003/AdminTips/ActiveDirectory/ForcingActiveDirectoryReplication.html

2) Rest the FRS service.
How:
Navigate to START>>RUN>>and type on the run line Services.msc.
Restart the file replication service on both DCs starting with the PDCe (holder of FSMO roles).

3) Use the burflag method to reset replications. On the PDCe use the authoritative reset. On any other DC replicating partner you want to use the non-authoritative reset. So, on the DC with errors, you want to start with that DC and use the proper burflag method to reset replication.

How to: (USE THIS INFORMATION VERY EXPLICITLY!
http://msmvps.com/blogs/bradley/archive/2009/11/27/burflags-and-journal-wrap.aspx
Avatar of ITPIP

ASKER

In the previous post I had said that DCDIAG /fix:DNS gave me an error of Invalid Syntax.  Is that the correct command?
I can't remember if that is a pipe or a colon:

DCdiag /test|DNS
or
DCdiag /test:DNS

DCdiag is a part of the 2003 server tools kit, that can be downloaded from the network, or you can find it on the installation CDs. It is a highly suggested tool kit for domain amdinistration.

http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

Another ensurance to make sure the SRV records are registered is to follow these command lines:
IPconfig /flushdns
IPconfig /registerdns
Net Stop Netlogon
Net Start Netlogon
Avatar of ITPIP

ASKER

It is a pipe.

Ok so I tried to force the replication according to the three ways you had outlined.  I am getting the same error.  When doing the burflag method I did it in this order.

DC3 which is the PDCe was set with burflag D4
DC1 was set with burflag D2
DC2 was set with burflag D2

Unfortunately when looking in the event logs for FRS I initially see the following information:

The FRS successfully added this computer to the following replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"

Information related to this event is shown below:
Computer DNS name is "DC3.domain.com"
Replica set member name is "DC3"
Replica set root path is "c:\windows\sysvol\domain"
Replica staging directory path is "c:\windows\sysvol\staging\domain"
Replica working directory path is "c:\windows\ntfrs\jet"


Then I see the next event as the following:



The FRS successfully added the connections shown below to the replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)

"DC1.domain.com"
"DC1.domain.com"
"DC2.domain.com"
"DC2.domain.com"


Then the next event is:


The FRS is no longer preventing the computer DC3 from becoming a domain controller.  The system volume has been succesfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL.

Type "net share" to check for the SYSVOL share.

Then after that I get the error:

The File Replication Service is having trouble enabling replication from DC2 to DC3 for c:\windows\sysvol\domain using the DNS name DC2.domain.com. FRS will keep retrying.
Following are some of the reasons you would see this warning.

[1] FRS can not correctly resolve the DNS name DC2.domain.com from this computer.
[2] FRS is not running on DC2.domain.com.
[3] The topology information in the Active Directory for this replica has not yet replicated to all the Domain Controllers.

This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.

Then I see another error message:

This one basically is the same is the on above only it references DC1 and DC3 instead of DC2 and DC3.


So evidently I am still coming out with the same error in my FRS logs after reseting the SRV records on my DNS server and then setting the burflags and restarting FRS.

Any ideas?
So, DC2 needs to register its SRV records in DNS. Make sure DC2 and DC3 point to the DNS server as its primary DNS server in the NIC configuraion. Then, go to each and use these four lines to register the other DC's SRV records within DNS

IPconfig /flushdns
IPconfig /registerdns
Net Stop Netlogon
Net Start Netlogon

Once they are registered within DNS, restart the FRS service or try to force replicate...
Avatar of ITPIP

ASKER

Ok.  I completed the tasks you had outlined in the last post and that did fix the issue of FRS replication from DC2 to DC3.  The DC1 to DC3 FRS 13508 event did come back though.  This error also exists on DC2.  It has an event 13508 in the FRS logs that references that replication from DC1 to DC2 is not working.

So the commanlity between DC2 and DC3's event logs is that they both cannot replicate from DC1.

What am I missing?

Thanks again for your help.
DCdiag is clear for DNS related errors.

If so, reset the FRS service on DC2 and DC3

If not, we must fix DNS first.

If resetting the FRS service doesn't work, we will have to use the Burflag method to fix FRS. Since DC2 and DC3 are not forest servers, you can use the NON-authoritative restore of the Sysvol and Netlogon share. That will reset your replication set.

How to: (USE THIS INFORMATION VERY EXPLICITLY!)
http://msmvps.com/blogs/bradley/archive/2009/11/27/burflags-and-journal-wrap.aspx
Avatar of ITPIP

ASKER

DNS errors still exist.  I have posted the results of DCDIAG /test:DNS ran from DC3 below.

You had mentioned we needed to fix the forwarders errors before but we hadn't covered how to do that.

I don't quite understand why there is a problem with the 3 forwarders listed at the top of these lists.  Those 3 are DNS servers provided by the ISP.

Any ideas?

Thanks.
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC3
      Starting test: Connectivity
         ......................... DC3 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC3

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: DC3.domain.com
            Domain: domain.com

                  
               TEST: Forwarders/Root hints (Forw)
                  Error: Forwarders list has invalid forwarder: 24.149.0.24 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.6 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.7 (<name unavailable>)
                  Error: Root hints list has invalid root hint server: a.root-servers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-servers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-servers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-servers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-servers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-servers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-servers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-servers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-servers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-servers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-servers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (198.32.64.12)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-servers.net. (202.12.27.33)
                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 24.149.0.7 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.7
               
            DNS server: 24.149.0.6 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.6
               
            DNS server: 24.149.0.24 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.24
               
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33
               
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42
               
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4
               
            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.32.64.12
               
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129
               
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30
               
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241
               
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17
               
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12
               
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201
               
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10
               
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4
               
            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.8.10.90
               
            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC3                      PASS PASS FAIL PASS WARN PASS n/a  
         
         ......................... domain.com failed test DNS

Open in new window

The forwarders are probably not bad. You are just using your loopback address as a legitimate address to respond to forward lookups. So, it will appear to be a problem in the DCdiag reports as the computer report will not be able to determine what 127.0.0.1 means. The router, will. It is advisable to go to each DC, and make sure you are not using the loopback address as a preferred or alternate DNS server...

The way I configure it is:
DC1:
Preferred: (DC1's IP address, (not the loopback address))
Alternate: DC2
DC2:
Preferred: DC2's IP address
Alternate: (DC1's IP address)
DC3:
Preferred: DC3's IP address
Alternate: DC1's IP address

With that changed, then go into the reverse lookup zone and eliminate all records to 127.0.0.1 in address arpa..
It will look like: 1.0.0.127..

Also elimiate all HOST A records in the forward lookup zone, and any SRV records or SAME as Host records that pertain to 127.0.0.1's IP..

Now, make sure on the DNS forwarders tab that recursion is enabled and the forwarder's IP addresses are correct, (as given by the ISP). Enabling recursion will allow Forward lookups and prevent from root hint server lookups.

Do this on all three DCs and your DNS errors should disappear.

NOTE: These forwarder or Root hint problems should not prevent from File replication services if these DCs are on the same broadcast domain, or have a PPP or dedicated line connection between them.
Avatar of ITPIP

ASKER

Ok, so since these are on the same broadcast domain I can ignore these forwarder errors for now and move onto trying the burflag method.

After doing the non-authoritative burflag method on DC2 and DC3 I now only see one 13508 error event in DC2 and DC3's FRS logs.  They both reference replication issues from DC1 to DC3.

So now I only see DC2 reporting on DC3s issue replicating from DC1 and not its own replication from DC1.

Any ideas?
This kind of depends upon the SRV records registration to DC3's SRV records. On the NIC configuration, what DNS server do you have as it's primary, and what do you have as its alternate?

It's my guess that DC3 still has not registered it's SRV records with DC1 and/or itself.

I am assuming all three DCs are global catalogs and all three are DNS servers.

Avatar of ITPIP

ASKER

I have DC3 listed as the primary DNS server for all 3 servers.  I have not set an alternate at this time.  Although I hadn't utilized them as alternative DNS servers, DC1 and DC2 do have DNS server running on them.

I noticed that when I followed the instructions on deleting the _MSDCS folder and re-registering on DC1 I don't see the SRV records for DC2 or DC3.  Those records never came back like they did when I followed this same process on DC2 and DC3.  They were there before I deleted that folder but they won't come back.

Also in answer to the other part of your question, yes all three servers are GC servers.

So I am thinking that whatever the reason that DC1 will not reregister the other two servers in it's MSDCS folder, is related to why I am getting the FRS errors regarding replication from DC1 to DC3.

So I have checked that all servers are pointing to the DC3 as their primary DNS server, I have tried to reregister all the SRV records and all servers but DC1 were successful in doing this and I have been able to resolve FRS from DC1 to DC2 but I am still having problems with FRS errors from DC1 to DC3.  Where do we go from here?

Thanks.
On DC1, go to the nic configuration and set it's primary DNS server as itself, with DC3 as the alternate.

Delete the MSDCS file folders and SRV records on DC1.

Now Go to the command prompt of DC1 and type:
Ipconfig /flushdns
IPconfig /registerdns
Net stop netlogon
Netstart netlogon
DCdiag /fix|DNS
---------------------------------------
On DC3

set itself to be the primary and DC1 to be the altnernate.

Go to the command prompt and type: DCDiag /fix:DNS
----------------------
On DC2
set itself to be the primary, and DC3 to be the altnernate

Go to the command prompt and type: DCDiag /fix:DNS
_________________


Now, perform a DCiag /test|DNS on all servers to see if you have any errors.

Then, force replicate between all servers if everything is dandy.
Avatar of ITPIP

ASKER

Well I was wrong about DC2 being a DNS server.  Sorry about that misinformation.  I think I was looking at one of the other DCs while I was checking if each one had DNS server running on them and must have not noticed I was looking at DC3 and not DC2.

So I completed all the tasks you listed above except for on DC2 I set DC3 to be it's primary and DC1 to be it's alternate.

I actually ran /test:DNS on all servers after I edited all DNS server settings for the NICs and DC1 and DC2 passed but DC3 did not and that was only because of the forwarders issues.

Now I have completed all tasks on DC1 and then did fix|DNS on all other DCs and now my _msdcs folder will not repopulate with all of the DCs SRV records.  So now on DNS in DC1 that folder only shows DC1 at the root of the _msdcs folder inside of domain.com.

DC3 has the same problem although I never deleted the _msdcs folder from that server on this go around.

Previous to me doing this the other two DCs were listed on both DNS servers.

So now when I run /test:DNS I get errors on DC2 about the CNAME,DC SRV, GC SRV records missing on DC3(DC2's primary DNS server).

I thought waiting it out might repopulate those records but it doesn't look like that is the case.

So now my question is how to I get the CNAME, DC SRV and GC SRV records back in the _msdcs folder on both DNS servers?  Something seems to be stopping DNS from repopulating this informaiton?

I can't complete the force replicate and check FRS till I get through the DNS erros but I think I might have taken myself back a step here.

Ideas?
Avatar of ITPIP

ASKER

Scratch that last post.  Must have forgotten to restart netlogon on all dCs.  This seemed to repopulate the _msdcs records correctly.  All /test:DNS tests have passed except DC3 which still has forwarder errors.  I will try the rest of the tasks you had outlined and check back in after that is done.

Thanks.
There is no need to have the delegation records back, They will replicate DNS..
Try to force replicate... Clean all event logs... Then look for any errors.
Avatar of ITPIP

ASKER

Tried force replicate from DC1 to DC3 and DC1 to DC2.  Still getting 13508 errors in both DC3 and DC2's FRS logs.  I think I should see a 13509 error when everything is good but no luck yet.

Any ideas of where to go next?

Thanks.
Well, let's get an idea of any existing DNS errors.

Go to each DC and type DCdiag /test|DNS
Also, let's test the netlogon service.

Go to the command prompt and type DCdiag /test|netlogons
Avatar of ITPIP

ASKER

Ok I am going to list all the results from /test:DNS ran from all three DCs.  I will then post the results of netlogon after this:
-----------------------------------------------------------------------------------------------------
DC1 Results:

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC1

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
           
            DC: DC1.domain.com
            Domain: domain.com

                 
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC1                         PASS PASS PASS PASS WARN PASS n/a  
         
         ......................... domain.com passed test DNS



-----------------------------------------------------------------------------------------------------
DC2 Results:

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC2
      Starting test: Connectivity
         ......................... DC2 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC2

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         ......................... domain.com passed test DNS

----------------------------------------------------------------------------------------------------
DC3 Results:

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC3
      Starting test: Connectivity
         ......................... DC3 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC3

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
           
            DC: DC3.domain.com
            Domain: domain.com

                 
               TEST: Forwarders/Root hints (Forw)
                  Error: Forwarders list has invalid forwarder: 24.149.0.24 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.6 (<name unavailable>)
                  Error: Forwarders list has invalid forwarder: 24.149.0.7 (<name unavailable>)
                  Error: Root hints list has invalid root hint server: a.root-servers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-servers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-servers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-servers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-servers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-servers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-servers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-servers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-servers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-servers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-servers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (198.32.64.12)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-servers.net. (202.12.27.33)
                 
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 24.149.0.7 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.7
               
            DNS server: 24.149.0.6 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.6
               
            DNS server: 24.149.0.24 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 24.149.0.24
               
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33
               
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42
               
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4
               
            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.32.64.12
               
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129
               
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30
               
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241
               
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17
               
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12
               
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201
               
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10
               
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4
               
            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.8.10.90
               
            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC3                      PASS PASS FAIL PASS WARN PASS n/a  
         
         ......................... domain.com failed test DNS

-------------------------------------------------------------------------------------------------

Earlier you had explained how to fix the forwarder issues but had mentioned that none of this should have any effect on the FRS issues.  Knowing that I have not tackled fixing any of those issues yet.

Next up is netlogons.  I will post seperate though.
Avatar of ITPIP

ASKER

Netlogons Results:


DC1 Results:


Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: NetLogons
         ......................... DC1 passed test NetLogons
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com


--------------------------------------------------------------------------------------------------
DC2 Results:
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC2
      Starting test: Connectivity
         ......................... DC2 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC2
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\DC2\netlogon)
         [DC2] An net use or LsaPolicy operation failed with error 1203, No network provider accepted the given network path..
         ......................... DC2 failed test NetLogons
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com


---------------------------------------------------------------------------------------------------
DC3 Results:

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC3
      Starting test: Connectivity
         ......................... DC3 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC3
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\DC3\netlogon)
         [DC3] An net use or LsaPolicy operation failed with error 1203, No network provider accepted the given network path..
         ......................... DC3 failed test NetLogons
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com

------------------------------------------------------------------------------------------------

As you can see the netlogons test fails on DC2 and DC3 but not DC1.  This matches up with which two servers are getting the FRS errors.

Let me know if you want more info.

Thanks for your help.
Your SRV records for DC2 and DC3 were not created........ That's the netlogon problems...


Now, Let's fix the Forwarders as well as another DNS problem in the reverse lookup zone....

1) On all servers, they should point to the actual IP address, rather than the loopback address for preferred DNS server, As an alternate DNS server on DC2 and DC3, they should have the IP address of DC1....

So, to clarify:
DC1 nics config should look like this:
Preferred DNS server: It's own IP
Alternate DNS server: another DC's IP address hopefully on the same site

DC2:
Preferred: It's own IP address
Alternate: DC1's IP address

DC3:
Preferred: It's own IP address
Alternate: DC1's IP address

--The reason for this, is when you register the SRV records in it's own DNS, then you will replicate that with DC1 and share those SRV records with DC1 as an alternate DC.

2) THEN in the reverse lookup zone on all DCs, eliminate ANY reverse pointers to the loopback address. That would be to any reference to 127.0.0.1, which will show up in the reverse lookup zone as 1.0.0.127...

--The reason is the loopback address causes confusion in DNS lookups

3) ON ALL THREE DCs,  Enable recursion on the forward lookup zone. That effectively makes these DCs look for a forwarding server. You see, recursive lookups have your DNS server perform an iterative query on behalf of the client, and that is the nature of a forwarding server....   The forwarding server's IP addresses are your ISP's address only. NO INTERNAL IP addresses for forwarding servers.

It appears that you have a server with recursion disabled and a couple servers with recursion enabled. This is the reason why your forwarding and root hints servers are showing up in DCdiag...

4) Now, make sure that ALL servers are Global catalog servers.

5) once done, go to the command prompt of both DC2 and DC3 and perform these commands at the command prompt.....

IPconfig /flushdns
IPconfig /registerdns
Net Stop Netlogon
Net Start Netlogon

then the most important command:
DCdiag /fix:DNS

This should register the SRV records on DC2 and DC3.

6) NOW it is time to replicate with DC1....Go into AD sites and services and replicate with DC1 with DC2 and DC1 with DC3... That should resolve any missing SRV records.

Let me know if you run into any problems during this process.



Avatar of ITPIP

ASKER

All those instructions make sense but the only thing I can't do is set DC2's DNS to itself because it's not running DNS server.  The other two servers are running DNS server but not DC2.  So the DNS settings on DC2's NIC currently are set with DC1 as primary and DC3 as secondary.  Will this make a difference while following your instructions?

Thanks.
Not really, but I would also recommend<< making DC2 a DNS server, register its SRV with itself.

REASON being is because if DC1 craps out on you, DC2 can quickly take over as PDC emulator, and DNS server.... All you have to do is add the role of DNS to DC2, and follow those above instructions...

Avatar of ITPIP

ASKER

Looking at the reverse lookup zone on both DC1 and DC3 I see no references to 1.0.0.127...  Also on both servers when right clicking on the server in the DNS windows and clicking on properties.  Then going to the advanced tab and looking at the checkbox for Disable recursion "(also disables forwarders)"  I see that both on DC1 and DC3 this box is not checked, which would mean that recursion is enabled.  Looking at the forwarders tab on DC1 and DC3 I notice also that the "Do not use recursion for this domain" box is NOT checked.  So it looks as though recursion is enabled.  Or am I reading this wrong?  I still haven't added DNS on DC2 yet but I hope to have a chance to do that tonight.

What do you make of the above information though?

Thanks.
It looks good so far..

Before installing DNS on DC2, make sure it's IP is the preferred and DC1 is the alternate on the NIC card configuration.

After enabling DNS on DC2, immediately go to the command prompt and register records.

IPconfig /flushdns
IPconfig /registerdns
Net Stop Netlogon
Net Start Netlogon

then the most important command:
DCdiag /fix:DNS

Make sure in the DNS snapin>>DNS server properties, that all three DCs are recognized as DNS servers that share information between each other. That will allow DNS edits to go between the three servers.

Now, make sure records are fixed on DC3

Then, force replicate between them.
Avatar of ITPIP

ASKER

Step 1 is done.  All the NICs on all the DCs are setup exactly as you specified above.

Step 2 you laid out above regarding looking for loopback addresses in the Reverse Lookup Zones of all DCs.  As posted in my last entry I could not find any references to the loopback address on any of the reverse lookup zones on any of the DCs.

Step 3 as I posted in my last entry, I looked in the proper location to see if the checkbox for disabling recursion was checked on every server and it is not.  So on DC3 I expected it to be disabled but the box wasn't checked so I am assuming that is meaning that recursion is enabled.  Which doesn't match up with what you had been saying regarding the root hint servers and forwarders.

Step 4 is tricky to pinpoint exactly.  In sites and services->sites->Default-First-Site-Name->Servers->DC->NTDS->right-click properties, all DCs have the box checked for global catalog.  Originally only DC1 was a Global Catalog but back sometime not too long ago I had enabled all DCs to be Global Catalog Servers via Sites and Services.  After reading the article http://support.microsoft.com/kb/313994

I noticed that it said AFTER a reboot, replication should start to occur and then you may see an Event ID 1119 in your Directory Services event logs.  I found this event on DC2 but not on DC3.  So I assumed I had rebooted DC2 most likely right after setting it up as GC but I had not done this on DC3.  So I rebooted DC3 last night and I have been waiting to see EVENT ID 1119 in the DS event logs but so far I haven't see it.  This leads me to believe that DC3 is not officially a GC server yet.  Would that be correct?  Maybe DC3 is not officially a GC Server yet and maybe that is part of the problem.

Step 5 I did this on both DC2 and DC3 this morning.

Step 6 I also did this for both DC2 and DC3.  I was assuming that in Sites and Services->Servers that you navigate to DC2's NTDS settings then find in the From Server colum which ever entry is for DC1 and you right click->replicate now.  Then do the same with DC3.

I was waiting to see FRS EVENT ID 13508 go away and be followed up by 13509 but this has not happened on DC2 or DC3 yet.

I still have not seen that yet so I don't know if the above directions worked.  Specifically I am wondering about the recursion and loopback information you posted as I cannot identify either one of those problems on any of my DCs.

Not sure where to go from here as I have completed everything you asked but still do not see any results.

Thanks for your help.
An important step is to register the SRV records on their own DNS services. I assume that was done.

Now, on each DC go to the command prompt and type:

DCdiag /test:FSMO. The FSMO role holder should be DC1 unless DC3 is its own AD domain..

If DC1,2,and3 see dc1 as the fsmo role holder, you are golden. Now all we should have to do is make sure the SRV records for DC1, 2, and 3 are on ALL three DCs.

http://support.microsoft.com/kb/241515

Once these SRV records are on their own DNS, now we can reset replication between them. The least evasive approach is to go into Services and stop the NTFRS replication service. or to go to the command prompt and type:

Net Stop NTFRS
Then
Net Start NTFRS

RESTART FRS: ON DC1 first, then on the other two afterwards.
Explanation:

Journal WRAP is a partial replication set. it means that FRS gets part way through its replication and then stops. To overcome journal Wrap, you have to fix the discrepancy that caused the service to choke in the first place. 99.99% of the time, that is a DNS discrepancy.

If Journal Wrap goes too long without replicating, you may have to reset the replication set using the Burflag method. You will use an authoritative restore on DC1, and a non authoritative restore on DC2 and DC3...

I believe we fixed DNS, you might run DCdiag /test:DNS one more time to see if there are any errors. Before doing so, delet the DNS event logs for a clean look at DNS in its current state.
Avatar of ITPIP

ASKER

Ok, not sure if you saw in my last post but DNS issues are not solved on DC3, and I have looked through all settings in DNS that you had suggested.  Including everything you had outlined regarding recursion.  

I still have the "Root hints list has invalid root hint server:" and "This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33" errors.

I have also looked through all of the SRV records on each DC and they all look correct after being recreated through the process you illustrated in a previous post.

So although I have been using your instructions to fix any dns issues I might have on DC3, I still cannot get those errors to go away when running dcdiag /test:DNS from that server.  Both DC1 and DC2 do not have this issue though.
HMM, where is that loopback address comming from.

In address arpa is definately reverse lookup zone. You expanded the reverse lookup and saw no reference to the loopback address in address arpa??

you can probably safely ignore the root hints issues because I do believe you are using forwarders with recursive lookups.

I would also like to verify you went to the command prompt and flushed the DNS cache>
IPconfig /flushdns
Avatar of ITPIP

ASKER

I know it is wierd because I was wondering where a loopback address is referenced as well.  I have checked the reverse lookup zone multiple times looking for any reference to 127.0.0.1 and I cannot find it.  All of the reverse records I have are for my subnet.

Also, those DNS tests are only failing on DC3 yet DC2 is still having FRS problems after following all of your previous steps.  I even tried the burflag method by setting D4 on DC1 and D2 on DC2 and DC3.  Then restarting FRS starting with DC1.  After I do this I get the message "The File Replication Service is no longer preventing the computer DC2 from becoming a domain controller......."  EVENT ID :13516.  Then two minutes later I get the 13508 message again.

So although I am still having trouble with my DNS issues on DC3 I was thinking that after fixing SRV records and restarting FRS on DC1, DC2 and DC3 I would see success at least for replication between DC1 and DC2.

Also I have run ipconfig /flushdns on DC3 multiple times but I just tried again and ran dcdiag /test:DNS to confirm and still have the errors regarding the loopback address and the forwarders.
Just a quick "drop-back-in". Which account is the ntfrs service running as on each DC?
Avatar of ITPIP

ASKER

All DCs show in the Log On tab for ntfrs to Log on as: Local System Account.
FRS will STOP if ONE DC in the topology is not replicating. THAT WILL BE DC3...

I don't understand what's happening to DC3's DNS, BUT that's OK.

Set DC3's preferred and alternate DNS to DC1 and DC2 respectively.

Now, Uninstall the DNS application and reinstall it.

Then, set DC3 as the preferred and DC1 as the alternate.

Go to the command prompt and type.

IPconfig /flushdns
IPconfig registerdns
Net stop Netlogon
Net start netlogon

DCdiag /fix:DNS

That should unhose DNS, because you are blowing DNS away and reinstalling it.

In the meantime, DC1 and DC2 will be your DNS servers for that site.

Avatar of ITPIP

ASKER

My apologies.  I was on travel for awhile and once I got back I didn't have a chance to try any of those steps from your last post but I found some freetime yesterday evening and I just tried your instructions step by step.

I then ran a DCDIAG /test:DNS and the output looks the same as it was before I uninstalled and reinstalled DNS on DC3.  I also restarted the FRS service looking to see if the reinstallation possibly helped solve the FRS problems and I am still having the same errors in the FRS Event Log as well.

So it appears that those steps did not work.

Again sorry for taking so long to get back to this.  It gets kind of crazy when you aren't in the office for a little while.

Thanks again for your help.
It appears the remaining errors are coming from the reverse lookup zone. Can you provide the DNS errors associated with your servers under a DCdiag /test DNS command?
Avatar of ITPIP

ASKER

I have attached the three servers DCdiag /test:DNS results.  Let me know if you need more info.

Thanks.
DC1 DNS Test Results

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC1

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: DC1.domain.com
            Domain: domain.com

                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC1                         PASS PASS PASS PASS WARN PASS n/a  
         
         ......................... domain.com passed test DNS

--------------------------------------------------------------------------------
DC2 DNS Test Results
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC2
      Starting test: Connectivity
         ......................... DC2 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC2

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: DC2.domain.com
            Domain: domain.com

                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC2                         PASS PASS PASS PASS WARN PASS n/a  
         
         ......................... domain.com passed test DNS
--------------------------------------------------------------------------------
DC3 DNS Test Results

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC3
      Starting test: Connectivity
         ......................... DC3 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC3

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : domain
   
   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:
            
            DC: DC3.domain.com
            Domain: domain.com

                  
               TEST: Forwarders/Root hints (Forw)
                  Error: Root hints list has invalid root hint server: a.root-servers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-servers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-servers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-servers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-servers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-servers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-servers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-servers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-servers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-servers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-servers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (198.32.64.12)
                  Error: Root hints list has invalid root hint server: l.root-servers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-servers.net. (202.12.27.33)
                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53
               
            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.8.10.90
               
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4
               
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10
               
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201
               
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12
               
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17
               
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241
               
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30
               
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129
               
            DNS server: 198.32.64.12 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.32.64.12
               
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4
               
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42
               
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC3                      PASS PASS FAIL PASS WARN PASS n/a  
         
         ......................... domain.com failed test DNS

Open in new window

Avatar of ITPIP

ASKER

Anyone have any ideas on this?  FRS still has errors and I am not sure where to go from here.

Thanks.
DNS is fixed:

Run the Burflag method of resetting FRS...

D4 is for authoritative restore. That is performed on your FSMO role holder.
D2 is a non-authoritative restore. That is done on your other DCs.

http://support.microsoft.com/kb/290762

-----------------------------------------------------------------------------------------------
Also remove all FRS event logs. FRS will freeze if there is a problem...
Avatar of ITPIP

ASKER

Ok but as stated in a previous post above there is not one server that holds all FSMO roles.  Two different servers have these roles so:

DC1 holds Schema Owner and Domain Owner
DC3 holds RID, PDC and Infrastructure Master.

Also from what we had gone over awhile ago in these posts the authoritative restore is to be done on the server with the "good" version of the sysvol.  DC3 upon looking holds nothing in it's sysvol folder.  DC1 however does hold data in it related to policies and scripts.  This is the whole goal of FRS isn't it?  To sync the sysvol to the other domain controllers in the domain.

If so wouldn't DC1 be authoritative and DC2 and DC3 be non-authoritative?  I think in the previous posts we had assumed that DC3 was the authortative server for this but I don't think this is right.  In the end I would like to make DC3 the authoritative server with the authoritative sysvol because I will be looking to soon decommission DC1.

So here is my thinking:
I need to get a working sysvol folder on DC3.  It almost looks like I could copy the sysvol data from DC1 and paste that data into their respective folders on DC3 and DC2 which hold nothing in their sysvol folder currently.  I haven't done this but isn't that the basic funtion of FRS?


Also should I assign all of my FSMO roles to DC3?  If I were to ignore my assumptions about the sysvol folder and just follow your instructions would that be the right thing to do before doing the burflag method?

I don't think we are quite on the same page here and I would like to make sure we get there.

Thanks.
If you're going to bulk reset your SYSVOL with D4/D2, then it dosn't matter which DC that holds any FSMO's. FRS couldn't care less as long they can talk to the replica set members (meaning DNS and replication topology needs to be ok without any FW dropping packets).

What counts is that the DC that is set to be authoritative holds a health SYSVOL if you want to keep your scripts and GPO's.

Have you run a "dcdiag /v /e /c /f:dcdiag.txt" and checked for other errors?

Verify DNS record used by replication: "dnslint /ad /s <ip-address of DNS server> /v"

I dunno if you have tried a tool like "Sonar" to see all FRS replica set members and status?
 
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8c8e0d90-a13b-4977-a4fc-3e2b67e3748e&displaylang=en

If you want to do an authoritative restore of SYSVOL:

Quick overview:

1. Stop the NtFrs service on every DC
2. Set the D4 flag on one DC that will be authoritative for the replica set(s).
3. Set the D2 flag on the other DC’s (non-authoritative)
4. Start the NtFrs service on the “D4¿ DC.
5. Check that Event ID 13553 and 13516 is logged.
6. If step 5 is ok, start NtFrs on the “D2¿ DC’s.


I participated in a thread with event 13508 without 13509. The problem was that the NtFRS service was set to run with different accounts (user accounts!). Problems like that is hard to find over a forum, so maybe you should open a support case with MS PSS.
Avatar of ITPIP

ASKER

DC1 would be where my "healthy" SYSVOL is.  After following the burflag method again and setting DC1 as the Authoritative Server I noticed in the FRS Event Logs that the only Events that have ever occured for the life of that Event Log are Service Start and Stop Events.  There are no other events in the FRS event log on DC1.  DC2 and DC3 however do have quite a few different FRS Events as decribed in the above posts.

Could this be the root of my problem?  It almost seems as though FRS is broken on DC1 and all of the other servers are just waiting for it to get back on the right track.  Since you mentioned that I should see 13553 and 13516 events in my authoritative DC FRS Event Logs and I do not it seems like I should focus on that first.

What do you think?
When you stop the ntfrs service and set the Burflags to D4 you should see some events in the FRS event log when ntfrs is started.

Verify that the Burflags key is changed back to "0". (this is the first things that should happen).

If the key is reverted, you should get a Event ID 13566 event in the FRS event log.. (authoritative restore in progress). If it isn't logged:

On DC1:

- verify that event log service is started
- check for other events
- dcdiag /v /c /f:dc1diag.txt (please attach the log)



Avatar of ITPIP

ASKER

I do not get an FRS Event ID 13566 on DC1 when I set the burflag to D4 and start the service again, following the procedure you outlined.  The burflag goes back to zero but the frs logs only show frs stop and start events and nothing else.

I am attaching the dcdiag you requested, run from DC1.

Thanks.
Domain Controller Diagnosis

Performing initial setup:
   * Verifying that the local machine DC1, is a DC. 
   * Connecting to directory service on server DC1.
   * Collecting site info.
   * Identifying all servers.
   * Identifying all NC cross-refs.
   * Found 3 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         * Active Directory RPC Services Check
         ......................... DC1 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\DC1
      Starting test: Replications
         * Replications Check
         * Replication Latency Check
            DC=ForestDnsZones,DC=domain,DC=com
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DomainDnsZones,DC=domain,DC=com
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=domain,DC=com
               Latency information for 19 entries in the vector were ignored.
                  19 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=domain,DC=com
               Latency information for 19 entries in the vector were ignored.
                  19 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=domain,DC=com
               Latency information for 17 entries in the vector were ignored.
                  17 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         ......................... DC1 passed test Replications
      Starting test: Topology
         * Configuration Topology Integrity Check
         * Analyzing the connection topology for DC=ForestDnsZones,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=DomainDnsZones,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Schema,CN=Configuration,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Configuration,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... DC1 passed test Topology
      Starting test: CutoffServers
         * Configuration Topology Aliveness Check
         * Analyzing the alive system replication topology for DC=ForestDnsZones,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=DomainDnsZones,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Schema,CN=Configuration,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Configuration,DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=domain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... DC1 passed test CutoffServers
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC DC1.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=domain,DC=com
            (NDNC,Version 2)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=domain,DC=com
            (NDNC,Version 2)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=domain,DC=com
            (Schema,Version 2)
         * Security Permissions Check for
           CN=Configuration,DC=domain,DC=com
            (Configuration,Version 2)
         * Security Permissions Check for
           DC=domain,DC=com
            (Domain,Version 2)
         ......................... DC1 passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\DC1\netlogon
         Verified share \\DC1\sysvol
         ......................... DC1 passed test NetLogons
      Starting test: Advertising
         The DC DC1 is advertising itself as a DC and having a DS.
         The DC DC1 is advertising as an LDAP server
         The DC DC1 is advertising as having a writeable directory
         The DC DC1 is advertising as a Key Distribution Center
         The DC DC1 is advertising as a time server
         The DS DC1 is advertising as a GC.
         ......................... DC1 passed test Advertising
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com
         ......................... DC1 passed test KnowsOfRoleHolders
      Starting test: RidManager
         * Available RID Pool for the Domain is 12020 to 1073741823
         * DC3.domain.com is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 11520 to 12019
         * rIDPreviousAllocationPool is 8020 to 8519
         * rIDNextRID: 8405
         ......................... DC1 passed test RidManager
      Starting test: MachineAccount
         Checking machine account for DC DC1 on DC DC1.
         * SPN found :LDAP/DC1.domain.com/domain.com
         * SPN found :LDAP/DC1.domain.com
         * SPN found :LDAP/DC1
         * SPN found :LDAP/DC1.domain.com/domain
         * SPN found :LDAP/e6ab78ac-6565-4693-9989-e31eb6c0b230._msdcs.domain.com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/e6ab78ac-6565-4693-9989-e31eb6c0b230/domain.com
         * SPN found :HOST/DC1.domain.com/domain.com
         * SPN found :HOST/DC1.domain.com
         * SPN found :HOST/DC1
         * SPN found :HOST/DC1.domain.com/domain
         * SPN found :GC/DC1.domain.com/domain.com
         ......................... DC1 passed test MachineAccount
      Starting test: Services
         * Checking Service: Dnscache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: RpcSs
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... DC1 passed test Services
      Starting test: OutboundSecureChannels
         * The Outbound Secure Channels test
         ** Did not run Outbound Secure Channels test
         because /testdomain: was not entered
         ......................... DC1 passed test OutboundSecureChannels
      Starting test: ObjectsReplicated
         DC1 is in domain DC=domain,DC=com
         Checking for CN=DC1,OU=Domain Controllers,DC=domain,DC=com in domain DC=domain,DC=com on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com in domain CN=Configuration,DC=domain,DC=com on 1 servers
            Object is up-to-date on all servers.
         ......................... DC1 passed test ObjectsReplicated
      Starting test: frssysvol
         * The File Replication Service SYSVOL ready test 
         File Replication Service's SYSVOL is ready 
         ......................... DC1 passed test frssysvol
      Starting test: frsevent
         * The File Replication Service Event log test 
         ......................... DC1 passed test frsevent
      Starting test: kccevent
         * The KCC Event log test
         Found no KCC errors in Directory Service Event log in the last 15 minutes.
         ......................... DC1 passed test kccevent
      Starting test: systemlog
         * The System Event log test
         Found no errors in System Event log in the last 60 minutes.
         ......................... DC1 passed test systemlog
      Starting test: VerifyReplicas
         ......................... DC1 passed test VerifyReplicas
      Starting test: VerifyReferences
         The system object reference (serverReference)

         CN=DC1,OU=Domain Controllers,DC=domain,DC=com and backlink on

         CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com

         are correct. 
         The system object reference (frsComputerReferenceBL)

         CN=DC1,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=domain,DC=com

         and backlink on CN=DC1,OU=Domain Controllers,DC=domain,DC=com are

         correct. 
         The system object reference (serverReferenceBL)

         CN=DC1,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=domain,DC=com

         and backlink on

         CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=com

         are correct. 
         ......................... DC1 passed test VerifyReferences
      Starting test: VerifyEnterpriseReferences
         ......................... DC1 passed test VerifyEnterpriseReferences
      Starting test: CheckSecurityError
         * Dr Auth:  Beginning security errors check!
         Found KDC DC1 for domain domain.com in site Default-First-Site-Name
         Checking machine account for DC DC1 on DC DC1.
         * SPN found :LDAP/DC1.domain.com/domain.com
         * SPN found :LDAP/DC1.domain.com
         * SPN found :LDAP/DC1
         * SPN found :LDAP/DC1.domain.com/domain
         * SPN found :LDAP/e6ab78ac-6565-4693-9989-e31eb6c0b230._msdcs.domain.com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/e6ab78ac-6565-4693-9989-e31eb6c0b230/domain.com
         * SPN found :HOST/DC1.domain.com/domain.com
         * SPN found :HOST/DC1.domain.com
         * SPN found :HOST/DC1
         * SPN found :HOST/DC1.domain.com/domain
         * SPN found :GC/DC1.domain.com/domain.com
         [DC1] No security related replication errors were found on this DC!  To target the connection to a specific source DC use /ReplSource:<DC>.
         ......................... DC1 passed test CheckSecurityError

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : domain
      Starting test: CrossRefValidation
         ......................... domain passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... domain passed test CheckSDRefDom
   
   Running enterprise tests on : domain.com
      Starting test: Intersite
         Skipping site Default-First-Site-Name, this site is outside the scope

         provided by the command line arguments provided. 
         ......................... domain.com passed test Intersite
      Starting test: FsmoCheck
         GC Name: \\DC1.domain.com
         Locator Flags: 0xe00001fc
         PDC Name: \\DC3.domain.com
         Locator Flags: 0xe00003fd
         Time Server Name: \\DC1.domain.com
         Locator Flags: 0xe00001fc
         Preferred Time Server Name: \\DC1.domain.com
         Locator Flags: 0xe00001fc
         KDC Name: \\DC1.domain.com
         Locator Flags: 0xe00001fc
         ......................... domain.com passed test FsmoCheck
      Starting test: DNS
         Test results for domain controllers:
            
            DC: DC1.domain.com
            Domain: domain.com

                  
               TEST: Authentication (Auth)
                  Authentication test: Successfully completed
                  
               TEST: Basic (Basc)
                   Microsoft(R) Windows(R) Server 2003, Enterprise Edition (Service Pack level: 2.0) is supported
                  NETLOGON service is running
                  kdc service is running
                  DNSCACHE service is running
                  DNS service is running
                  DC is a DNS server
                  Network adapters information:
                  Adapter [00000001] Intel(R) PRO/1000 MT Network Connection:
                     MAC address is 00:0D:56:24:52:2E
                     IP address is static
                     IP address: 10.10.0.4
                     DNS servers:
                        10.10.0.4 (<name unavailable>) [Valid]
                        10.10.0.5 (<name unavailable>) [Valid]
                  The A record for this DC was found
                  The SOA record for the Active Directory zone was found
                  The Active Directory zone on this DC/DNS server was found (primary)
                  Root zone on this DC/DNS server was not found
                  
               TEST: Forwarders/Root hints (Forw)
                  Recursion is enabled
                  Forwarders Information: 
                     24.149.0.24 (<name unavailable>) [Valid] 
                     24.149.0.6 (<name unavailable>) [Valid] 
                     24.149.0.7 (<name unavailable>) [Valid] 
                  
               TEST: Delegations (Del)
                  No delegations were found in this zone on this DNS server
                  
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure domain.com.
                  Test record _dcdiag_test_record added successfully in zone domain.com.
                  Test record _dcdiag_test_record deleted successfully in zone domain.com.
                  
               TEST: Records registration (RReg)
                  Network Adapter [00000001] Intel(R) PRO/1000 MT Network Connection:
                     Matching A record found at DNS server 10.10.0.4:
                     DC1.domain.com

                     Matching CNAME record found at DNS server 10.10.0.4:
                     e6ab78ac-6565-4693-9989-e31eb6c0b230._msdcs.domain.com

                     Matching DC SRV record found at DNS server 10.10.0.4:
                     _ldap._tcp.dc._msdcs.domain.com

                     Matching GC SRV record found at DNS server 10.10.0.4:
                     _ldap._tcp.gc._msdcs.domain.com

         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 10.10.0.4 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server 
               Name resolution is funtional. _ldap._tcp SRV record for the forest root domain is registered 
               
            DNS server: 10.10.0.5 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server 
               Name resolution is funtional. _ldap._tcp SRV record for the forest root domain is registered 
               
            DNS server: 24.149.0.24 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server 
               
            DNS server: 24.149.0.6 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server 
               
            DNS server: 24.149.0.7 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server 
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: domain.com
               DC1                         PASS PASS PASS PASS WARN PASS n/a  
         
         ......................... domain.com passed test DNS

Open in new window

hmm..weird.  Your dcdiag of DC1 looks just perfect.

When you set the D4 flag on this DC it should log that it will become authoritative for the replica set (SYSVOL)

Did you try "sonar.exe" to see if you spot something? (i.e. the replica set members etc)

Do you have any events in the Ntfrs event log on DC1?



Avatar of ITPIP

ASKER

I'm going to look over the documentation of sonar to see if I missed something the last time I ran it because it looked pretty normal to me.  It listed both servers it was supposed to be replicating to and I didn't see anything that looked like warnings or errors.

I do not have anything in the File Replication Service Event log besides service stopping, stopped and service started messages.  There are no other entries in the FRS event log for the entire history of the FRS event log on DC1 except for service starting and stopped messages.

It seems like there is something wrong with the FRS on DC1 even though this server has something in its SYSVOL folder and niether DC2 or DC3 have anything in there SYSVOL folders.
FRSDiag will give you more details than sonar.

http://www.microsoft.com/downloads/en/details.aspx?FamilyId=43CB658E-8553-4DE7-811A-562563EB5EBF&displaylang=en

The logs should say something when you set the burflags.


I think you should first focus on DC1 since it looks like it will not get authoritative for the SYSVOL replica set (if your FRS event log ain't disabled).
FYI, this dosn't look like an "ordinary" error, so if you're in a hurry you should open a support case with Microsoft.
Here is a good FRSDiag blog entry from the DS team. Give it a read.

http://blogs.technet.com/b/askds/archive/2008/05/30/how-to-get-the-most-from-your-frsdiag.aspx
Avatar of ITPIP

ASKER

I'm not in a huge hurry.  I am only trying to fix any problems on my domain so I can be ready to decomission DC1 and replace it with a Server 2008 domain controller in the future.  In order to be ready for this I need to fix any issues with my AD.  If it comes down to it I might get MS involvled.  I am trying to avoid that though.

I have attached the output from the FRSDiag on DC3 as it has errors in it.  The file looks very similiar to what DC2's FRSDiag output looks like.  DC1 however passes every test.

Thanks.
------------------------------------------------------------
FRSDiag v1.7 on 11/8/2010 2:20:54 PM
.\DC3 on 2010-11-08 at 2.20.54 PM
------------------------------------------------------------

Checking for errors/warnings in FRS Event Log .... 	
NtFrs	11/8/2010 2:04:24 PM	Warning	13508	The File Replication Service is having trouble enabling replication  from DC1 to DC3 for c:\windows\sysvol\domain using the DNS name DC1.domain. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name DC1.domain from this computer.     [2] FRS is not running on DC1.domain.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.	
NtFrs	11/8/2010 2:04:24 PM	Warning	13508	The File Replication Service is having trouble enabling replication  from DC2 to DC3 for c:\windows\sysvol\domain using the DNS name DC2.domain. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name DC2.domain from this computer.     [2] FRS is not running on DC2.domain.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.	
NtFrs	11/8/2010 2:02:44 PM	Warning	13565	File Replication Service is initializing the system volume with data from another  domain controller. Computer DC3 cannot become a domain controller until this process  is complete. The system volume will then be shared as SYSVOL.        To check for the SYSVOL share, at the command prompt, type:    net share        When File Replication Service completes the initialization process, the SYSVOL  share will appear.        The initialization of the system volume can take some time.  The time is dependent on the amount of data in the system volume,  the availability of other domain controllers, and the replication  interval between domain controllers.	
NtFrs	11/8/2010 2:02:42 PM	Warning	13566	File Replication Service is scanning the data in the system volume. Computer DC3  cannot become a domain controller until this process is complete.  The system volume will then be shared as SYSVOL.        To check for the SYSVOL share, at the command prompt, type:    net share        When File Replication Service completes the scanning process, the SYSVOL  share will appear.        The initialization of the system volume can take some time.  The time is dependent on the amount of data in the system volume.	
NtFrs	11/7/2010 3:51:41 PM	Warning	13508	The File Replication Service is having trouble enabling replication  from DC1 to DC3 for c:\windows\sysvol\domain using the DNS name DC1.domain. FRS will keep retrying.     Following are some of the reasons you would see this warning.         [1] FRS can not correctly resolve the DNS name DC1.domain from this computer.     [2] FRS is not running on DC1.domain.     [3] The topology information in the Active Directory for this replica has not  yet replicated to all the Domain Controllers.         This event log message will appear once per connection, After the problem  is fixed you will see another event log message indicating that the connection  has been established.	
NtFrs	11/4/2010 10:02:34 AM	Warning	13565	File Replication Service is initializing the system volume with data from another  domain controller. Computer DC3 cannot become a domain controller until this process  is complete. The system volume will then be shared as SYSVOL.        To check for the SYSVOL share, at the command prompt, type:    net share        When File Replication Service completes the initialization process, the SYSVOL  share will appear.        The initialization of the system volume can take some time.  The time is dependent on the amount of data in the system volume,  the availability of other domain controllers, and the replication  interval between domain controllers.
	WARNING: Found Event ID 13508 errors without trailing 13509 ... see above for (up to) the 3 latest entries!

 ......... failed 4
Checking for errors in Directory Service Event Log .... passed
Checking for minimum FRS version requirement ... passed
Checking for errors/warnings in ntfrsutl ds ... passed
Checking for Replica Set configuration triggers... passed
Checking for suspicious file Backlog size... passed
Checking Overall Disk Space and SYSVOL structure (note: integrity is not checked)... passed
Checking for suspicious inlog entries ... passed
Checking for suspicious outlog entries ... passed
Checking for appropriate staging area size ... passed
Checking for errors in debug logs ...
	ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     1388:   904: S0: 14:17:59> :SR: Cmd 01454ba8, CxtG 5b83590c, WS ERROR_ACCESS_DENIED, To   DC1.domain Len:  (362) [SndFail - Send Penalty]
	ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     1388:   877: S0: 14:20:20> :SR: Cmd 01454fb0, CxtG 5b83590c, WS ERROR_ACCESS_DENIED, To   DC1.domain Len:  (362) [SndFail - rpc call]
	ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     1388:   904: S0: 14:20:20> :SR: Cmd 01454fb0, CxtG 5b83590c, WS ERROR_ACCESS_DENIED, To   DC1.domain Len:  (362) [SndFail - Send Penalty]
	ERROR on NtFrs_0005.log : "EPT_S_NOT_REGISTERED(This may indicate that DNS returns the IP address of the wrong computer. Check DNS records being returned, Check if FRS is currently running on the target server. Check if Ntfrs is registered with the End-Point-Mapper on target server!)" : <SndCsMain:                     1388:   884: S0: 14:02:44> :SR: Cmd 01458910, CxtG 075b2240, WS EPT_S_NOT_REGISTERED, To   DC2.domain Len:  (362) [SndFail - rpc exception]
	ERROR on NtFrs_0005.log : "EPT_S_NOT_REGISTERED(This may indicate that DNS returns the IP address of the wrong computer. Check DNS records being returned, Check if FRS is currently running on the target server. Check if Ntfrs is registered with the End-Point-Mapper on target server!)" : <SndCsMain:                     1388:   883: S0: 14:02:54> ++ ERROR - EXCEPTION (000006d9) :  WStatus: EPT_S_NOT_REGISTERED
	ERROR on NtFrs_0005.log : "EPT_S_NOT_REGISTERED(This may indicate that DNS returns the IP address of the wrong computer. Check DNS records being returned, Check if FRS is currently running on the target server. Check if Ntfrs is registered with the End-Point-Mapper on target server!)" : <SndCsMain:                     1388:   884: S0: 14:02:54> :SR: Cmd 01458b08, CxtG 075b2240, WS EPT_S_NOT_REGISTERED, To   DC2.domain Len:  (362) [SndFail - rpc exception]

	Found 7400 ERROR_ACCESS_DENIED error(s)! Latest ones (up to 3) listed above
	Found 8 EPT_S_NOT_REGISTERED error(s)! Latest ones (up to 3) listed above

 ......... failed with 7408 error entries
Checking NtFrs Service (and dependent services) state...
	ERROR : Cannot access SYSVOL share on DC3
	ERROR : Cannot access NETLOGON share on DC3
 ......... failed 2
Checking NtFrs related Registry Keys for possible problems...
	SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SysvolReady = 0 :: ERROR: SysvolReady is not set to 1 :: SYSVOL is likely not Sharing! This key should NOT be changed manually but this should be addressed! See article KB.327781 (How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server) for further information!
failed with 1 error(s) and 0 warning(s)

Checking Repadmin Showreps for errors...passed

Open in new window

Please attach the Connstat.txt from all DCs and a sceenshot of the NTDS Settings object in AD Sites & Services (expand them all).
Do you have any problems with AD replication? (i.e. create a user on DC1, is it replicated to DC2 and DC3?)
Avatar of ITPIP

ASKER

The connstat.txt file in DC1 is blank.  Below is connstat.txt from DC2 and DC3.  

Sending an image of the NTDS settings might be a little tougher as I have been renaming my servers and domain names for privacy when posting data here.  I will see what I can come up with for that.

Thanks.
connstat.txt from DC2




   Replica: DOMAIN SYSTEM VOLUME (SYSVOL SHARE) (c280448f-702c-480f-87aff7c5a08dcd1a)
      Member: DC2         ServiceState: 3  (ACTIVE)  OutLogSeqNum: 1        OutlogCleanup: -1        Delta: 2       

      Config Flags: Multimaster Seeding PrimaryUndefined 
      Root Path   : c:\windows\sysvol\domain
      Staging Path: c:\windows\sysvol\staging\domain
      File Filter : *.tmp, *.bak, ~*
      Dir Filter  : 

                                                                                                 Send           Cleanup     Cos
      Partner         I/O   State        Rev      LastJoinTime            OLog State      Leadx  Delta   Trailx  Delta  LMT Out     Last VVJoin

domain\DC1$          In   Unjoined       0                     Time???
domain\DC3$       Out  Unjoined       8                     Time???  OLP_UNJOINED          0 1             0 1        0  0                           
domain\DC1$          Out  Unjoined       0                     Time???  OLP_UNJOINED          0 1             0 1        0  0                           
domain\DC3$       In   Unjoined       0                     Time???

--------------------------------------------------------------------------------
Connstat.txt from DC3

   Replica: DOMAIN SYSTEM VOLUME (SYSVOL SHARE) (c280448f-702c-480f-87aff7c5a08dcd1a)
      Member: DC3      ServiceState: 3  (ACTIVE)  OutLogSeqNum: 1        OutlogCleanup: -1        Delta: 2       

      Config Flags: Multimaster Seeding PrimaryUndefined 
      Root Path   : c:\windows\sysvol\domain
      Staging Path: c:\windows\sysvol\staging\domain
      File Filter : *.tmp, *.bak, ~*
      Dir Filter  : 

                                                                                                 Send           Cleanup     Cos
      Partner         I/O   State        Rev      LastJoinTime            OLog State      Leadx  Delta   Trailx  Delta  LMT Out     Last VVJoin

domain\DC1$          In   Unjoined       0                     Time???
domain\DC1$          Out  Unjoined       0                     Time???  OLP_UNJOINED          0 1             0 1        0  0                           
domain\DC2$          In   Unjoined       0                     Time???
domain\DC2$          Out  Unjoined       8                     Time???  OLP_UNJOINED          0 1             0 1        0  0

Open in new window

Avatar of ITPIP

ASKER

I have no replication issues with users.  Just tested to confirm that.  Created a user on DC1 and DC2 and DC3 both have that user in their AD Users and Computers.

Thanks.
Avatar of ITPIP

ASKER

Without posting a screenshot I can tell you what the NTDS settings look like under Sites->Servers

DC1 shows DC3 and DC1 in its settings

DC2 shows DC1 and DC3 in its settings

DC3 shows DC1 and DC2 in its settings.

All of those show <automatically generated> under the name column for each entry.

Each "From Site" column holds Default-First-Site-Name

Each type column shows Connection under the Type column

Hope that explains enough.

Thanks.
Please verify that this was not a typo:

DC1 shows DC3 and DC1 in its settings
Avatar of ITPIP

ASKER

Yep, typo.  
That should read "DC1 shows DC3 and DC2 in its settings"

I double checked in sites and services.  Just typed it wrong.

Thanks.
DC2 and DC3's SYSVOL is in a seeding state and is waiting for an authoritative inbound partner (DC1).

I'm not sure why DC1 is missing its Connstat.txt. There is little documentation around this. I'm not sure if it thinks it's out of the replica set...
Avatar of ITPIP

ASKER

Exactly.  They are looking for their authoritative SYSVOL from DC1 and its not providing it.  

Also just to note when running FRSDIAG from DC1 and choosing Tools-> Force Replication on Target Servers.

DC1 comes back with

Detecting this machine's domain role... Domain Controller
Gathering ntfrsutl sets output and gathering all Upstream Partners... Done!
Triggering Pull replication FROM all detected Upstream Partners...
     Could not detect any upstream partners, this server seems to be Orphaned!  You should double check this!

All Done!


The other two servers finish this command without issue.  I have been trying to track that error down but I haven't had much luck yet.

Thanks.
It looks like you have checked DNS with ChiefIT by your side, so I guess it's ok. If you don't have any rough records in the _msdcs sub-domain I would think of rebuilding the SYSVOL three.

I'm not sure if you have set the D4 flag on DC2 or DC3 earlier, but DC1 will not become authoritative (DC2 and DC3 is in an initial state and DC1 is missing the Connstat).

If you want to proceed without calling PSS, my 10 cents goes to rebuilding SYSVOL.

See: How to rebuild the domain SYSVOL replica set across enterprise environments

http://support.microsoft.com/kb/315457

It's up to you to decide.

I'm not sure ChiefIT is reading this and if he have any other suggestions?
Avatar of ITPIP

ASKER

Appreciate the suggestion.  I will give it a read and proceed from there.

Thanks for the help.
> "   Could not detect any upstream partners, this server seems to be Orphaned! "

Does AD replicaction work between DC1 and DC2?

If yes, DC1 is orphaned from the SYSVOL replica set.
Avatar of ITPIP

ASKER

Yes AD replication works between DC1 and DC2.  I can see a user created on DC1's Users and Computers on DC2's User and Computers.  

So if DC1 is orphaned would rebuilding the SYSVOL replica set be my next and probably only option?
AD replication and FRS both uses DNS, LDAP, RPC (and KCC) to be able to replicate.

Since AD replication is ok the replication topology, DNS and "RPC end-point mapper" should be ok.

As said, there is little "how to intepret FRS debug logs" documentation avalible, so if you want to continue without MS a SYSVOL rebuild might work.

Another option is to backup the SYSVOL files on DC1, demote it and set ie. DC2 as authoritative. (I'm not sure you will succeed getting DC2 out of the seeding state without problems)
Avatar of ITPIP

ASKER

I have one more question about this.  Since my goal all along has been to get my domain healthy for the addition of a Server 2008 DC I was wondering if this even needs to be done.

One of the funny things I read right at the end of the "How to rebuild the SYSVOL and it's content in a domain" is the "how to temporarily stabalize the domain SYSVOL tree".

It basically sounds like a copy and paste into the other two DCs corresponding SYSVOL folder.  Knowing that you can get away with that, couldn't I just do that, then demote DC1, then set burflags to D4 on DC3 and D2 on DC2?  Of course before I demoted DC1 I would move its FSMO roles over to DC3.  I might have to manually create the NETLOGON share on both DC2 and DC3, but looking at DC1 that share just points to the scripts folder.  Once that is done couldn't I then continue on my merry way with readying the domain for the next Server 2008 DC?

Am I making this too simple?  I only want to make sure that if I decomission DC1 my domain will continue to funtion normally.  I am worried about that because I had a hardware failure on DC1 months ago and when that was down I had many problems such as users not being able to login and outlook clients not being able to connect to exchange.  Once that server was back up all those problems went away.  I did some work to try and make the domain more fault tolerant by making the two other DCs Global Catalog servers which wasn't done previously and also moving some of the FSMO roles to DC3, the healthiest of the 3 domain controllers.

So if I can easily cut DC1 out of the picture and just rely on DC2 and DC3 I would be happy because then I can work on getting my new Server 2008 DC on the domain and continue to move forward.

What do you think?
Sometimes the simple solution is the best solution... I can't tell you what will work for sure, but I would try to get DC1 back and stable in the replica set. It looks like it's orphan (from the FRS side of view, not as a DC).

From the Connstat file both DC2 and DC3 knows its Outbound partners.
"OutlogCleanup: -1" just shows it's waiting for a partner connection (I verified this in a lab).

I'm a little worried if you force DC1 out, the remaining DC's will stay put in the seeding state.
Avatar of ITPIP

ASKER

Well I tried to make some time for this last week and it just didn't work out.  I started to go through the rebuild and when I got to the step to run the following command:
ntfrsutl ds |findstr /i "root stage"
Nothing was returned on DC1.

I thought I could run the linkd command to recreate the reparse point but I was getting an error for the staging area step.  Basically it would read the whole command till it got to the space in "staging area" and then would error out because it didn't know what directory I was referring to.  I think the error was something like Could not create link at c:\...\staging

Even though the command was run witht the full path to the staging area folder in the SYSVOL.

Thats not a great explanation of exactly what happened but I hoped to have a chance over the holidays to try this again and read up a little bit more about the linkd command.

I think you are right snus that the best and probably only way to fix this is to rebuild the SYSVOL but I want to fix it before confirming that and submitting the points.  I imagine I will be able to repost better or final results by next week.
ASKER CERTIFIED SOLUTION
Avatar of snusgubben
snusgubben
Flag of Norway image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of ITPIP

ASKER

You know what snus, in the moment while I was doing this I don't think I noticed I wasn't using the quotes.  It makes perfect sense right now but at the time I was kind of rushed.

I will start the SYSVOL rebulid process over again and I'm sure I will get past the linkd command now that I see my error.  I will also read that link you sent me before hand to get a better handle on this.

As you said DC1 is obviously missing something.  It's wierd that the command doesn't return anything because if you navigate to the SYSVOL directory and it's contents in explorer you will see that the reparse points are setup "correctly" as the folder SYSVOL\Sysvol\domain.com is pointing to SYSVOL\domain.

You mentioned rebuilding the SYSVOL on DC2 and DC3 but the only problem I see with that is that I would have to manually copy and paste the "domain" folder contents from DC1 to DC2 and DC3 because DC1 is the only server that has all of the group policies and login scripts(because FRS never worked right).  If it's just that simple I wouldn't mind just leaving DC1 out of it but it seems like DC1 needs to be fixed before I can take it off the domain only because that was the original Domain Controller.  It seems like DC2 and DC3 were all added later but FRS never worked right when those were added as domain controllers.  I think if I can get this working then I can safely move the FSMO roles to DC2 and/or DC3 and demote DC1.  But to demote DC1 before that seems like I might be taking a chance that SYSVOL will not ever be right on the remaining two domain controllers again.

Thanks again for the suggestions.  I hope to be able to retry the rebuild over the holiday and post back Monday or Tuesday with the results.
Please try this on all DCs:

DCdiag /fix|DNS
I would also like to see if we have a couple servers that are confused as to who owns the different FSMO roles:

Could you try this:
DCdiag /test:FSMO

on all three servers. They should all say the exact same thing.
Also, check this thread out on your LSA policy issue:

This could resolve all your remaining replication probs.

https://www.experts-exchange.com/questions/25148164/Unable-to-connect-to-the-NETLOGON-share-An-net-use-or-LsaPolicy-operation-failed-with-error-1203.html
Avatar of ITPIP

ASKER

Chief,

The thread you posted does look promising but it references Windows Server 2000.  I would like to try this but I want to make sure this doesn't cause any surprises.

Can I do this on Server 2003?  Also, am I correct in assuming that I would add this file on all three servers?

I will say his problem looks very similiar to mine.  It appears he just stumbled upon it differently.
I once had the 13559 error on one DC back in the 2000 days, and followed the recomendation. I have never seen a KB describing this for 2003 and never.

According to this technet blog it fixed a 2003 DC:

http://blogs.technet.com/b/carlh/archive/2010/06/22/frs-problem-when-virtualising-a-domain-controller.aspx

If this will help you out I can't tell :)
Avatar of ITPIP

ASKER

I was just writing this when I saw your reply Snus but see below for how it was FIXED:

I got it!  I really can't believe I have finally gotten this fixed.  It took way too long but snus you steered me in the right direction.

I had to read and re-read the link from Microsoft you sent me called "Recovering missing FRS objects and FRS attributes in Active Directory".  I still didn't quite understand the instructions that it was providing for usage of ADSIedit but after running "ntfrsutl ds" on DC1 and seeing it return "DC1 IS NOT A MEMBER OF ANY SET!" I started doing some research into that and I think one of the first things that popped up was the following link:http://www.shantilal.net/technotes/1.html

This is basically a simplified version of the link you initially provided.  I followed those directions suplemented by the Microsoft KB article and all is working great now.  All three DC's now have a working SYSVOL.

I wish I would have gone in this direction initially but this being a rather new issue to me and not knowing a lot about how FRS works I couldn't quite figure out which direction to go in.

Chief, your input was helpful but it didn't really steer me in the right direction for me to feel like you provided the correct solution.  I have been looking over this thread and I can't find any posts provided by you that I could accept as a partial solution as it would not provide the correct solution to anyone using this post to help solve this specific problem for themselves.  I do appreciate your help in troubleshooting any possible DNS issues but in the end that is not what the problem was.

Again, I appreciate everyone's help in solving this issue.  EE comes through again!
Avatar of ITPIP

ASKER

Following the link to KB 312862 which led me to the link I provided below solved this issue for me.  My problem DC had been orphaned from the FRS set and needed to be re-added via ADSIedit.

Thanks for your help!
Glad you got it back on track.

Cheers!