Link to home
Start Free TrialLog in
Avatar of sisaacso170
sisaacso170Flag for United States of America

asked on

AD replication errors, RPC server is unavailable

Over the weekend i created two Server 2008 R2 servers (dc1 and dc2) and made them both domain controllers.  There was already a current server 2003 (HTE)acting as a domain controller.  It took awhile to promote the first server since it was going from 32 bit to 64 bit, but after all command line preps it required, I finally got it working.  I then transferred the fsmo roles to dc1.  Adding the second dc was pretty simple after the first one was done.  It worked fine for two days, but today it started taking a long time for computers to login to the domain.  I checked the error logs and there were a ton of errors with event id 1058 (group policy errors).  When i try to open AD users and computers, I get an error message that says "RPC server unavailable", and it won't open.  If i reboot, it will sometimes open up AD users and computers, but after a few minutes it will cease working.  I did a net share, and see that my sysvol share and netlogon share are not even there on dc1 and dc2, but is there on HTE.  I didn't check if the shares were there before now before now, but everything was working well before today.  The below output is from a dcdiag.  You can see i have some errors.  I had many more before i turned netbios on the server.  I think I have a DNS issue and the shares not being there for netlogon and sysvol.  I also don't understand the advertising error, acts like it is trying to look at the wrong server.  I have spent hours on this and can not figure it out, any guidance would be appreciated.

One note, I do not know if it is the netbios being turned on is what made some of my errors go away, but it does seem to have helped.  AD doesn't seem to be so sluggish and unresponsive now.

C:\Windows\system32>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = DC1
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\DC1
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\hte.loganutah.org,
         when we were trying to reach DC1.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... DC1 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... DC1 passed test FrsEvent
      Starting test: DFSREvent
         ......................... DC1 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... DC1 passed test SysVolCheck
      Starting test: KccEvent
         ......................... DC1 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... DC1 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... DC1 passed test MachineAccount
      Starting test: NCSecDesc
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=ForestDnsZones,DC=loganutah,DC=org
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=DomainDnsZones,DC=loganutah,DC=org
         ......................... DC1 failed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\DC1\netlogon)
         [DC1] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... DC1 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... DC1 passed test ObjectsReplicated
      Starting test: Replications
         ......................... DC1 passed test Replications
      Starting test: RidManager
         ......................... DC1 passed test RidManager
      Starting test: Services
         ......................... DC1 passed test Services
      Starting test: SystemLog
         A warning event occurred.  EventID: 0x000003F6
            Time Generated: 02/24/2010   22:35:05
            Event String:
            Name resolution for the name www.microsoft.com timed out after none
of the configured DNS servers responded.
         ......................... DC1 passed test SystemLog
      Starting test: VerifyReferences
         ......................... DC1 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : loganutah
      Starting test: CheckSDRefDom
         ......................... loganutah passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... loganutah passed test CrossRefValidation

   Running enterprise tests on : loganutah.org
      Starting test: LocatorCheck
         ......................... loganutah.org passed test LocatorCheck
      Starting test: Intersite
         ......................... loganutah.org passed test Intersite

C:\Windows\system32>

C:\Windows\system32>net share

Share name   Resource                        Remark

----------------------------------------------------------------
C$           C:\                             Default share
IPC$                                         Remote IPC
ADMIN$       C:\Windows                      Remote Admin
The command completed successfully.


C:\Windows\system32>

Avatar of XU_LY
XU_LY

Please see http://support.microsoft.com/kb/967482.
http://support.microsoft.com/kb/829306.
http://technet.microsoft.com/en-us/library/cc754463(WS.10).aspx

If you have not run the adprep /rodcprep command, Dcdiag.exe returns an error when it runs the NCSecDesc test. This test checks that the security descriptors on the naming context heads have appropriate permissions for replication. The error indicates that the Enterprise Domain Controllers group does not have Replicating Directory Changes In Filtered Set access rights for the DNS application directory partitions. If you do not plan to add an RODC to the forest, you can disregard this error. If you plan to add an RODC to the forest, you must run adprep /rodcprep. For adprep /rodcprep, you can run the version of Adprep.exe that appears in either Windows Server 2008 or Windows Server 2008 R2 because that parameter performs the same set of operations in each version. For more information about running adprep /rodcprep, see Prepare a Forest for a Read-Only Domain Controller.
if your sysvol and netlogon are not shared then the DC is not advertizing itself as a suitable Domain Controller. Domain Controllers log 13516 event id in the FRS event logs indicating the DC has advertized itself as a Domain Controller. Can you check to see if 13516 is missing?

Since your getting RPC server unavailable, I am also guessing that replication/DNS issues
where are the DC pointing for Primary DNS and alternate DNS?
run repadmin/showrepl from command line , what do get in the result?

Regards,
Shahid
Avatar of Darius Ghassem
Here is the thing with Windows 2008 servers you need to always run dcdiag before moving any roles since Windows 2008 servers like to not create SYSVOL and netlogon. Also, they fail adversting as well.

Is you Windows 2003 server still a DC?

Make sure the DCs only point to internal DNS servers.

http://support.microsoft.com/kb/967336

http://support.microsoft.com/kb/947022/en-us

Avatar of sisaacso170

ASKER

I apologize in advance for hte huge post about to follow :)
Just as an FYI, I am running in 2003 server functional level

Post 1
This is not a read-only domain controller, so I assume I can ignore the first post.  I did look through the links though, appreciate those.  

Post 2
I am back to square one now, after a few reboots before posting this thread and enabling netbios, it seemed to act better.  Now it is tonce again taking forever to log machines into the domain, some servers I can't even log into with RDP (gives and RPC server unavailable error).  When i try to oepn active directory users and computers on the dc's I get the error;

Naming information cannot be located bedause: The RPC server is unavailable.  

I do not see the event id 13516 under administrative events.  I see the following error repeated every 6-7 minutes though;
The processing of Group Policy failed. Windows attempted to read the file \\loganutah.org\sysvol\loganutah.org\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.

Under the details view tab it shows;
+ System
  - Provider
   [ Name]  Microsoft-Windows-GroupPolicy
   [ Guid]  {AEA1B4FA-97D1-45F2-A64C-4D69FFFD92C9}
   EventID 1058
   Version 0  
   Level 2  
   Task 0  
   Opcode 1  
   Keywords 0x8000000000000000  
  - TimeCreated
   [ SystemTime]  2010-02-26T18:06:02.593296800Z  
   EventRecordID 4172  
  - Correlation
   [ ActivityID]  {5C62817F-6BD6-4D07-B181-9DCFDA140440}  
  - Execution
   [ ProcessID]  836
   [ ThreadID]  2400  
   Channel System  
   Computer DC1.loganutah.org  
  - Security
   [ UserID]  S-1-5-18  
- EventData
  SupportInfo1 4
  SupportInfo2 816
  ProcessingMode 0
  ProcessingTimeInMilliseconds 70000
  ErrorCode 67
  ErrorDescription The network name cannot be found.  
  DCName hte.loganutah.org
  GPOCNName CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=loganutah,DC=org
  FilePath \\loganutah.org\sysvol\loganutah.org\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini


There is also another error that repeats once an hour;
The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

Under Details it shows;
+ System
  - Provider
   [ Name]  Microsoft-Windows-DfsSvc
   [ Guid]  {7DA4FE0E-FD42-4708-9AA5-89B77A224885}
   [ EventSourceName]  DfsSvc
  - EventID 14550
   [ Qualifiers]  49152  
   Version 0  
   Level 2  
   Task 0  
   Opcode 0  
   Keywords 0x80000000000000  
  - TimeCreated
   [ SystemTime]  2010-02-26T17:10:32.000000000Z  
   EventRecordID 4162
   Correlation  
  - Execution
   [ ProcessID]  0
   [ ThreadID]  0  
   Channel System  
   Computer DC1.loganutah.org  
   Security
- EventData
   BA060000
--------------------------------------------------------------------------------
Binary data:
In Words
0000: 000006BA    
In Bytes
0000: BA 06 00 00               º...

Here is the output from the repadmin/showrepl;
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>repadmin/showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 1ff5bbcb-6229-4df8-8baf-4867bf6a1811
DSA invocationID: 1bfb3cf0-dff3-449c-8462-51cbed562ee2

==== INBOUND NEIGHBORS ======================================

DC=loganutah,DC=org
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: bbb1a11c-79ac-427a-b44b-3975cca1e7db
        Last attempt @ 2010-02-26 10:57:06 was successful.
    Default-First-Site-Name\HTE via RPC
        DSA object GUID: da098450-3269-428c-bf26-2492075e82a3
        Last attempt @ 2010-02-26 11:22:58 was successful.

CN=Configuration,DC=loganutah,DC=org
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: bbb1a11c-79ac-427a-b44b-3975cca1e7db
        Last attempt @ 2010-02-26 10:57:06 was successful.
    Default-First-Site-Name\HTE via RPC
        DSA object GUID: da098450-3269-428c-bf26-2492075e82a3
        Last attempt @ 2010-02-26 10:57:06 was successful.

CN=Schema,CN=Configuration,DC=loganutah,DC=org
    Default-First-Site-Name\HTE via RPC
        DSA object GUID: da098450-3269-428c-bf26-2492075e82a3
        Last attempt @ 2010-02-26 10:57:06 was successful.
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: bbb1a11c-79ac-427a-b44b-3975cca1e7db
        Last attempt @ 2010-02-26 10:57:07 was successful.

DC=DomainDnsZones,DC=loganutah,DC=org
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: bbb1a11c-79ac-427a-b44b-3975cca1e7db
        Last attempt @ 2010-02-26 10:57:07 was successful.
    Default-First-Site-Name\HTE via RPC
        DSA object GUID: da098450-3269-428c-bf26-2492075e82a3
        Last attempt @ 2010-02-26 10:57:07 was successful.

DC=ForestDnsZones,DC=loganutah,DC=org
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: bbb1a11c-79ac-427a-b44b-3975cca1e7db
        Last attempt @ 2010-02-26 10:57:07 was successful.
    Default-First-Site-Name\HTE via RPC
        DSA object GUID: da098450-3269-428c-bf26-2492075e82a3
        Last attempt @ 2010-02-26 10:57:07 was successful.

C:\Windows\system32>

I am pointing to itself (dc1) and dc2 as the primary and secondary dns.  It passed the repadmin, but when i run dcdiag /test:dns i get the following output;
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>dcdiag /test:dns

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = DC1
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\DC1
      Starting test: Connectivity
         ......................... DC1 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\DC1

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...
         ......................... DC1 passed test DNS

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : loganutah

   Running enterprise tests on : loganutah.org
      Starting test: DNS
         Test results for domain controllers:

            DC: DC1.loganutah.org
            Domain: loganutah.org


               TEST: Delegations (Del)
                  Error: DNS server: clusternode1.loganutah.org.
                  IP:<Unavailable> [Missing glue A record]
                  Error: DNS server: dc1.loganutah.org. IP:10.1.15.44
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dc2.loganutah.org. IP:10.1.15.45
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dns2.loganutah.org. IP:10.1.21.18
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: hte.loganutah.org. IP:10.1.15.17
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: lc2_dns_server.loganutah.org.
                  IP:<Unavailable> [Missing glue A record]
                  Error: DNS server: nw2.loganutah.org. IP:10.1.15.14
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: oes1.loganutah.org. IP:10.1.15.30
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: recserver.loganutah.org. IP:<Unavailable>
                  [Missing glue A record]
                  Error: DNS server: zenworks.loganutah.org. IP:10.1.15.20
                  [Broken delegated domain loganutah.org.loganutah.org.]

         Summary of test results for DNS servers used by the above domain
         controllers:

            DNS server: 10.1.15.14 (nw2.loganutah.org.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DN
S server 10.1.15.14
            DNS server: 10.1.15.17 (hte.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.20 (zenworks.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.30 (oes1.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.44 (dc1.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.45 (dc2.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.21.18 (dns2.loganutah.org.)
               1 test failure on this DNS server

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
            _________________________________________________________________
            Domain: loganutah.org
               DC1                          PASS PASS PASS FAIL PASS PASS n/a

         ......................... loganutah.org failed test DNS

C:\Windows\system32>
The only dns servers i care about are hte, dc1, and dc2.  The others will be going away once i get this resolved.

POST 3
I am beginning to think the sysvol and netlogon shares were not created, as you suggest.  Advertising is failing also, assuming because of no shares. The windows server 2003 machine is still a dc, it is the only one showing a sysvol and netlogon share.  I would like to demote it as soon as i get everything working right on the new servers. New dc's are pointing to themselves and each other for dns.
On your first link it specifies the resolution;
To resolve this problem choose one of the following options:

1.       Resolve any possible name resolution or network connectivity issue that would prevent communication with the defined "Parent Computer" --I can ping the other dc's by name, without even having to use FQDN, so i think this is not a problem?

2.       Modify the following registry to point to an available source domain controller:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\SysVols\Seeding SysVols\contoso.com ---This does not even exist in my registry, once i get to the parameters folder, there is no SysVols folder to choose (probably because sysvol is not shared out?)

Parent Computer"="DC1.contoso.com"

On your second link, there is a notation that says;
Note This article does not apply if both NETLOGON and SYSVOL shares are missing.

Since I am missing both shares, then this article does not pertain to me I believe.  Advise if otherwise.

POST 4
Great article, but this does seem to be for someone who did a restore form backup, so not quite the same as my problem.  Nonetheless, gave some very good command lines to try. The netdom query gave me;

C:\Windows\system32>netdom query /domain:loganutah.org fsmo
Schema master               DC1.loganutah.org
Domain naming master        DC1.loganutah.org
PDC                         DC1.loganutah.org
RID pool manager            DC1.loganutah.org
Infrastructure master       DC1.loganutah.org
The command completed successfully.
C:\Windows\system32>

From what i read, server03 uses frs, but 2008 uses dfs?  Assuming since the older dc is on server03, the newer server will need to use fsr to replicate information?  I believe the sysvol and netlogon not being shared out is the root of my problem.  I have created several single dc's before, this is my first time creating a multiple dc environment, so I am really struggling.

Any other links or guidance would be appreciated, thanks for any help.
ASKER CERTIFIED SOLUTION
Avatar of Darius Ghassem
Darius Ghassem
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Disable IPV6. Post ipconfig /all
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : DC1
   Primary Dns Suffix  . . . . . . . : loganutah.org
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : loganutah.org

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-50-56-A9-5B-ED
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.1.15.44(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.1.15.1
   DNS Servers . . . . . . . . . . . : 10.1.15.44
                                       10.1.15.45
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{BD2533B7-1D2B-442E-8621-C2F0FD838701}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

C:\Windows\system32>

There is only one adapter.  
So you are saying to follow the directions for adding the netlogon share, and i may get lucky and have the sysvol created as well.  

I don't know if i quite understand the following part of your post, please let me know if i have it right;

Now I would recommend moving all FSMO roles over a existing DC before promoting the new servers (I currently have two 2008 r2 servers i promoted to dc's, one of them has the fsmo roles, do u want me to move the fsmo roles back to the old dc on server 2003?). Demote the new servers run a metadata cleanup to remove any lingering objects from AD. Delete all DNS records for the failed DCs. (Could i just create another server 2008 r2 and try to move the fsmo roles to it?  I want to be rid of the server 2003 server as soon as i can).  Do you have a good link explaining metadata cleanup process?   This is kind of a last resort for me, hoping to fix the current problem before going through the process of demoting the new servers.  I will try the netlogon fix soon though, thanks.
what he means is that since your dc promotion didn't go well, you need to demote your failing DCs. but before that you first need to migrate the FSMO. demote the failing DCs. remove references from DNS.

you are now back to square one. Check your logon/authentication. Have the issues gone away?

Now introduce your ready to introduce the server. promote the first 2008 dc. let it replicate, run dcdiag and repadmin to check replication and dns. Once convince, promote the other server following the same.

When you make sure everythings working, then move FSMO to one of the 2008 servers. Then take the 2003 DC offline(don't demote yet) check if the things are working. Your logon authentication should work fine now. Now demote the 2003 DC and check everything again.

Hopefully things will work out. The key is let the DC advertised themselves as DC to provide logon and group policies and stuff. I have seen DCs taking a whole day to replicate the changes even if the you another DC is on the same subnet.

Hope that will make more sense.

Shahid.
My problem with doing that is all 400 computers in our domain use those two new 2008r2 servers as primary and secondary dns (i am in the process of moving off novell to AD, which is why I am not in full blown panic mode here since only a few machines are logging into the domain). That is why i was wondering if promoting a new server would be better.  I could then run the dcdiag and repadmin to make sure it was working.  If so, then transfer fsmo roles to it.  My thought was to create  a working dc1new and dc2new, and making them fully functional.   I could then just give them the ip of the current dc1 and dc2 and be good.  I could demote the bad dc's.  One thing I did do is within an hour of promoting and transferring fsmo roles to dc1, I shut it down to take a snapshot.  Don't know if that could have done anything if it wasn't done replicating.  My preference again would be to fix the two dc's, so I don't have to go through this whole process, but will do what needs to be done to get it working.  Thanks for the clarification.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
You can leave DNS on the servers DNS and AD are seperate services and allow you enough time to demote and get the steps done to promote again.

Or you can call MS pay them some money allow them to remotely access the environment to try to get the system healthly. Keyword is TRY.
i will transfer back to the old dc then and try to fix.  If i end up demoting the 08R2 servers, assume i willneed to run the metadata cleanup before running dcpromo again.  Will I need to change the server name before promoting again?  I read a couple of posts where users said there can be issues if you remove a dc and try to rejoin with the same name.  Perhaps they didn't do a metadata cleanup though, what are our thoughts on that?  Thanks for the suggestions, I will be trying them late tonight so I won't have any downtime for the current users.
Given proper time for replication, the state of the AD will be consistent. Metadata cleanup is not necessary if DC demote gracefully, however, doing cleanup will not bring any harm..
The issue you are referring to occurs, when users demote a DC and then promote way too quickly and this normally occurs in large AD environment spanning many sites have large replication latency.
I was looking forward to your comments on connection objects in AD sites and services and DNS Resource records.

Regards,
Shahid
One more note, the netlogon fix from http://support.microsoft.com/kb/947022/en-us specifies to change the registry value to a 0, then change back to 1. The value in the two new domain controllers already is 0, but on the old dc it has a 1.  Hoping when i change the value to a 1 it magically fixes everything, but there is no way it is that easy.  Will let you know what happens when i change that late tonight.  Just wondering if I should change the fsmo role back to the old dc before attempting any more fixes, can you advise?
Yes, please move all FSMO and services back to old DC>
Moved FSMO roles back to HTE (original domain controller).  Demoted dc2, and it went fine.  Ran metadata cleanup, but dc2 server was not even listed for me to chose.  I expected this because Server 2008 R2 is supposed to do this automatically.  I went into AD sites and services to check, DC2 was listed, but no NTDS settings were there (which I also expected).  I deleted dc2 from sites and services and figured everything should be okay to promote again.  The promo went fine, though a warning comes up about a dns delegation problem.  I wish I had written down the error, but when i googled it last week, my searches acted like it was to be expected and to just click next.  I guess I could do another demotion and promote again to get the error, hoping someone knows what I am talking about and can verify if it was okay to just continue.  After rebooting from the dcpromo, i do a net share and sysvol and netlogon are still not listed as a shared folder on dc2.  I am at a loss here, does anyone have any ideas?
Ok, should have demoted then repromoted with the same issue, right?

Did you delete all DNS records in DNS for the demoted DC?

The error would be nice to see what is actually means.

Make sure that the demoted DC is pointing to your primary DC for DNS only. Don't change this setting until your dcdiag fully passes.
Same issue, correct.  The only thing left in DNS on the PDC was the A Name where all the other ip entries are (within the loganutah.org folder).  It was not in the msdcs folder anymore.  Did I need to remove every reference?

Really irritated about hitting the next button before copying the error, guess I can demote and promote once more to get it.  I did have dc2 as the secondary dns, guess I can get rid of that and only have the PDC as the dns.
Well DC2 is fine I was really wanted to make sure that you didn't have the server pointing to itself for DNS.

Make sure to remove any AV that is installed. Also, disable all firewalls on server.
The warning is;
A delegation for this DNS server cannot be created becasue the authoritative parent zone cannot be found or it does not run Windows DNS server.  If you are integrating with an existing DNS infrastructure, you should manually create a delegation to this DNS server in the parent zone to ensure reliable name resolution from outside the domain "loganutah.org".  Otherwise, no action is required.  Do you want to continue?

I have hit yes the last two times, but we know how that has turned out.  Currently, HTE (the PDC) is running DNS and pointing to itself only for DNS.   lcdc2 (i created a new dc from scratch just for the heck of it) is pointing to HTE only as its DNS server.  lcdc is new, just joined to the domain, and in the middle of upgrading to a DC.  I am going to leave it at this screen and wait for any direction someone can provide.  I thought a new server would help rule out any old files not being cleaned up.  I ran a dcdiag /test:dns on the HTE (PDC) to see if it would pass, and it has a lot more errors which I am posting below.  I wonder if this has something to do with my problems.
Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.
C:\Documents and Settings\administrator.HTE>dcdiag /test:dns
Domain Controller Diagnosis
Performing initial setup:
   Done gathering initial info.
Doing initial required tests
   Testing server: Default-First-Site-Name\HTE
      Starting test: Connectivity
         ......................... HTE passed test Connectivity
Doing primary tests
   Testing server: Default-First-Site-Name\HTE

DNS Tests are running and not hung. Please wait a few minutes...
   Running partition tests on : ForestDnsZones
   Running partition tests on : DomainDnsZones
   Running partition tests on : Schema
   Running partition tests on : Configuration
   Running partition tests on : loganutah
   Running enterprise tests on : loganutah.org
      Starting test: DNS
         Test results for domain controllers:

            DC: hte.loganutah.org
            Domain: loganutah.org

               TEST: Forwarders/Root hints (Forw)
                  Error: Root hints list has invalid root hint server: a.root-se
rvers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-se
rvers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-se
rvers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-se
rvers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-se
rvers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-se
rvers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-se
rvers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-se
rvers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-se
rvers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-se
rvers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-se
rvers.net. (193.0.14.129)
                  Error: Root hints list has invalid root hint server: l.root-se
rvers.net. (199.7.83.42)
                  Error: Root hints list has invalid root hint server: m.root-se
rvers.net. (202.12.27.33)

               TEST: Delegations (Del)
                  Warning: DNS server: clusternode1.loganutah.org. IP: <Unavaila
ble> Failure:Missing glue A record
                  Error: DNS server: dc1.loganutah.org. IP:10.1.15.44 [Broken de
legated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dc2.loganutah.org. IP:10.1.15.45 [Broken de
legated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dns2.loganutah.org. IP:10.1.21.18 [Broken d
elegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: hte.loganutah.org. IP:10.1.15.17 [Broken de
legated domain loganutah.org.loganutah.org.]
                  Warning: DNS server: lc2_dns_server.loganutah.org. IP: <Unavai
lable> Failure:Missing glue A record
                  Error: DNS server: nw2.loganutah.org. IP:10.1.15.14 [Broken de
legated domain loganutah.org.loganutah.org.]
                  Error: DNS server: oes1.loganutah.org. IP:10.1.15.30 [Broken d
elegated domain loganutah.org.loganutah.org.]
                  Warning: DNS server: recserver.loganutah.org. IP: <Unavailable
> Failure:Missing glue A record
                  Error: DNS server: zenworks.loganutah.org. IP:10.1.15.20 [Brok
en delegated domain loganutah.org.loganutah.org.]

         Summary of test results for DNS servers used by the above domain contro
llers:

            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 202.12.27.33

            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 199.7.83.42

            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 198.41.0.4

            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 193.0.14.129

            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.58.128.30

            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.5.5.241

            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.36.148.17

            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.33.4.12

            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.228.79.201

            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.203.230.10

            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.112.36.4

            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.8.10.90

            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.63.2.53

            DNS server: 10.1.21.18 (dns2.loganutah.org.)
               1 test failure on this DNS server
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.21.18

            DNS server: 10.1.15.45 (dc2.loganutah.org.)
               1 test failure on this DNS server
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.45

            DNS server: 10.1.15.44 (dc1.loganutah.org.)
               1 test failure on this DNS server
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.44

            DNS server: 10.1.15.30 (oes1.loganutah.org.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 10.1.15.30
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.30

            DNS server: 10.1.15.20 (zenworks.loganutah.org.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 10.1.15.20
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.20

            DNS server: 10.1.15.17 (hte.loganutah.org.)
               1 test failure on this DNS server
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.17

            DNS server: 10.1.15.14 (nw2.loganutah.org.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 10.1.15.14
               Delegation is broken for the domain loganutah.org.loganutah.org.
on the DNS server 10.1.15.14

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: loganutah.org
               hte                          PASS PASS FAIL FAIL PASS PASS n/a

         ......................... loganutah.org failed test DNS

C:\Documents and Settings\administrator.HTE>

There are several DNS server listed of which only two were being used before i came here.  We are running on Novell here, trying to move to AD.  The primary DNS up to a week ago was teh novell server, and the secondary server was a Server 2008 machine running DNS only.  Before adding dc1 and dc2, I added DNS to HTE,   Everything seemed to be working okay, as when I added new pc's to the domain, they would populate in the dns entries on HTE.  HTE DNS seems to work fine, I can get anywhere or ping any pc I want.  

I decided to include the dcdiag /test:dns from the failed second dcpromo of dc2.  When I say failed, it actually completes, but during the process I have said yes to the DNS delegation warning both times.  Other than that, everything has went off without a hitch.
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\administrator.LOGANUTAH>dcdiag /test:dns

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = DC2
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\DC2
      Starting test: Connectivity
         ......................... DC2 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\DC2

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...
         ......................... DC2 passed test DNS

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : loganutah

   Running enterprise tests on : loganutah.org
      Starting test: DNS
         Test results for domain controllers:

            DC: DC2.loganutah.org
            Domain: loganutah.org


               TEST: Delegations (Del)
                  Error: DNS server: clusternode1.loganutah.org.
                  IP:<Unavailable> [Missing glue A record]
                  Error: DNS server: dc1.loganutah.org. IP:10.1.15.44
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dc2.loganutah.org. IP:10.1.15.45
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: dns2.loganutah.org. IP:10.1.21.18
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: hte.loganutah.org. IP:10.1.15.17
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: lc2_dns_server.loganutah.org.
                  IP:<Unavailable> [Missing glue A record]
                  Error: DNS server: nw2.loganutah.org. IP:10.1.15.14
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: oes1.loganutah.org. IP:10.1.15.30
                  [Broken delegated domain loganutah.org.loganutah.org.]
                  Error: DNS server: recserver.loganutah.org. IP:<Unavailable>
                  [Missing glue A record]
                  Error: DNS server: zenworks.loganutah.org. IP:10.1.15.20
                  [Broken delegated domain loganutah.org.loganutah.org.]

         Summary of test results for DNS servers used by the above domain
         controllers:

            DNS server: 10.1.15.14 (nw2.loganutah.org.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DN
S server 10.1.15.14
            DNS server: 10.1.15.17 (hte.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.20 (zenworks.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.30 (oes1.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.44 (dc1.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.15.45 (dc2.loganutah.org.)
               1 test failure on this DNS server

            DNS server: 10.1.21.18 (dns2.loganutah.org.)
               1 test failure on this DNS server

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
            _________________________________________________________________
            Domain: loganutah.org
               DC2                          PASS PASS PASS FAIL PASS PASS n/a

         ......................... loganutah.org failed test DNS

C:\Users\administrator.LOGANUTAH>

Thank you for your assistance, hopefully someone has an idea what is wrong.
Just for the heck of it, I decided to not continue, then unselected dns and moved forward.  It continued without any warning or errors, then I rebooted when it was done.  I figured I could always add DNS later, a post here on EE actually recommended doing it that way.
https://www.experts-exchange.com/questions/23473042/A-delegation-for-this-DNS-server-cannot-be-created-because-the-authoritative-parent-zone-cannot-be-found.html
Once i rebooted, I ran a net share, and the sysvol and netlogon folders are still not shared out.  I am completely bewildered.  Brand new server, with first time promotion to DC and no errors or warnings.  Here is the dcdiag from lcdc2 after I rebooted;

C:\Windows\system32>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = lcdc2
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\LCDC2
      Starting test: Connectivity
         ......................... LCDC2 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\LCDC2
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\hte.loganutah.org,
         when we were trying to reach LCDC2.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... LCDC2 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... LCDC2 passed test FrsEvent
      Starting test: DFSREvent
         ......................... LCDC2 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... LCDC2 passed test SysVolCheck
      Starting test: KccEvent
         A warning event occurred.  EventID: 0x00000266
            Time Generated: 03/01/2010   15:49:03
            Event String:
            NTDS (460) NTDSA: Database 'C:\Windows\NTDS\ntds.dit': The secondary
 index 'PDNT_index' of table 'datatable' may be corrupt. If there is no later ev
ent showing the index being rebuilt, then please defragment the database to rebu
ild the index.
         A warning event occurred.  EventID: 0x800005B7
            Time Generated: 03/01/2010   15:49:06
            Event String:
            Active Directory Domain Services has detected and deleted some possi
bly corrupted indices as part of initialization.
         A warning event occurred.  EventID: 0x80000B46
            Time Generated: 03/01/2010   15:51:08
            Event String:
            The security of this directory server can be significantly enhanced
by configuring the server to reject SASL (Negotiate,  Kerberos, NTLM, or Digest)
 LDAP binds that do not request signing (integrity verification) and LDAP simple
 binds that  are performed on a cleartext (non-SSL/TLS-encrypted) connection.  E
ven if no clients are using such binds, configuring the server to reject them wi
ll improve the security of this server.
         ......................... LCDC2 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... LCDC2 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... LCDC2 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... LCDC2 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\LCDC2\netlogon)
         [LCDC2] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... LCDC2 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... LCDC2 passed test ObjectsReplicated
      Starting test: Replications
         ......................... LCDC2 passed test Replications
      Starting test: RidManager
         ......................... LCDC2 passed test RidManager
      Starting test: Services
         ......................... LCDC2 passed test Services
      Starting test: SystemLog
         A warning event occurred.  EventID: 0x8000001D
            Time Generated: 03/01/2010   15:51:06
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate
 to use for smart card logons, or the KDC certificate could not be verified. Sma
rt card logon may not function correctly if this problem is not resolved. To cor
rect this problem, either verify the existing KDC certificate using certutil.exe
 or enroll for a new KDC certificate.
         A warning event occurred.  EventID: 0x000003F6
            Time Generated: 03/01/2010   15:51:12
            Event String:
            Name resolution for the name 10.in-addr.arpa timed out after none of
 the configured DNS servers responded.
         ......................... LCDC2 passed test SystemLog
      Starting test: VerifyReferences
         ......................... LCDC2 passed test VerifyReferences


   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : loganutah
      Starting test: CheckSDRefDom
         ......................... loganutah passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... loganutah passed test CrossRefValidation

   Running enterprise tests on : loganutah.org
      Starting test: LocatorCheck
         ......................... loganutah.org passed test LocatorCheck
      Starting test: Intersite
         ......................... loganutah.org passed test Intersite

C:\Windows\system32>
I don't think that link pertains to me, because I am running at 2003 domain functional level.  I have cleaned some items up though, and now all my servers pass the dcdiag /test:dns.   To help some poor soul out, i updated my support tools on the 2003 server, which cleaned up over half my errors.  There was also an entry in dns , that was causing delegation errors.  It kept saying;
Broken delegated domain loganutah.org.loganutah.org

Under the forward lookup zone there was a loganutah.org.  Farther down underneath that there was another entry that just said org, and it had loganutah in it also.  Deleted the org folder and all my dns errors went away on all domain controllers.

Moving on, so I did a demotion of dc2 and promoted again.  Of course it failed, I am beginning to think I am cursed and want to beat the previous Net Admin that left me this mess (I have only been here a couple of months).   I believe my problem now is FRS, as HTE (server 03 PDC) has the following FRS errors(event id 13506).

The File Replication Service failed a consistency check
  (QKey != QUADZERO)
in "QHashInsertLock:" at line 696.
 
The File Replication Service will restart automatically at a later time. If this problem persists a subsequent entry in this event log describes the recovery procedure.
 For more information about the automatic restart right click on My Computer and then click on Manage, System Tools, Services, File Replication Service, and Recovery.

For more information, see Help and Support Center at

immediately after i get an info notice in the event log that gives the following e (event id 13516);
The File Replication Service is no longer preventing the computer HTE from becoming a domain controller. The system volume has been successfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL.
 
Type "net share" to check for the SYSVOL share.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.



Then i get the following error (event id 13555)
The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:
 
 Recovery Steps:
 
 [1] The error state may clear itself if you stop and restart the FRS service. This can be done by performing the following in a command window:
 
    net stop ntfrs
    net start ntfrs
 
If this fails to clear up the problem then proceed as follows.
 
 [2] For Active Directory Domain Controllers that DO NOT host any DFS alternates or other replica sets with replication enabled:
 
If there is at least one other Domain Controller in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
 
If there are NO other Domain Controllers in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and choose the Advanced option which marks the sysvols as primary.
 
If there are other Domain Controllers in this domain but ALL of them have this event log message then restore one of them as primary (data files from primary will replicate everywhere) and the others as non-authoritative.
 
 
 [3] For Active Directory Domain Controllers that host DFS alternates or other replica sets with replication enabled:
 
 (3-a) If the Dfs alternates on this DC do not have any other replication partners then copy the data under that Dfs share to a safe location.
 (3-b) If this server is the only Active Directory Domain Controller for this domain then, before going to (3-c),  make sure this server does not have any inbound or outbound connections to other servers that were formerly Domain Controllers for this domain but are now off the net (and will never be coming back online) or have been fresh installed without being demoted. To delete connections use the Sites and Services snapin and look for
Sites->NAME_OF_SITE->Servers->NAME_OF_SERVER->NTDS Settings->CONNECTIONS.
 (3-c) Restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
 (3-d) Copy the data from step (3-a) above to the original location after the sysvol share is published.
 
 
 [4] For other Windows servers:
 
 (4-a)  If any of the DFS alternates or other replica sets hosted by this server do not have any other replication partners then copy the data under its share or replica tree root to a safe location.
 (4-b)  net stop ntfrs
 (4-c)  rd /s /q  c:\windows\ntfrs\jet
 (4-d)  net start ntfrs
 (4-e)  Copy the data from step (4-a) above to the original location after the service has initialized (5 minutes is a safe waiting time).
 
Note: If this error message is in the eventlog of all the members of a particular replica set then perform steps (4-a) and (4-e) above on only one of the members.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Next info notice just says FRS is stopping (event id 13502)


Next error is (event id 13505)
The File Replication Service has stopped after taking an assertion failure.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

I then run a net stop and net start on ntfrs from the command line to restart the service.  Which led to this next error (event id 13559)
The File Replication Service has detected that the replica root path has changed from "c:\windows\sysvol\domain" to "c:\windows\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path.
This was detected for the following replica set:
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
 
Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file.
 
 [1] At the first poll which will occur in 5 minutes this computer will be deleted from the replica set.
 [2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.

For more information, see Help and Support Center at

Everytime I do a stop and start I get the same error (Event ID 13559).  I looked back through the event log and this error has been there for months, when it was the only DC on the network(event id 13559).  I believe if I get this error fixed, it may resolve my problems.   When I do a dcdiag on HTE, I get a failed test for frsevent, but it is clean besides that.  I feel like I am close, and the reason the other DC's won't share out the sysvol and netlogn folders is because of the FRS problem on HTE (PDC).  I will continuing searching for the problem, but wanted to post anupdate and hopefully get some replies back on the frs problem.  Le me know if you think I am headed in the right direction.


My assestment tell me that you might have to go for "D2/D4" as mentioned in http://support.microsoft.com/kb/823230
This you will have to do in conjuction with dcgpofix. Beware this is often used as the last resort.
http://blogs.technet.com/janelewis/archive/2006/09/22/458132.aspx

How many group policies have you implemented? Are you running any custom GPs besides Default Domain Policy and Default Domain Controller Policy?

Do you have any reservations restoring DDP and DDCP to original state ( as they were when the first domain controller was setup in you domain)?

I am currently doing a similar activity for a client, however the FRS restore is being performed on 432 DCs spread across 390+ sites.

Regards,
Shahid
I am tempted to try this;

next error (event id 13559)
The File Replication Service has detected that the replica root path has changed from "c:\windows\sysvol\domain" to "c:\windows\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path.
This was detected for the following replica set:
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
 
Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file.
 
 [1] At the first poll which will occur in 5 minutes this computer will be deleted from the replica set.
 [2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.

My concern is that if I create the NTFRS_CMD_FILE_MOVE_ROOT file, what will happen to the current data underneath the domain folder?  This is the only functioning DC, the other two do not have their sysvol and netlogon share working, so I don't know how the data will be replicated in the only working DC (HTE).   Others have reported success using this method, will I lose anything if i run this with only one working DC?  Have looked at the D2/D4, hoping to not have to attempt that if at all possible.  THere are not any group policies that are implemented.  I don't think I have any reservations about restoring DDP and DDCP, though I would probably research a little more before doing so. Please let me kn owif you think creating the NTFRS_CMD_FILE_MOVE_ROOT file is a good idea, thanks.
Sorry for the late responce.
From what I understand the NTFRS_CMD_FILE_MOVE_ROOT trick works with 2000 DCs, not sure if they work with 2003 DCs. Truth be told, haven't tried it yet. My assessment after reading some of the post off the internet is that it does pretty much, what the D2 option does. In your case, you would be putting the empty file on the 2003 DC and since there are no other working DC, there won't be a healthy sysvol to replicate from.

As far as the issue of the effects of this activity, there is a chance of that sysvol get viped clean or maybe the sysvol policies and scripts would be moved in to the NTFRS_PreExisting Folder. Either way, I strongly recommend taking backup before proceding.
Also copy the settings in your gpos incase you might have to recreate them.
http://support.microsoft.com/kb/887440

Regards,
Shahid
It has been several days, wanted to wait to verify before posting.  The fix was to create the file NTFRS_CMD_FILE_MOVE_ROOT.  The whole problem behind my ordeal was that FRS was not functioning correctly on the original 2003 DC.  Once that was fixed, the sysvol and netlogon shares appeared almost immediately on the 2008 DC.  I spent way too much time looking at the 2008 DC at first.  My problem now is I don't know who to give credit fro the points, I never received a direct link for my problem (though several people were helpful).   Is there a moderator who can spread the points out?  Thanks for all the input from everyone!
Its a reward on its own knowing that you got it resolved. I understand that you will be migrating to a total Windows 2008 DC Environment. Move to DFS-R as soon as possible. My best wishes to you.  

Since I never tried NTFRS_CMD_FILE_MOVE_ROOT trick, I do have some questions, if you don't mind me asking.
What exactly happened when you placed this file and started the FRS? what exact events did you get?
Did it keep the current file infrastructure in sysvol intact?
Any thing particular that you thought was interesting?

Regards,
Shahid
Sorry for the delay, I was gone for a week.  After i created the file, it only took about five minutes for the following to show up in the event log;

All problems preventing updates to the Active Directory Database have been cleared. New updates to the Active Directory database are succeeding. The Net Logon service has restarted.

Event ID 1394 was what showed up in the event log.  I waited about one week before removing the old DC, things have been going well now.  How do i move to DFS-R?  Since I only have two domain controllers running 2008 R2, I assume this is much more preferred.  If you had a good link fo rhtis, I would appreciate thanks.

I still do not know who to give credit for the points, maybe split them up?  That's the only fair thing I can think of.
It nice to know that you problems were resolved. As for you request for the link, here is a good place to start.
http://blogs.technet.com/askds/archive/2009/05/01/sysvol-migration-from-frs-to-dfsr-whitepaper-released.aspx
 
As for the credit part, thats always upto the author.

Best Regards,
Shahid
Question was not fully answered by the posts, but they definitely pointed me in the right direction.  Since two users consistently posted, I am splitting the points between them, appreciate all their help.