Major Domain Controller Problems

to: rbarnhardt,
  Well the primary DNS on the 08 machines was itself (not using the loopback address), but I changed it and tried again with no luck.  I get the message:  

Managing the network session with VMDC10.osdb.oh.gov failed
"this network connection does not exist"

VMDC10 is the 2003 Server.  

I should state that all three of these machines were built on VMWare ESXi 3.5 platforms

Are you running active directory integrated DNS.

If you can't demote gracefully then you will want to use a metadata cleanup

From   http://blogs.technet.com/askds/archive/2009/06/05/dc-s-and-vm-s-avoiding-the-do-over.aspx

1) Forcefully demote the DC by running dcpromo /forceremoval. This will remove AD from the server without attempting to replicate any changes off. Once it is done and you reboot the server and it will be a standalone serve in a workgroup.
2) Run a metadata cleanup of the DC that was demoted per KB article 216498 on one of the replication partners.
3) If the demoted server held any of the FSMO (Flexible Single Master Operations) roles then use the KB article 255504 to seize the roles to another DC.
4) Once replication has occurred end to end in your environment you can rejoin the demoted server back to the domain then promote to a DC

For step 2 another good article is here http://www.petri.co.il/delete_failed_dcs_from_ad.htm

For others that come across this question.  In 2008 you can use the ADUC GUI for metadata cleanup   http://technet.microsoft.com/en-us/library/cc816907(WS.10).aspx ...but in this case it sounds like the 2003 box will be up.

You could also try to figure out why the domain is having issues, but if you want to rebuild the other two DCs and can't demote gracefully then that is why I've included the steps.

Thanks

Mike

can you run dcdiag and netdiag on the 2003 server and post the results.
Also run dcdiag on both 2008 servers and post the resuts.

Are the servers all Global Catalogue servers?  If your DNS has not been configured for IPv6 then disable this on both the WIndows 2008 servers, this can often cause problems.

Set the Windows 2003 server as the primary DNS server on all the servers, what I would suggest that for the time being you uninstall DNS from the 2 2008 servers, we can then get them all working properly and re-install DNS if required.

Well I have three zones that are in the DNS and one is the Standard Primary and the other two yes are Active Directory-Integrated Primary.

I can't run dcdiag or netdiag on teh 03 server, right now, I have to find the CD to run them, but as soon as I get a min I will  run dcdiag on both 08 servers and post.  You want them with the /v /e /c switches or just a basic dcdiag?  Which will help more?

And I've seen the metadata clean up, but in all honesty I was hoping to make this a graceful process.  I hae a few hundred users (I work at a school) and I'd like to do this with the least amount of interruption as possible.  Look for the logs here in a few mins.

Thanks!

just basic dcdiag we can do more again later if needed.
You can download the windows 2003 support tools from here: http://www.microsoft.com/downloads/details.aspx?FamilyId=6EC50B78-8BE1-4E81-B3BE-4E7AC4F0912D&displaylang=en

Don't do a cleanup just yet this is probably recoverable.

Ok.  Here are 3 long DCDIAG Tests:

*** Here's the first one ****

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = VDC30
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Connectivity
         ......................... VDC30 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov,
         when we were trying to reach VDC30.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... VDC30 failed test Advertising
      Starting test: FrsEvent
         ......................... VDC30 passed test FrsEvent
      Starting test: DFSREvent
         ......................... VDC30 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... VDC30 passed test SysVolCheck
      Starting test: KccEvent
         ......................... VDC30 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... VDC30 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... VDC30 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... VDC30 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC30\netlogon)
         [VDC30] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... VDC30 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... VDC30 passed test ObjectsReplicated
      Starting test: Replications
         [Replications Check,VDC30] DsReplicaGetInfo(PENDING_OPS, NULL) failed,
         error 0x2105 "Replication access was denied."
         ......................... VDC30 failed test Replications
      Starting test: RidManager
         ......................... VDC30 passed test RidManager
      Starting test: Services
            Could not open NTDS Service on VDC30, error 0x5 "Access is denied."
         ......................... VDC30 failed test Services
      Starting test: SystemLog
         An Warning Event occurred.  EventID: 0x000003FC
            Time Generated: 09/08/2009   14:54:56
            Event String:
            Scope, 10.0.0.0, is 98 percent full with only 15 IP addresses remaining.
         ......................... VDC30 passed test SystemLog
      Starting test: VerifyReferences
         Some objects relating to the DC VDC30 have problems:
            [1] Problem: Missing Expected Value
             Base Object:
            CN=NTDS Settings,CN=VDC30,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DSA Object"
             Value Object Attribute Name: serverReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
         ......................... VDC30 failed test VerifyReferences
   
   
   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation
   
   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation
   
   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
   
   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
   
   Running partition tests on : osdb
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: LocatorCheck
         ......................... osdb.oh.gov passed test LocatorCheck
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite


**** Here's another one****

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = VDC20
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VDC20
      Starting test: Connectivity
         ......................... VDC20 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VDC20
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov,
         when we were trying to reach VDC20.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... VDC20 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... VDC20 passed test FrsEvent
      Starting test: DFSREvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... VDC20 failed test DFSREvent
      Starting test: SysVolCheck
         ......................... VDC20 passed test SysVolCheck
      Starting test: KccEvent
         ......................... VDC20 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... VDC20 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... VDC20 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... VDC20 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC20\netlogon)
         [VDC20] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... VDC20 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... VDC20 passed test ObjectsReplicated
      Starting test: Replications
         [Replications Check,VDC20] DsReplicaGetInfo(PENDING_OPS, NULL) failed,
         error 0x2105 "Replication access was denied."
         ......................... VDC20 failed test Replications
      Starting test: RidManager
         ......................... VDC20 passed test RidManager
      Starting test: Services
            Could not open NTDS Service on VDC20, error 0x5 "Access is denied."
         ......................... VDC20 failed test Services
      Starting test: SystemLog
         An Warning Event occurred.  EventID: 0x8000001D
            Time Generated: 09/08/2009   14:42:13
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons, or the KDC certificate could not be verified. Smart card logon may not function correctly if this problem is not resolved. To correct this problem, either verify the existing KDC certificate using certutil.exe or enroll for a new KDC certificate.
         An Warning Event occurred.  EventID: 0x8000001D
            Time Generated: 09/08/2009   14:49:19
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons, or the KDC certificate could not be verified. Smart card logon may not function correctly if this problem is not resolved. To correct this problem, either verify the existing KDC certificate using certutil.exe or enroll for a new KDC certificate.
         ......................... VDC20 passed test SystemLog
      Starting test: VerifyReferences
         Some objects relating to the DC VDC20 have problems:
            [1] Problem: Missing Expected Value
             Base Object:
            CN=NTDS Settings,CN=VDC20,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DSA Object"
             Value Object Attribute Name: serverReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
         ......................... VDC20 failed test VerifyReferences
   
   
   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation
   
   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation
   
   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
   
   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
   
   Running partition tests on : osdb
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: LocatorCheck
         ......................... osdb.oh.gov passed test LocatorCheck
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite

*** Here' the last one***

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Connectivity
         ......................... VMDC10 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Replications
         ......................... VMDC10 passed test Replications
      Starting test: NCSecDesc
         ......................... VMDC10 passed test NCSecDesc
      Starting test: NetLogons
         ......................... VMDC10 passed test NetLogons
      Starting test: Advertising
         ......................... VMDC10 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VMDC10 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VMDC10 passed test RidManager
      Starting test: MachineAccount
         ......................... VMDC10 passed test MachineAccount
      Starting test: Services
         ......................... VMDC10 passed test Services
      Starting test: ObjectsReplicated
         ......................... VMDC10 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VMDC10 passed test frssysvol
      Starting test: frsevent
         ......................... VMDC10 passed test frsevent
      Starting test: kccevent
         ......................... VMDC10 passed test kccevent
      Starting test: systemlog
         ......................... VMDC10 passed test systemlog
      Starting test: VerifyReferences
         Some objects relating to the DC VMDC10 have problems:
            [1] Problem: Missing Expected Value

             Base Object: CN=VMDC10,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov

             Base Object Description: "DC Account Object"

             Value Object Attribute Name: frsComputerReferenceBL

             Value Object Description: "SYSVOL FRS Member Object"

             Recommended Action: See Knowledge Base Article: Q312862

             
            [1] Problem: Missing Expected Value

             Base Object:

            CN=NTDS Settings,CN=VMDC10,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov

             Base Object Description: "DSA Object"

             Value Object Attribute Name: serverReferenceBL

             Value Object Description: "SYSVOL FRS Member Object"

             Recommended Action: See Knowledge Base Article: Q312862

             
         ......................... VMDC10 failed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck




I will get the netdiag done here in a bit.  ( I actually coach and have practice right now )

Have a look here : http://technet.microsoft.com/en-us/library/cc734096(WS.10).aspx

For the smart card error on VDC20

Here are the three netdiag logs.  Yes all of the DC's are actually virtual machines (again, not my setup).  My goal is to upgrade everything to a server 2008 Active Directory environment, but as you can tell I have a lot of work.  I think there is even information from OLD DC's hanging out in the system.  

NetDiag for Server 2003 Machine.  

    Computer Name: VMDC10
    DNS Host Name: VMDC10.osdb.oh.gov
    System info : Microsoft Windows Server 2003 R2 (Build 3790)
    Processor : x86 Family 15 Model 6 Stepping 8, GenuineIntel
    List of installed hotfixes :
        KB921503
        KB923561
        KB924667-v2
        KB925398_WMP64
        KB925876
        KB925902
        KB926122
        KB926139-v2
        KB927891
        KB929123
        KB930178
        KB931768
        KB931784
        KB931836
        KB932168
        KB933360
        KB933566-IE7
        KB933729
        KB933854
        KB935839
        KB935840
        KB935966
        KB936021
        KB936357
        KB936782
        KB937143-IE7
        KB938127-IE7
        KB938464
        KB939653-IE7
        KB941202
        KB941568
        KB941569
        KB941644
        KB941672
        KB941693
        KB942615-IE7
        KB942763
        KB942830
        KB942831
        KB943055
        KB943460
        KB943484
        KB943485
        KB943729
        KB944533-IE7
        KB944653
        KB945553
        KB946026
        KB948496
        KB948590
        KB948745
        KB949014
        KB950759-IE7
        KB950760
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951746
        KB951748
        KB952004
        KB952069
        KB952954
        KB953838-IE7
        KB953839
        KB954211
        KB954550-v5
        KB954600
        KB955069
        KB955839
        KB956390-IE7
        KB956391
        KB956572
        KB956744
        KB956802
        KB956803
        KB956841
        KB957095
        KB957097
        KB958644
        KB958687
        KB958690
        KB959426
        KB960225
        KB960715
        KB960803
        KB960859
        KB961063
        KB961064
        KB961118
        KB961260-IE7
        KB961371
        KB961371-v2
        KB961373
        KB961501
        KB963027-IE7
        KB967715
        KB968389
        KB968537
        KB969805
        KB969883
        KB969897-IE7
        KB969898
        KB970238
        KB970483
        KB970653-v3
        KB971032
        KB971557
        KB971633
        KB971657
        KB972260-IE7
        KB973346
        KB973354
        KB973507
        KB973540
        KB973815
        KB973869
        Q147222


Netcard queries test . . . . . . . : Passed



Per interface results:

    Adapter : Local Area Connection

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VMDC10
        IP Address . . . . . . . . : 10.0.1.10
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 127.0.0.1
                                     10.0.1.20


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names is missing.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Passed


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Passed
    PASS - All the DNS entries for DC are registered on DNS server '127.0.0.1' and other DCs also have some of the names registered.
    PASS - All the DNS entries for DC are registered on DNS server '10.0.1.20' and other DCs also have some of the names registered.


Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The browser is bound to 1 NetBt transport.


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Skipped


Kerberos test. . . . . . . . . . . : Passed


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

****NetDiag for 2008 Server 1 ****



    Computer Name: VDC20
    DNS Host Name: VDC20.osdb.oh.gov
    System info : Windows Server (R) 2008 Enterprise (Build 6001)
    Processor : Intel64 Family 15 Model 6 Stepping 8, GenuineIntel
    Hotfixes : none detected


Netcard queries test . . . . . . . : Passed
    [WARNING] The net card 'RAS Async Adapter' may not be working because it has not received any packets.
    [WARNING] The net card 'Teredo Tunneling Pseudo-Interface' may not be working.
    GetStats failed for 'isatap.{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}'. [ERROR_GEN_FAILURE]



Per interface results:

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VDC20
        IP Address . . . . . . . . : 10.0.1.20
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 10.0.1.10
                                     10.0.1.20
                                     10.0.1.30


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
            No names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Failed
    [WARNING] Ths system volume has not been completely replicated to the local machine. This machine is not working properly as a DC.


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Failed
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] No DNS servers have the DNS records for this DC registered.


Redir and Browser test . . . . . . : Failed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    The browser is bound to 1 NetBt transport.
    [FATAL] Cannot send mailslot message to 'NETWORK*' via browser. [ERROR_INVALID_FUNCTION]


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Passed
    Secure channel for domain 'NETWORK' is to '\\VMDC10.osdb.oh.gov'.


Kerberos test. . . . . . . . . . . : Failed
    [FATAL] Cannot lookup package Kerberos.
    The error occurred was: (null)


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

***** Netdiag 2008 Server 2 ****




    Computer Name: VDC30
    DNS Host Name: VDC30.osdb.oh.gov
    System info : Windows Server (R) 2008 Enterprise (Build 6001)
    Processor : Intel64 Family 15 Model 6 Stepping 8, GenuineIntel
    Hotfixes : none detected


Netcard queries test . . . . . . . : Passed
    [WARNING] The net card 'RAS Async Adapter' may not be working because it has not received any packets.
    [WARNING] The net card 'Teredo Tunneling Pseudo-Interface' may not be working.
    GetStats failed for 'isatap.{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}'. [ERROR_GEN_FAILURE]



Per interface results:

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VDC30
        IP Address . . . . . . . . : 10.0.1.30
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 127.0.0.1


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
            No names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Failed
    [WARNING] Ths system volume has not been completely replicated to the local machine. This machine is not working properly as a DC.


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Failed
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] No DNS servers have the DNS records for this DC registered.


Redir and Browser test . . . . . . : Failed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    The browser is bound to 1 NetBt transport.
    [FATAL] Cannot send mailslot message to 'NETWORK*' via browser. [ERROR_INVALID_FUNCTION]


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Passed
    Secure channel for domain 'NETWORK' is to '\\VMDC10.osdb.oh.gov'.


Kerberos test. . . . . . . . . . . : Failed
    [FATAL] Cannot lookup package Kerberos.
    The error occurred was: (null)


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

can you confirm that all 3 domain controllers can ping each other via IP address?
Can they all also ping the gateway?

Can you set all domain controllers to just use the WIndows 2003 DNS server for DNS and make sure all 3 servers have the actual IP address of the DNS server and not the loopback address (127.0.0.1)

Yes they can all ping each other by IP and actually can ping by name.  I will change all the server DNS setups tomorrow morning.

Once you have changed the DNS configuration restart the netlogon service on each Domain Controller so it registers its DNS records.

Give this 10 minutes or so then run the netdiag commands again and post the results.

Ok.  Here you go.  3 more netdiag's again.. I have not had time yet to try and recover some of the missing FRS objects.  I saw those late last week, but haven't attempted to actually recover them yet.

Server 2003 Log:

    Computer Name: VMDC10
    DNS Host Name: VMDC10.osdb.oh.gov
    System info : Microsoft Windows Server 2003 R2 (Build 3790)
    Processor : x86 Family 15 Model 6 Stepping 8, GenuineIntel
    List of installed hotfixes :
        KB921503
        KB923561
        KB924667-v2
        KB925398_WMP64
        KB925876
        KB925902
        KB926122
        KB926139-v2
        KB927891
        KB929123
        KB930178
        KB931768
        KB931784
        KB931836
        KB932168
        KB933360
        KB933566-IE7
        KB933729
        KB933854
        KB935839
        KB935840
        KB935966
        KB936021
        KB936357
        KB936782
        KB937143-IE7
        KB938127-IE7
        KB938464
        KB939653-IE7
        KB941202
        KB941568
        KB941569
        KB941644
        KB941672
        KB941693
        KB942615-IE7
        KB942763
        KB942830
        KB942831
        KB943055
        KB943460
        KB943484
        KB943485
        KB943729
        KB944533-IE7
        KB944653
        KB945553
        KB946026
        KB948496
        KB948590
        KB948745
        KB949014
        KB950759-IE7
        KB950760
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951746
        KB951748
        KB952004
        KB952069
        KB952954
        KB953838-IE7
        KB953839
        KB954211
        KB954550-v5
        KB954600
        KB955069
        KB955839
        KB956390-IE7
        KB956391
        KB956572
        KB956744
        KB956802
        KB956803
        KB956841
        KB957095
        KB957097
        KB958644
        KB958687
        KB958690
        KB959426
        KB960225
        KB960715
        KB960803
        KB960859
        KB961063
        KB961064
        KB961118
        KB961260-IE7
        KB961371
        KB961371-v2
        KB961373
        KB961501
        KB963027-IE7
        KB967715
        KB968389
        KB968537
        KB969805
        KB969883
        KB969897-IE7
        KB969898
        KB970238
        KB970483
        KB970653-v3
        KB971032
        KB971557
        KB971633
        KB971657
        KB972260-IE7
        KB973346
        KB973354
        KB973507
        KB973540
        KB973815
        KB973869
        Q147222


Netcard queries test . . . . . . . : Passed



Per interface results:

    Adapter : Local Area Connection

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VMDC10
        IP Address . . . . . . . . : 10.0.1.10
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 10.0.1.10
                                     10.0.1.20


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names is missing.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Passed


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Passed
    PASS - All the DNS entries for DC are registered on DNS server '10.0.1.10' and other DCs also have some of the names registered.
    PASS - All the DNS entries for DC are registered on DNS server '10.0.1.20' and other DCs also have some of the names registered.


Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The browser is bound to 1 NetBt transport.


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Skipped


Kerberos test. . . . . . . . . . . : Passed


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

**** First Server 2008 Log *****



    Computer Name: VDC20
    DNS Host Name: VDC20.osdb.oh.gov
    System info : Windows Server (R) 2008 Enterprise (Build 6001)
    Processor : Intel64 Family 15 Model 6 Stepping 8, GenuineIntel
    Hotfixes : none detected


Netcard queries test . . . . . . . : Passed
    [WARNING] The net card 'RAS Async Adapter' may not be working because it has not received any packets.
    [WARNING] The net card 'Teredo Tunneling Pseudo-Interface' may not be working.
    GetStats failed for 'isatap.{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}'. [ERROR_GEN_FAILURE]



Per interface results:

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VDC20
        IP Address . . . . . . . . : 10.0.1.20
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 10.0.1.10
                                     10.0.1.20
                                     10.0.1.30


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
            No names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Failed
    [WARNING] Ths system volume has not been completely replicated to the local machine. This machine is not working properly as a DC.


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Failed
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] No DNS servers have the DNS records for this DC registered.


Redir and Browser test . . . . . . : Failed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{4C1FF1EF-D4A0-45C9-BD16-046CF89FC11E}
    The browser is bound to 1 NetBt transport.
    [FATAL] Cannot send mailslot message to 'NETWORK*' via browser. [ERROR_INVALID_FUNCTION]


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Passed
    Secure channel for domain 'NETWORK' is to '\\VMDC10.osdb.oh.gov'.


Kerberos test. . . . . . . . . . . : Failed
    [FATAL] Cannot lookup package Kerberos.
    The error occurred was: (null)


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

**** Second Server 2008 Log *******

    Computer Name: VDC30
    DNS Host Name: VDC30.osdb.oh.gov
    System info : Windows Server (R) 2008 Enterprise (Build 6001)
    Processor : Intel64 Family 15 Model 6 Stepping 8, GenuineIntel
    Hotfixes : none detected


Netcard queries test . . . . . . . : Passed
    [WARNING] The net card 'RAS Async Adapter' may not be working because it has not received any packets.
    [WARNING] The net card 'Teredo Tunneling Pseudo-Interface' may not be working.
    GetStats failed for 'isatap.{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}'. [ERROR_GEN_FAILURE]



Per interface results:

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VDC30
        IP Address . . . . . . . . : 10.0.1.30
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.0.1
        Dns Servers. . . . . . . . : 10.0.1.10
                                     10.0.1.30


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
            No names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Failed
    [WARNING] Ths system volume has not been completely replicated to the local machine. This machine is not working properly as a DC.


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Failed
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] Could not open file C:\Windows\system32\config\netlogon.dns for reading.
    [FATAL] No DNS servers have the DNS records for this DC registered.


Redir and Browser test . . . . . . : Failed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{04242F71-D5F4-43D3-BFC0-9E9EDB4D6B79}
    The browser is bound to 1 NetBt transport.
    [FATAL] Cannot send mailslot message to 'NETWORK*' via browser. [ERROR_INVALID_FUNCTION]


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Passed
    Secure channel for domain 'NETWORK' is to '\\VMDC10.osdb.oh.gov'.


Kerberos test. . . . . . . . . . . : Failed
    [FATAL] Cannot lookup package Kerberos.
    The error occurred was: (null)


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

I just noticed in the logs that the DNS records didn't register.  I'll have to wait some more time and see if that resolves itself again, or if we (as I suspected) have some DNS problems going on.

All the logs are complaining they cannot reach the default gateway, this is on a different subnet to the IP addresses, are you definately able to ping 10.0.0.1

I would also suggest that at the moment you take out all other DNS server entries and just use the one server (preferably the 2003 server)

I believe the address 10.0.0.1 is actually a router.  The tech that setup all the DC's, also setup the network.  He's still working with us, but now I've taken over more control of the DC's and he's going to concentrate on other tasks.

If I take out all the  DNS enteries how will that effect my users who are looking to the two 08 servers ( .30 and .20) as their DNS servers?  Should I change all my DHCP settings to all point to the 03 server?

can you ping that router?
This will be your gateway.

It might be worth chaning the DHCP scope to just use the 2003 DNS (if this is the one we are sure works?) this way if you need to remove DNS from the two 08 servers it will not effect your users.

You know honestly I haven't tried.  I actually tapped in and our users are being sent to a 10.0.1.1 gateway.  I think he setup the servers directly to the router itself.  I'll have to check more when I get in the office tomorrow.  When I VPN in, the settings are much different and I can't even find my own computer by name!  Ha!!!!  Now you understand my troubles.  One person set everything up, and we assumed it was working and as you can see it's not at all - Not the way it was intended.

The gateway address is used to gain access to another network this could be a VLAN, another subnet or the internet.

The chances are if the clients are using 10.0.1.1 and it is working then the servers should also be configured for this, especially as it's in the same subnet as the servers.  Try changing them to this and then restart the netlogon services again.

Now that I've switched them all another issue has popped up.  The person that setup the domain (osdb.oh.gov) also set another name as NETWORK...here's one of the logs to reflect what has happened after I changed everything to reflect the same gateway as what the users are seeing.::

Server 2003 Log


    Computer Name: VMDC10
    DNS Host Name: VMDC10.osdb.oh.gov
    System info : Microsoft Windows Server 2003 R2 (Build 3790)
    Processor : x86 Family 15 Model 6 Stepping 8, GenuineIntel
    List of installed hotfixes :
        KB921503
        KB923561
        KB924667-v2
        KB925398_WMP64
        KB925876
        KB925902
        KB926122
        KB926139-v2
        KB927891
        KB929123
        KB930178
        KB931768
        KB931784
        KB931836
        KB932168
        KB933360
        KB933566-IE7
        KB933729
        KB933854
        KB935839
        KB935840
        KB935966
        KB936021
        KB936357
        KB936782
        KB937143-IE7
        KB938127-IE7
        KB938464
        KB939653-IE7
        KB941202
        KB941568
        KB941569
        KB941644
        KB941672
        KB941693
        KB942615-IE7
        KB942763
        KB942830
        KB942831
        KB943055
        KB943460
        KB943484
        KB943485
        KB943729
        KB944533-IE7
        KB944653
        KB945553
        KB946026
        KB948496
        KB948590
        KB948745
        KB949014
        KB950759-IE7
        KB950760
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951746
        KB951748
        KB952004
        KB952069
        KB952954
        KB953838-IE7
        KB953839
        KB954211
        KB954550-v5
        KB954600
        KB955069
        KB955839
        KB956390-IE7
        KB956391
        KB956572
        KB956744
        KB956802
        KB956803
        KB956841
        KB957095
        KB957097
        KB958644
        KB958687
        KB958690
        KB959426
        KB960225
        KB960715
        KB960803
        KB960859
        KB961063
        KB961064
        KB961118
        KB961260-IE7
        KB961371
        KB961371-v2
        KB961373
        KB961501
        KB963027-IE7
        KB967715
        KB968389
        KB968537
        KB969805
        KB969883
        KB969897-IE7
        KB969898
        KB970238
        KB970483
        KB970653-v3
        KB971032
        KB971557
        KB971633
        KB971657
        KB972260-IE7
        KB973346
        KB973354
        KB973507
        KB973540
        KB973815
        KB973869
        Q147222


Netcard queries test . . . . . . . : Passed



Per interface results:

    Adapter : Local Area Connection

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : VMDC10
        IP Address . . . . . . . . : 10.0.1.10
        Subnet Mask. . . . . . . . : 255.255.240.0
        Default Gateway. . . . . . : 10.0.1.1
        Dns Servers. . . . . . . . : 10.0.1.10
                                     10.0.1.20


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messen
r Service', <20> 'WINS' names is missing.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Passed


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Ser
ce', <03> 'Messenger Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Passed
    PASS - All the DNS entries for DC are registered on DNS server '10.0.1.10'
nd other DCs also have some of the names registered.
    PASS - All the DNS entries for DC are registered on DNS server '10.0.1.20'
nd other DCs also have some of the names registered.


Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{8FEA3DF5-B7DD-4057-97E5-1FF880615006}
    The browser is bound to 1 NetBt transport.


DC discovery test. . . . . . . . . : Failed
        [FATAL] Cannot find DC in domain 'NETWORK'. [ERROR_NO_SUCH_DOMAIN]


DC list test . . . . . . . . . . . : Failed
        'NETWORK': Cannot find DC to get DC list from [test skipped].


Trust relationship test. . . . . . : Skipped


Kerberos test. . . . . . . . . . . : Skipped
        'NETWORK': Cannot find DC to get DC list from [test skipped].


LDAP test. . . . . . . . . . . . . : Failed
    Cannot find DC to run LDAP tests on. The error occurred was: The specified
omain either does not exist or could not be contacted.

        [WARNING] Cannot find DC in domain 'NETWORK'. [ERROR_NO_SUCH_DOMAIN]


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

OK so what is NETWORK?
Is this the domain the DC's are in?
Does NETWORK exist?
They still have other servers in as their DNS entries, can you set them all to the Windows 2003 server for PRIMARY DNS and then restart the netlogon service.

Once we are happy that they are working with DNS on here we can troubleshoot the other configurations.

There also seems to be something else going on with your IP configuration.
Can you run ipconfig /all and post the results

NETWORK was another name given to the domain for our users to see when they sign on.  Again I didn't set that up.  NETWORK does exist it's just a more simple name for osdb.oh.gov which is the FQDN.

Now I set everything back to Gateway of 10.0.0.1 (partially because I had a problem that was unrelated, but I wanted to make sure).

All of the ipconfig's are below.  I haven't changed the gateway back just yet, however all the DNS points only to the Server 2003 machine.

Server 08 #1

Windows IP Configuration

   Host Name . . . . . . . . . . . . : VDC20
   Primary Dns Suffix  . . . . . . . : osdb.oh.gov
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : osdb.oh.gov
                                       oh.gov

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
2
   Physical Address. . . . . . . . . : 00-0C-29-5F-1F-EF
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.0.1.20(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 10.0.0.1
   DNS Servers . . . . . . . . . . . : 10.0.1.10
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{4C1FF1EF-D4A0-45C9-BD16-046CF89F
11E}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 9:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 02-00-54-55-4E-01
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Server 2008 #2

Windows IP Configuration

   Host Name . . . . . . . . . . . . : VDC30
   Primary Dns Suffix  . . . . . . . : osdb.oh.gov
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : osdb.oh.gov
                                       oh.gov

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-0C-29-BC-B8-F7
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.0.1.30(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 10.0.0.1
   DNS Servers . . . . . . . . . . . : 10.0.1.10
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{04242F71-D5F4-43D3-BFC0-9E9EDB4D6
B79}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 9:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 02-00-54-55-4E-01
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Server 2003 Machine:

Windows IP Configuration

   Host Name . . . . . . . . . . . . : VMDC10
   Primary Dns Suffix  . . . . . . . : osdb.oh.gov
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : osdb.oh.gov
                                       oh.gov

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : VMware Accelerated AMD PCNet Adapter
   Physical Address. . . . . . . . . : 00-0C-29-45-FE-E8
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 10.0.1.10
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 10.0.0.1
   DNS Servers . . . . . . . . . . . : 10.0.1.10

Now I can change all the gateway's back because I think I solved the other problem.  I don't think they were related but that was the last thing changed and some network resources were screwed up (mapped network drives)

Also interesting enough, I can't ping 10.0.0.1 OR 10.0.1.1 which is the gateway setup for the servers AND the workstations.  I wonder what's up with that.

it may well be set to reject ping requests.
OK, so they are all now set to use the Windows 2003 DNS for DNS, can you under TCP/IP properties and the advanced tab enter the DNS suffix (osdb.oh.giv) in the "DNS Suffix for this connection" and check the box for "Use this connection's DNS suffix in DNS registration"

restart the netlogon services and check the DNS server to see if the servers have registered there SRV records under _msdcs folder

Ok.  I've set the TCP/IP as you wanted, and all have a CNAME and NS record in _msdcs folder on the DNS servers.  

OK, so now run DCDIAG?

Ok.  Here are the new dcdiag files:

Server 2003 New DCDIAG:
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Connectivity
         ......................... VMDC10 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Replications
         ......................... VMDC10 passed test Replications
      Starting test: NCSecDesc
         ......................... VMDC10 passed test NCSecDesc
      Starting test: NetLogons
         ......................... VMDC10 passed test NetLogons
      Starting test: Advertising
         ......................... VMDC10 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VMDC10 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VMDC10 passed test RidManager
      Starting test: MachineAccount
         ......................... VMDC10 passed test MachineAccount
      Starting test: Services
         ......................... VMDC10 passed test Services
      Starting test: ObjectsReplicated
         ......................... VMDC10 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VMDC10 passed test frssysvol
      Starting test: frsevent
         ......................... VMDC10 passed test frsevent
      Starting test: kccevent
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 09/09/2009   08:39:48
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 09/09/2009   08:39:48
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 09/09/2009   08:39:48
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 09/09/2009   08:39:48
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 09/09/2009   08:39:48
            (Event String could not be retrieved)
         ......................... VMDC10 failed test kccevent
      Starting test: systemlog
         ......................... VMDC10 passed test systemlog
      Starting test: VerifyReferences
         Some objects relating to the DC VMDC10 have problems:
            [1] Problem: Missing Expected Value

             Base Object: CN=VMDC10,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov

             Base Object Description: "DC Account Object"

             Value Object Attribute Name: frsComputerReferenceBL

             Value Object Description: "SYSVOL FRS Member Object"

             Recommended Action: See Knowledge Base Article: Q312862

             
            [1] Problem: Missing Expected Value

             Base Object:

            CN=NTDS Settings,CN=VMDC10,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov

             Base Object Description: "DSA Object"

             Value Object Attribute Name: serverReferenceBL

             Value Object Description: "SYSVOL FRS Member Object"

             Recommended Action: See Knowledge Base Article: Q312862

             
         ......................... VMDC10 failed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck

Server 2008 #1 New Dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Connectivity
         ......................... VDC30 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Replications
         ......................... VDC30 passed test Replications
      Starting test: NCSecDesc
         ......................... VDC30 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC30\netlogon)
         [VDC30] An net use or LsaPolicy operation failed with error 67, The network name cannot be found..
         ......................... VDC30 failed test NetLogons
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov, when we were trying to reach VDC30.
         Server is not responding or is not considered suitable.
         ......................... VDC30 failed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VDC30 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VDC30 passed test RidManager
      Starting test: MachineAccount
         ......................... VDC30 passed test MachineAccount
      Starting test: Services
         ......................... VDC30 passed test Services
      Starting test: ObjectsReplicated
         ......................... VDC30 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VDC30 passed test frssysvol
      Starting test: frsevent
         ......................... VDC30 passed test frsevent
      Starting test: kccevent
         ......................... VDC30 passed test kccevent
      Starting test: systemlog
         ......................... VDC30 passed test systemlog
      Starting test: VerifyReferences
         Some objects relating to the DC VDC30 have problems:
            [1] Problem: Missing Expected Value
             Base Object: CN=VDC30,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DC Account Object"
             Value Object Attribute Name: frsComputerReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
            [1] Problem: Missing Expected Value
             Base Object:
            CN=NTDS Settings,CN=VDC30,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DSA Object"
             Value Object Attribute Name: serverReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
         ......................... VDC30 failed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck

Server 2008 #2 New DCdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VDC20
      Starting test: Connectivity
         ......................... VDC20 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VDC20
      Starting test: Replications
         ......................... VDC20 passed test Replications
      Starting test: NCSecDesc
         ......................... VDC20 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC20\netlogon)
         [VDC20] An net use or LsaPolicy operation failed with error 67, The network name cannot be found..
         ......................... VDC20 failed test NetLogons
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov, when we were trying to reach VDC20.
         Server is not responding or is not considered suitable.
         ......................... VDC20 failed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VDC20 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VDC20 passed test RidManager
      Starting test: MachineAccount
         ......................... VDC20 passed test MachineAccount
      Starting test: Services
         ......................... VDC20 passed test Services
      Starting test: ObjectsReplicated
         ......................... VDC20 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VDC20 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... VDC20 failed test frsevent
      Starting test: kccevent
         ......................... VDC20 passed test kccevent
      Starting test: systemlog
         ......................... VDC20 passed test systemlog
      Starting test: VerifyReferences
         Some objects relating to the DC VDC20 have problems:
            [1] Problem: Missing Expected Value
             Base Object: CN=VDC20,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DC Account Object"
             Value Object Attribute Name: frsComputerReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
            [1] Problem: Missing Expected Value
             Base Object:
            CN=NTDS Settings,CN=VDC20,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DSA Object"
             Value Object Attribute Name: serverReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862
             
         ......................... VDC20 failed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck

OK, next step is to run through this document: http://support.microsoft.com/kb/312862

Yeah I was afraid of that..Ha!!!!!  I'll start running through that now and see what I can come up with.  

You find anything your unsure of just ask.

Ok, well I don't have a null server reference, but I do have a null FRS DomainController name.  I don't see anywhere that it tells how to fix that attribute?

( and I don't even see NTDS Settings or nTFRSMember anywhere in my ADSI Edit settings )

Ok.  I'm not seeing any of the attributes or containers this article talks about.  I also just remembered, that this server reports that it isn't even part of a set.  I don't even know if any FRS Replica sets were even setup!

These are setup automatically when you make it a domain controller.
In ADSI Edit expand the Domain Containter then system etc etc as per the document, there should be at least 1 entry in there.

Have you checked for the event log that is mentioned in the document?

can you browse to \\servername\sysvol for each of the domain controllers?

I can only browse to the SYSVOL for the 2003 server.  The 2008's never grab everything and shared it once they did have it copied over.

They talk about the NTDS Settings object in the Config in ADSI Edit.  I don't have the NTDS Settings object.  I have NTDS Quota?  

Which section are you looking at?

I'm in ADSI Edit, under Configuration container.  

Sorry I meant of the document

I am wondering (only because I have just been involved in another post with a similar issue) if your SYSVOL is not being shared properly

Can you try this: http://support.microsoft.com/kb/290762
Follow the: Nonauthoritative restore section

I'm just starting at the top where it talks about a null Server reference.  I wanted to check EVERYTHING to make sure I have everything I need in order. (Repairing the null Server-Reference attributes)  I just want to double check all of it, but I'm not seeing a lot.  

Now I DID find the two attributes that I am missing, but I'm not sure yet how to fill them in.  I think it's just the DN of the primary server, but I haven't read through all of it yet.

I've already tried the non-authoritative restore.  I think part of the problem is the server doesn't even seem to have a FRS Replica set.  Without a FRS replica set and some server references, I'm not even sure how it would attempt to share the sysvol

OK can you post the ntfrsutl ds results?

Here's a post of the results before I actually try to recover the replica set.  This is from teh 2003 Server.


NTFRS CONFIGURATION IN THE DS
SUBSTITUTE DCINFO FOR DC
   FRS  DomainControllerName: (null)
   Computer Name            : VMDC10
   Computer DNS Name        : VMDC10.osdb.oh.gov

BINDING TO THE DS:
   ldap_connect     : VMDC10.osdb.oh.gov
   DsBind     : VMDC10.osdb.oh.gov

NAMING CONTEXTS:
   SitesDn    : CN=Sites,cn=configuration,dc=osdb,dc=oh,dc=gov
   ServicesDn : CN=Services,cn=configuration,dc=osdb,dc=oh,dc=gov
   DefaultNcDn: DC=osdb,DC=oh,DC=gov
   ComputersDn: CN=Computers,DC=osdb,DC=oh,DC=gov
   DomainCtlDn: OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
   Fqdn       : CN=VMDC10,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
   Searching  : Fqdn

COMPUTER: VMDC10
   DN   : cn=vmdc10,ou=domain controllers,dc=osdb,dc=oh,dc=gov
   Guid : 315b9584-fee3-48ad-8cb40c2ee4115667
   UAC  : 0x00082000
   Server BL : CN=VMDC10,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Confi
guration,DC=osdb,DC=oh,DC=gov
   Settings  : cn=ntds settings,cn=vmdc10,cn=servers,cn=default-first-site-name,
cn=sites,cn=configuration,dc=osdb,dc=oh,dc=gov
   DNS Name  : VMDC10.osdb.oh.gov
   WhenCreated  : 7/6/2007 0:24:3 Eastern Standard Time Eastern Daylight Time [300]
   WhenChanged  : 9/4/2009 1:45:54 Eastern Standard Time Eastern Daylight Time [300]
   VMDC10 IS NOT A MEMBER OF ANY SET!

OK so in ADSI Edit, expand the DOMAIN containter then expand your domain name then expand Domain Controllers and expand your domain controller (VMDC10) under NTFS Subscriptions presumably this is empty?

If so right click on NTFRS Subscriptions and select New Object select nTFRSSubscriber and then enter the CN Value as Domain System Volume (SYSVOL share)

Actually just found this so you can follow this to re-create the replica: http://www.shantilal.net/technotes/1.html

I don't have any NTFS subscriptions at all.  I'm actually going to put things on hold for just a min.  My wheels are turning and I'm starting to get a grip slightly on what's going on here, and I'm going to run through recovering the replica set per that other document.  I'm starting to see where some of these items are so maybe I'll get lucky.

Ok.  I don't have the NTFS Subscriptions as described.  Shall I just create that new container and then add in the ntfrsSubscriber attribute?

Not sure to be honest I have never seen that, try it and find out.

I am suprised it's not there, can youpost a screen capture?

Here ya go!  I just hope I'm in the correct place and not making an a** out of myself!  I've never had to dig this DEEP into AD to fix something - It's a whole new world!

How bizarre!
I have never seen that before.

If you expand System then File Replication ervice then Domain System Volume

What do you see.

Well I'm actually trying to add my member back in now so I actually see under File Replication Service:

CN=osdb.oh.gov
       CN=vmdc10.osdb.oh.gov

NOW I have to try and recreate the Ntfrs Subscriptions and frsMemberReference information, correct?

Before you start manually adding entries in ADSI Edit can you try this: http://support.microsoft.com/kb/315457

Sure I'll give it a try

Ok.  Here's another one to throw at you.

In this step:
 GUID is the GUID of the domain system volume replica set that is shown in the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Replica Sets\GUID

There isn't any GUID in that Replica Set - I've even attached the screen shot!

let me have a look into this.
Can you just double check that the File Replication Service is running?

This DC is not in a good way!!
Your GUID should be: 315b9584-fee3-48ad-8cb40c2ee4115667

I turned it off per the instructions of the document in order to rebuild.  Prior to that yes FRS was running, but as we know the SYSVOL and NETLOGON weren't being copied and the server's weren't part of any Replica Set.

I know the DC is not good, and it's our ONLY DC that will accept user logons and actually verify security identities!

OK carry on with the document, I have posted the GUID that it should need.

So I guess I just add a new Key under Replica Sets and Cumulative Replica Sets with that GUI ID, then add in the BurFlags and then set them?  

Try it.  I don't think at this stage we have much to loose!

How's it going?

Well just got back from lunch.  Realized that as I'm following this SYSVOL rebuild that the other two DC's also don't have the GUID in the Replica Set's entry in the registry.  I guess I have to add those back in, but I still don't actually have members as part of a set.  

What I would recommend is getting this 2003 server working then we can look at the other 2, it may be better to dcpromo them and then repromote them but I want to help you to make sure you have 1 good working domain controller first.

Well I was actually trying to demote the other two, allow them to run just as DNS servers, and then work on the 2003, but I couldn't even demote!.

Now I'm just trying to get the replication set correctly setup, but even that's not working completely yet.

Once you gave the 2003 server working you should be able to demote the others if not we can force demotion.

How far through the repair document did you get?

Well I just need to figure out what it means to have this thing running like it's supposed too.  We trusted the other tech and as I'm finding out it wasn't setup properly

I tried everything I could, but I still haven't been able to get through Recovering FRS objects yet.  

Have you run all the commands in the document?

There are a couple of other options:

> install another 2003 DC so we have a copy of the active directory, then DCPROMO all the others.
> take a system state backup, and rebuild

how do you want to proceed?
Donyou want to carry on with those docs firstto see if you can sort it?

Honestly I would like to see if I can carry forward and see if I can sort it out.  I thought about building another 2003 server, but I don't know if it'll even matter since the Replica Set on the 2003 server isn't correct right now.  If it's not part of a replica set how will it replicate to any other server (even another 2003 server?)

If I'm creating the NtFrsSubscriber object what should go in the value field?  My server name or the name of the domain??

it should say: Domain System Volume (SYSVOL share)

Which in my case is just osdb.oh.gov right?  That's what I put because that's the sysvol share I'm working with.

No it actually needs to be the text I posted.
See the link

Wow, ok, I'm all screwed up then.  I thought it just wanted the name, not the exact text...

It's ok just go back and delete it then recreate it

Ha!!! Already there.  Just hoping I do it correctly.  Man, gone are the days when you can just delete something and start again!  

You follow that link you will be fine!

Here's a new ntfrsutl ds log after making some changes

(from 2003 server)

NTFRS CONFIGURATION IN THE DS
SUBSTITUTE DCINFO FOR DC
   FRS  DomainControllerName: (null)
   Computer Name            : VMDC10
   Computer DNS Name        : VMDC10.osdb.oh.gov

BINDING TO THE DS:
   ldap_connect     : VMDC10.osdb.oh.gov
   DsBind     : VMDC10.osdb.oh.gov

NAMING CONTEXTS:
   SitesDn    : CN=Sites,cn=configuration,dc=osdb,dc=oh,dc=gov
   ServicesDn : CN=Services,cn=configuration,dc=osdb,dc=oh,dc=gov
   DefaultNcDn: DC=osdb,DC=oh,DC=gov
   ComputersDn: CN=Computers,DC=osdb,DC=oh,DC=gov
   DomainCtlDn: OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
   Fqdn       : CN=VMDC10,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
   Searching  : Fqdn

COMPUTER: VMDC10
   DN   : cn=vmdc10,ou=domain controllers,dc=osdb,dc=oh,dc=gov
   Guid : 315b9584-fee3-48ad-8cb40c2ee4115667
   UAC  : 0x00082000
   Server BL : CN=VMDC10,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
   Settings  : cn=ntds settings,cn=vmdc10,cn=servers,cn=default-first-site-name,
cn=sites,cn=configuration,dc=osdb,dc=oh,dc=gov
   DNS Name  : VMDC10.osdb.oh.gov
   WhenCreated  : 7/6/2007 0:24:3 Eastern Standard Time Eastern Daylight Time [300]
   WhenChanged  : 9/4/2009 1:45:54 Eastern Standard Time Eastern Daylight Time [300]

   SUBSCRIPTION: NTFRS SUBSCRIPTIONS
      DN   : cn=ntfrs subscriptions,cn=vmdc10,ou=domain controllers,dc=osdb,dc=oh,dc=gov
      Guid : d8a3b8af-c655-472e-98f7fa7411afa8f6
      Working       : (null)
      Actual Working: c:\windows\ntfrs
      WhenCreated  : 9/9/2009 13:57:10 Eastern Standard Time Eastern Daylight Time [300]
      WhenChanged  : 9/9/2009 13:57:10 Eastern Standard Time Eastern Daylight Time [300]

      SUBSCRIBER: DOMAIN SYSTEM VOLUME (SYSVOL SHARE)
         DN   : cn=domain system volume (sysvol share),cn=ntfrs subscriptions,cn
=vmdc10,ou=domain controllers,dc=osdb,dc=oh,dc=gov
         Guid : 73ab9206-9b96-4875-8060f03b13e6afab
         Member Ref: CN=vmdc10,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=osdb,DC=oh,DC=gov
         Root      : c:\windows\sysvol\domain
         Stage     : c:\windows\sysvol\staging\domain
         WhenCreated  : 9/9/2009 14:18:15 Eastern Standard Time Eastern Daylight Time [300]
         WhenChanged  : 9/9/2009 14:20:54 Eastern Standard Time Eastern Daylight Time [300]
   
Subscriber Member Back Links:
      cn=vmdc10,cn=domain system volume (sysvol share),cn=file replication service,cn=system,dc=osdb,dc=oh,dc=gov

SETTINGS: FILE REPLICATION SERVICE
   DN   : cn=file replication service,cn=system,dc=osdb,dc=oh,dc=gov
   Guid : 3b0ba5a4-5857-428d-8e03f975b2a91cba
   WhenCreated  : 7/6/2007 0:22:14 Eastern Standard Time Eastern Daylight Time [300]
   WhenChanged  : 7/6/2007 0:22:14 Eastern Standard Time Eastern Daylight Time [300]

   SET: DOMAIN SYSTEM VOLUME (SYSVOL SHARE)
      DN   : cn=domain system volume (sysvol share),cn=file replication service,
cn=system,dc=osdb,dc=oh,dc=gov
      Guid : 44bd7800-68c6-455f-a7155f03f91c3f61
      Type          : (null)
      Primary Member: (null)
      File Filter   : (null)
      Dir  Filter   : (null)
      FRS Flags     : (null)
      WhenCreated  : 9/9/2009 14:15:25 Eastern Standard Time Eastern Daylight Time [300]
      WhenChanged  : 9/9/2009 14:15:25 Eastern Standard Time Eastern Daylight Time [300]

      MEMBER: VMDC10
         DN   : cn=vmdc10,cn=domain system volume (sysvol share),cn=file replica
tion service,cn=system,dc=osdb,dc=oh,dc=gov
         Guid : a71def17-bdac-4ceb-82dae1ca01a3afa4
         Server Ref     : CN=NTDS Settings,CN=VMDC10,CN=Servers,CN=Default-First
-Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
         Computer Ref   : cn=vmdc10,ou=domain controllers,dc=osdb,dc=oh,dc=gov
         Cracked Domain : osdb.oh.gov
         Cracked Name   : 00000002 NETWORK\VMDC10$
         Cracked Domain : osdb.oh.gov
         Cracked Name   : fffffff4 S-1-5-21-2862464823-3494292689-1683498531-1002

         Computer's DNS : VMDC10.osdb.oh.gov
         WhenCreated  : 9/9/2009 14:17:4 Eastern Standard Time Eastern DaylightTime [300]
         WhenChanged  : 9/9/2009 14:17:4 Eastern Standard Time Eastern DaylightTime [300]

      MEMBER: VDC30
         DN   : cn=vdc30,cn=domain system volume (sysvol share),cn=file replication service,cn=system,dc=osdb,dc=oh,dc=gov
         Guid : 9793935d-5db3-4a18-89d51c3e38e84769
         Server Ref     : CN=NTDS Settings,CN=VDC30,CN=Servers,CN=Default-First-
Site-Name,CN=Sites,CN=Configuration,DC=osdb,DC=oh,DC=gov
         Computer Ref   : cn=vdc30,ou=domain controllers,dc=osdb,dc=oh,dc=gov
         Cracked Domain : osdb.oh.gov
         Cracked Name   : 00000002 NETWORK\VDC30$
         Cracked Domain : osdb.oh.gov
         Cracked Name   : fffffff4 S-1-5-21-2862464823-3494292689-1683498531-329
8
         Computer's DNS : VDC30.osdb.oh.gov
         WhenCreated  : 9/9/2009 14:20:35 Eastern Standard Time Eastern Daylight Time [300]
         WhenChanged  : 9/9/2009 14:20:51 Eastern Standard Time Eastern Daylight Time [300]

Ok that's good, now run dcdiag on it?

Server 2003 new DCdiag:
Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Connectivity
         ......................... VMDC10 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VMDC10
      Starting test: Replications
         ......................... VMDC10 passed test Replications
      Starting test: NCSecDesc
         ......................... VMDC10 passed test NCSecDesc
      Starting test: NetLogons
         ......................... VMDC10 passed test NetLogons
      Starting test: Advertising
         ......................... VMDC10 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VMDC10 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VMDC10 passed test RidManager
      Starting test: MachineAccount
         ......................... VMDC10 passed test MachineAccount
      Starting test: Services
         ......................... VMDC10 passed test Services
      Starting test: ObjectsReplicated
         ......................... VMDC10 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VMDC10 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         ......................... VMDC10 failed test frsevent
      Starting test: kccevent
         ......................... VMDC10 passed test kccevent
      Starting test: systemlog
         ......................... VMDC10 passed test systemlog
      Starting test: VerifyReferences
         ......................... VMDC10 passed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck


Server 2008 new DCdiag:


Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Connectivity
         ......................... VDC30 passed test Connectivity

Doing primary tests
   
   Testing server: Default-First-Site-Name\VDC30
      Starting test: Replications
         ......................... VDC30 passed test Replications
      Starting test: NCSecDesc
         ......................... VDC30 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC30\netlogon)
         [VDC30] An net use or LsaPolicy operation failed with error 67, The network name cannot be found..
         ......................... VDC30 failed test NetLogons
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov, when we were trying to reach VDC30.
         Server is not responding or is not considered suitable.
         ......................... VDC30 failed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VDC30 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VDC30 passed test RidManager
      Starting test: MachineAccount
         ......................... VDC30 passed test MachineAccount
      Starting test: Services
         ......................... VDC30 passed test Services
      Starting test: ObjectsReplicated
         ......................... VDC30 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VDC30 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... VDC30 failed test frsevent
      Starting test: kccevent
         ......................... VDC30 passed test kccevent
      Starting test: systemlog
         ......................... VDC30 passed test systemlog
      Starting test: VerifyReferences
         ......................... VDC30 passed test VerifyReferences
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom
   
   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck


Those two object errors are gone, but I assume because it can't connect to the netlogon share is just because it may not have all replicated just yet?

Well done you now seem to have a fully working Windows 2003 domain controller.

You've made a lot of changes so I would suggest leaving it now to catch up with itself and then we can come back to the 2008 DC's

what timezone are you in? Well done for sticking with it and getting this far!

Thanks!  I'm in Eastern Time zone.  This has not been easy but wow you've been patient with me.

One more question.  I added in the NTFRS Subscriptions object to the third 2008 server.  It was only picked up on one VDC30 and not 20.  Not a problem adding it in right?

Is that -5GMT?

It shouldn't be a problem because we will probably need to do that on the 2008 machines.

I would leave that now to settle down and we can pick it back up tomorrow (it's 8pm here so time for some dinner!)

Oh wow!!!  Go eat!!!! We can pick it up tomorrow just fine!

It's looking a lot better than it was.
Worst case scenario mow we force the 2008 DC's to demote cleanup AD then promote them again.

But that's the easy bit!

Ok.  Well I got both 2008 servers as members of the replica set.  now it's time to rest for the night and enjoy the progress so far!

Where did you add them?

When you say "Well I got both 2008 servers as members of the replica set" which process did you use to do this?

I add them as members in ADSIEdit as ntfrsmembers - Although I'm still not sure if the set is setup correctly.  I'm seeing errors in the event log that the frsReplica Set Type isn't correct. I thought that would autopopulate, but it says not set.

OK, I probably wouldn't have added the other servers into the NTFRSMembers.

Might be worth removing them from there.
Have you restarted the File Replication Service on all the machines?

can you post the full event log you are receiving?

Should I remove the 2008 servers NTFRS Subscriptions as well if I'm removing them as ntfrsMembers?

Following is the summary of warnings and errors encountered by File Replication Service while polling the Domain Controller VDC20.osdb.oh.gov for FRS replica set configuration information.
 
 The nTFRSReplicaSet object cn=domain system volume (sysvol share),cn=file replication service,cn=system,dc=osdb,dc=oh,dc=gov has a invalid value for the attribute frsReplicaSetType.


That was the one error I was getting.  I deleted VDC20 and VDC30 from ntfrs Members and subscriptions.  I can always add them back in if needed.  Now VMDC10 (Server 2003) is the only ntfrs member and subscriber.

So we should have a fully functional 2003 DC now.
Lets have a look at the 2008 server (can we do 1 at a time)

Can you run DCDIAG and NTFRSUTL ds
on one of the 2008 DC's

that error message was that on the 2003 server?

No that message was on one of the 2008 servers. Here's the post for the first 2003 server I want to make a DC

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\VDC20
      Starting test: Connectivity
         ......................... VDC20 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\VDC20
      Starting test: Replications
         ......................... VDC20 passed test Replications
      Starting test: NCSecDesc
         ......................... VDC20 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\VDC20\netlogon)
         [VDC20] An net use or LsaPolicy operation failed with error 67, The net
work name cannot be found..
         ......................... VDC20 failed test NetLogons
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\VMDC10.osdb.oh.gov, whe
n we were trying to reach VDC20.
         Server is not responding or is not considered suitable.
         ......................... VDC20 failed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... VDC20 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... VDC20 passed test RidManager
      Starting test: MachineAccount
         ......................... VDC20 passed test MachineAccount
      Starting test: Services
         ......................... VDC20 passed test Services
      Starting test: ObjectsReplicated
         ......................... VDC20 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... VDC20 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... VDC20 failed test frsevent
      Starting test: kccevent
         ......................... VDC20 passed test kccevent
      Starting test: systemlog
         ......................... VDC20 passed test systemlog
      Starting test: VerifyReferences
         Some objects relating to the DC VDC20 have problems:
            [1] Problem: Missing Expected Value
             Base Object: CN=VDC20,OU=Domain Controllers,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DC Account Object"
             Value Object Attribute Name: frsComputerReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862

            [1] Problem: Missing Expected Value
             Base Object:
            CN=NTDS Settings,CN=VDC20,CN=Servers,CN=Default-First-Site-Name,CN=S
ites,CN=Configuration,DC=osdb,DC=oh,DC=gov
             Base Object Description: "DSA Object"
             Value Object Attribute Name: serverReferenceBL
             Value Object Description: "SYSVOL FRS Member Object"
             Recommended Action: See Knowledge Base Article: Q312862

         ......................... VDC20 failed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : osdb
      Starting test: CrossRefValidation
         ......................... osdb passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... osdb passed test CheckSDRefDom

   Running enterprise tests on : osdb.oh.gov
      Starting test: Intersite
         ......................... osdb.oh.gov passed test Intersite
      Starting test: FsmoCheck
         ......................... osdb.oh.gov passed test FsmoCheck


NTFRSUTL DS won't run.  It errors out with an APPCRASH now.  Also

Where was this diag from?

That DCDIAG was from a 2008 server

What happens when you try to DCPROMO this server?

Do I need to delete the DNS delegations?

Ok here's what it gives me:

Managing the network session with vmdc10.osdb.oh.gov failed
The network connection does not exist.

I think the best course of action is to uinstall DNS from the 2 Windows 2008 servers.  Did you reconfigure the DHCP scope so the clients were only pointing to the 2003 server?

Then we force the 2 Windows 2008 servers to demote themselves (1 by 1) and then do a METADATA Cleanup.

What do you think?

Well I  guess we can try it.  I want to run that through my boss to see what she says.  It may take a day or two once I do reconfigure all the DHCP to run through one server.  Makes me a little nervous to do this in a live environment, but I do know that the 2003 Server is running properly.

Give me a few to talk with her and see - I think if we can do it one by one and use one 2008 server and the 2003 as DNS see how that goes, then take down the second DNS server that may be a better way.

Ok.  We're going to backup all the DNS and AD structure and settings just in case, and once that happens we will start with one of the 2008 Servers and start tearing it down and force demoting it.  I'll hit this back up in an  hour or so and let you know

Ok.  DHCP has been changed so that all users point to other DNS servers (one which is a 2008 server that I'll change once we're ready to force demote that one).  I've never done a demote /forceremove so I'm not exactly sure what I'm in for.

Ok.  Force Removal was a success, now it's on to removing the role and onto Metadata cleanup.

Ok - Well the metadata clean up didn't go as well.  I followed some instructions, but I'm getting an error at a point where it states:

DsRemoveDsServerW error - Access Denied.

I was able to use the 2003 server and ntdsutil to get through selection of the server, and two pop-up boxes that asked me if I was sure I wanted to get rid of this server.  Some of the errors were FRS errors, but I figured that was ok, because FRS wasn't working anyhow.  I did shut down the 2008 server I'm trying to remove.

Well in working through the article I tried to delete the server from the Sites and Services and when I delete the NTDS Settings it acts like it's deleting and then nothing.  If I try to delete the server itself it tells me I don't have sufficient rights - Which is odd because I'm logged on as the domain admin!  Weird!

Wooohoooo.  Got it.  Found out that the group Everyone was in the object security with Deny rights to delete anything.  Cleared that and I was able to get it knocked out of the Sites and Services, and in the Configuration object of ADSIEdit

excellent, so that DC has been completely removed now?

Well I'm going to check in the DNS and make sure there are no _msdcs records or Reverse Lookup records left, then I THINK it's out.  It's been powered off so I know it's not being use for anything right now.

So just to confirm you have done a METADATA cleanup and cleared DNS etc?

Is it essential that this server keeps its name?

Well when I did the METADATA clean up I got some errors, but I'm going to go through it one more time and make SURE it's gone.  Don't worry, not changing the name.  Just going through DNS right now.

I was just thinking it might actually be better if you could change the name?
Once you have confirmed a complete cleanup you can then try to DCPROMO back as a domain controller.

Well my server is no longer showing up as a server in the Site for the metadata cleanup, it's not in DNS anywhere, and I don't see it anywhere in the ADSIEdit.

Perfect, so power back up the 2008 box make sure it's pointing at your 2003 server for DNS (no other entries)

Make sure it can browser to \\2003servername\sysvol

Then try and DCPROMO it back into the domain.

So I need to add back in the AD Roles and DNS roles before I dcpromo or will that happen automatically?  I'm still powering up and will verify the sysvol share here in a min.

Nevermind.  Answered my own question.  In the process of DCPROMO now.  We'll see what happens.

I would leave the DNS roles off for now
If you just run DCPROMO it will do the rest for you if it finds that some of the installation is missing.

Well now I'm restarting without any errors during the DCPROMO.  I didn't install DNS, but I did make it a GC so that once we replicate the SYSVOL and NETLOGON it can handle user logons.

Excellent, let me know how it goes.

DCDIAG passes everything expect sysvol and netlogon which is to be expected since they haven't replicated yet (which means it's not advertising as a DC just yet).  

The nTFRSReplicaSet object cn=domain system volume (sysvol share),cn=file replication service,cn=system,dc=osdb,dc=oh,dc=gov has a invalid value for the attribute frsReplicaSetType.

I am seeing this error in the event log from about 5 mins ago.  I wonder if that will prevent it from trying to replicate the sysvol and netlogon shares...

Hmm, lets give it a bit of time

Ok.  I was thinking it just tried too soon.   Won't the ReplicaSetType be automatically generated by the server itself?  

Yes it should be.

Well then I'll let it sit for a few hours and see what does.

What is that value set to?

Well as soon as I find out where that value is, I'll let you know.  Ha!!!

It's all here: http://support.microsoft.com/kb/312862

Yeah I think I missed a step in there.  I found the spot where I forgot to set the ReplicaType.  I set it now and I think I'll restart all FRS processes and see what happens from that point.

Different Error now, but I've seen this on the net somewhere.  I think I need to force replication in the Sites and Services.  I can browse to \\server2003name\sysvol just fine so I know it can reach it.


Description:
The File Replication Service is having trouble enabling replication from VMDC10.osdb.oh.gov to VDC20 for c:\windows\sysvol\domain using the DNS name VMDC10.osdb.oh.gov. FRS will keep retrying.
 Following are some of the reasons you would see this warning.
 
 [1] FRS can not correctly resolve the DNS name VMDC10.osdb.oh.gov from this computer.
 [2] FRS is not running on VMDC10.osdb.oh.gov.
 [3] The topology information in the Active Directory Domain Services for this replica has not yet replicated to all the Domain Controllers.
 
 

In sites and services expand the domain controller then NTDS settings and right click on a replication partner and select replicate now.

That's what I did.  Now I'm going to start and stop NTFRS on the 08 Machine to force it to try again.

Ok.  It's giving me errors about the other 2008 DC, but that's ok because we're going to rip that one out later anyhow.  So far it's just stating that until it copies the sysvol and netlogon it won't become a DC. I'm going to monitor it for a bit and see what happens.  

Ehhh.  Still getting that it can't resolve using the FQDN DNS name, but I can ping using that name.  Odd

Ok you are finding that everything with active directory takes time ;-)

check the \\2008servername to see what shares are available.

Do you have the correct DNS suffix in the advanced settings of tcp/ip?
Have you got IPv6 disabled?

Yeah.  Everything with DNS is correct and I looked at DNS to make sure it was there and it was, and yes IP6 is disabled.  Odd that I can ping but it's still having trouble.  Well yeah I guess even when it said replication was successful it'll take time to actually happen?!?! Seems like if it was replicated and everything in the DCDIAG is clean (other than SYSVOL and NETLOGON) it should replicate fairly easily.

Let it settle down for a bit it's probably still doing it's thing.

Yeah.  No problem.  I'll let it sit for a while and then check in on it.  

Well I'm getting the same DNS error on both computers that they can't contact each other through the DNS name, so it must be that the replication of the connections hasn't happened just yet.  Either that or there must be some sort of schedule where FRS is running on one server when it's not running on the other (if that's even possible)

Are you able to restart both servers?

If you browse to \\2008server do you see any shares?

If you ping 2008servername.domainname from the 2003 server what do you get?
Also try nslookup 2008servername

What is the full event log error?

Have they both got only the 2003 server as DNS entry?

I may have actually found a problem.  When I did the nslookup here's what happened

C:\Documents and Settings\DCAdmin.VMDC10>nslookup vdc20
*** Can't find server name for address 10.0.1.10: Non-existent domain
Server:  UnKnown
Address:  10.0.1.10

Name:    vdc20.osdb.oh.gov
Address:  10.0.1.20

I changed some DNS to take out the other 2008 server and now I'm going to restart the NTFRS setup on both now that I get the correct response from NSLOOKUP

Well it's still reporting some trouble, but I'm going to let it go for a while and let all the replication just try to happen on it's own.  Hopefully everything will connect up within a few hours or a day.  It's not like I'm any worse off right now than I was before.  It's just a step or two away from being set.  Then I can work on the other 2008 server.

Your certainly a lot better off than you were when we started let it settle down then give me some info, DCDIAGS and full event log messages.

Feed me and I will process ;-)

Ha!!! Yeah I can see I'm better off for sure.  Now that I've actually had to dive in I can't believe how much there really is, especially when it's not working correctly.  I'll give it some good time and then I'll run the DCDIAGS.  The only thing I'm seeing so far is just the lack of the sysvol and netlogon which will come through as soon as it resolves the server by DNS name.  

Ok, well the sync did not take place over the weekend.  I restarted the NTFRS process and I'm still getting the error that it can't find the computer by DNS name.  I know everything had to have replicated over the weekend, and I can actually open up the command prompt to ping by name.  Maybe we need to check the FRS process on the Server 2003 server.

can you post the full event log you are receiving please?

The File Replication Service is having trouble enabling replication from VMDC10 to VDC20 for c:\windows\sysvol\domain using the DNS name VMDC10.osdb.oh.gov. FRS will keep retrying.
 Following are some of the reasons you would see this warning.
 
 [1] FRS can not correctly resolve the DNS name VMDC10.osdb.oh.gov from this computer.
 [2] FRS is not running on VMDC10.osdb.oh.gov.
 [3] The topology information in the Active Directory Domain Services for this replica has not yet replicated to all the Domain Controllers.
 
 This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.

This is from teh 2008 Server that I'm trying to replicate to.

and if you do:

NSLOOKUP VMDC10.osdb.oh.gov
from the Windows 2008 server what do you get?

C:\Users\dcadmin>nslookup vmdc10.osdb.oh.gov
Server:  vmdc10.osdb.oh.gov
Address:  10.0.1.10

Name:    vmdc10.osdb.oh.gov
Address:  10.0.1.10

and presumably that is correct?

Well yeah, that's the correct IP address and I'm guessing it's pulling that server name, because that's the DNS server that I have in VDC20 (server 2008 machine)

Think maybe the D4/D2 BurFlag restore would help?

it's worth a go, it can't do any harm.

Yeah!!!!!  Check this out!!!!!!!  This is from teh 2008 Server!!!!!

The File Replication Service successfully added this computer to the following replica set:
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
 
Information related to this event is shown below:
Computer DNS name is "VDC20.osdb.oh.gov"
Replica set member name is "VDC20"
Replica set root path is "c:\windows\sysvol\domain"
Replica staging directory path is "c:\windows\sysvol\staging\domain"
Replica working directory path is "c:\windows\ntfrs\jet"

**** Next Message ****

The File Replication Service successfully added this computer to the following replica set:
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
 
Information related to this event is shown below:
Computer DNS name is "VDC20.osdb.oh.gov"
Replica set member name is "VDC20"
Replica set root path is "c:\windows\sysvol\domain"
Replica staging directory path is "c:\windows\sysvol\staging\domain"
Replica working directory path is "c:\windows\ntfrs\jet"

The File Replication Service successfully added the connections shown below to the replica set:
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
 
    Inbound from    "VMDC10.osdb.oh.gov"
    Outbound to    "VMDC10.osdb.oh.gov"

OK excellent that's brilliant news!

Yeah!!!  Now I just have to wait for the actual replication to take place.  Think I should force it or just let it happen in the next 15mins and monitor the event logs?!

Nevermind.  I just answered my own question..Ha!!!

The File Replication Service is no longer preventing the computer VDC20 from becoming a domain controller. The system volume has been successfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL.
 
Type "net share" to check for the SYSVOL share.


Wooooohoooooo!!!!!!!  Now I'm just waiting for the netlogon share to become replicated!

Patience! It will happen!

You know what comes next though don't you?

Ummm.. Working on the next 2008 server I would imagine?!

(and yeah I know about the patience, I'm just excited this is working like it should now

Yes, you need to follow the same process on the other 2008 server and you should all be sorted!

Yeah, should be interesting.  I'm thinking I should add DNS back into the 2008 server as a backup, or you think I'll be ok with one DNS server for right now?

Sadly the other 2008 server is actually the only DHCP server we have.  I have to be careful how I approach this one.

I would leave the DNS off at the moment and lets get the other server working.

You can always install DHCP on the other 2008 server now it is working and transfer the scopes over?

Yeah I thought about that.  I'm going to do that now so it's ready to handle everything.

Now when I have all three servers up and running, I should spread out the FSMO roles correct?  Also, this will allow one of the 2008 servers to pick-up user sign-ons if the primary (2003) server goes down?

You need to make them Global Catalogues this will allow them to authenticate and it is good practice to spread the FSMO roles out.

Lets get the other one as a DC first though ;-)

Oh I will.  I'm working the DHCP for the first 08 server so I can transfer that to there while I work on the other 08 server

Ok.  DHCP is switched over and working, and now it's time to work on the second 08 Server.  Oh boy!

Good luck! Let me know if you need anything!

Just make sure have DNS configured for the 2003 server before you start work on it.

Oh yeah.  All DNS is running through the 2003 server.  Everything is primary to that server.

Well I re-joined the seond 2008 server and it actually was replicating from the first 2008 server and not the 2003 server, but it was able to replicate the first time.  Now I'm just waiting until I get confirmation that the SYSVOL came over.  

It's not unusual for it to pick a different replica that is quite normal.
Well done!

Yeah, I have some weird errors in the dcdiag log, but I'm going to wait a while and let the system calm down and work through some of that.  Some of the errors I think are from previous problems.  

Now I just have to add DNS back in to the 2008 servers, because our ultimate goal is to dcpromo the 2003 server.

Here's an odd situation.  The NETLOGON folder is now gone from teh 2003 server!?!?!  Why would that happen?  I didn't delete anything?

Check the services and make sure all the ones that are set to automatically are started specifically the server service

I would hold off changing DNS until we have all the issues sorted, keep it simple for now.

Well Server and Net Logon are started and set to automatic.
( yeah, just need to make sure I re-create the 08 machines as DNS before I decommision the 03 server in a few weeks if it's all working well.  Once I verify the netlogon correctly I'm going to shut down the 03 server and see if the 08 server's pick the logons )

I also noticed that my Scripts folder has gone missing from

c:\windows\sysvol\sysvol\osdb.oh.gov\Scripts

That Scripts folder is gone.  Could that be why the netlogon folder is gone?  (on 03 server)

Ok before you do that you will need to make the other domain controllers Global Catalogue servers, to do this go into Active Directory Sites and Services expand each domain controller and right click on NTDS Settings select properties and then check the box to make it a GC.

You will also need an alternative DNS in place.

I would just let it all settle down first and see if it comes back

Ok.   I thought maybe I could just re-create the folder, and restart.  The information should still be in the registry and it should just come back after a restart I would think.

And all the other servers are Global Catalogue servers.  That's one thing I did from the start.  I knew I would have to make another DNS server, but I'll let it go for a while before I test anything anyhow.  I want to make sure everything is running fine status quo first, and make sure no serious errors in the DCDIAG's on all three servers.

The Netlogon service could not create server share C:\Windows\SYSVOL\sysvol\osdb.oh.gov\SCRIPTS.  The following error occurred:
The system cannot find the file specified.+

I see that on the 08 server I just brought back.  I don't see that on the first 2008 server I brought back, however I don't have the netlogon share on any of them, so I guess it just all needs time to catch up.

I expect so let it settle down then we can start the diagnostic process.

We will get there so don't worry!
At least we know we have at least 1 good 2008 DC.

We have moved forward from where we started this process well done for sticking with it.

Well thanks.  I guess I can get some web design work done while I'm letting them settle down for a while.  I'll let things go for a while and then I'll run some diags' and post them for you to look at.!

Well I even restarted the 2003 server the netlogon share folder didn't come back, so we'll have to take a look at that tomorrow and see what's going on there.  I'm just wondering if I create the Scripts folder again and restart the system may just recreate the hierarchary and set it back up, but I"m going to let it sit for the night and see what happens.  

Well my Scripts folder is back and shared (so Netlogon on teh 2003 server is back), but the folder is empty.  The scripts are actually in AD, and I can browse to find them, but it's not actually populated in the sysvol/osdb.oh.gov/Scripts folder.

OK, anny errors in the event logs on the 3 DC's that might help?

Can you run: NTFRSUTL ds

Everything comes back clean.  I recreated that folder, but now I need to move the scripts to there to force, and then restart NTFRS to replicate.  I just have to find out where the scripts actually are located.

Ok.  Nevermind.  I found where scripts are (with inside the Policies folder), but do I copy the entire policy folder into the Scripts folder for it to actually take place?  

If you copy them manually you may end up with double of everything (FRS is a funny beast!)
Check the File Replication Service is running on all servers (might be worth restarting it)

Check all event logs.

Check \\servername\sysvol for each server what do you get?

Well on the 2003 server I see my domain with the Policies and Scripts folder inside.  Although Scripts is empty.  On one 2008 server all of a sudden Sysvol disappeared.
 
On the other 2008 Server I actually have two Policy and two Scripts but the duplicate's look like this:

Policies_NTFRS_0056154a
Scripts_NTFRS_03f0afa2

But I don't see that on the 2003 server. ...Odd

I'm willing to try copying it in manually to see if that helps.  Do I just copy the entire policy object for that VB Script that I need to use?

I'm starting to see _NTFRS folders in all my servers, so maybe it's just taking some time to rebuild the Scripts folder?!?!  

Well I guess even though it either wasn't there or it was pulling it down, I've had several Windows XP users sign on with no problems this morning...Interesting.

Well, I don't think it's actually fully functional.  I just signed on a different computer and not all my drive mappings came through (which are all done with a .vbd script).  All my script folders are still empty.

Ok.  Here's an interesting thought.  I just signed onto an XP computer just fine, and all my drive mappings and everything worked fine, but with Vista it didn't!.  Now that's no fun.  I think everything is still trying to replicate, but I don't see why the Scripts folder would still be empty

(any why would I start getting _NTFRS folders as well for both Policies and Scripts?)

This is what happens when they are manually created ;-(

I think you need to perform an authoritative restore on your FRS as per before

Before you do that make sure you backup your scripts folder and backup any Group Policies using Group Policy managemant console.
Something is not quite right with the replication.
Nothing in any of the event logs?

Nothing in the event logs at all.  Other than when I start/stop NTFRS

Well I got rid of the odd folder names (http://support.microsoft.com/?id=328492) and was able to restore the original Policies objects back to normal.  Still no Scripts inside the scripts folder....Odd.  Some people are logging on and getting everything and other's aren't.

So which servers have got something in the scripts folder?

Actually none of the servers have anything in the scripts folder, however everything seems to be stable.  I added DNS back into the 2008 servers (so I had a backup in case something happened) and it was successful with no errors (on both servers).  

People are signing onto their computers, and group policies seem to be taking place, drives are getting mapped per the .vbs script we have running and internet is flowing just fine.....Some of the connections are a little slow, and some people may need to restart to grab the policy again, but other than that, I've only had 2 complaints all morning.

OK so there were no login scripts?
OK so we are all good?

Well there is a logon script in AD.  We created a new GPO and made a user logon script (to map network drives based on Group Membership), but it's just not showing anything in the Scripts folder.  However, people are logging on just fine and getting their mapped network drives with no issues.  Very odd.

Maybe it's working because the Policies folder is there and the policy has a Scripts folder that contains everything.  Maybe the actual Scripts folder doesn't need to actually have anything in there.

Now I need to spread out my FMSO roles to keep everything in check.

The scripts folder will only have login scripts that are not part of the policies.

See here for transfering FSMO roles: http://support.microsoft.com/kb/324801

as you plan to demote the 2003 server transfer them between the 2 2008 servers.
Edit your DHCP settings so the 2008 servers are being used for DNS
modify the servers DNS so they are also using the 2008 DNS and then remove the global catalogue setting for the 2003 server.

Well I'm not ready to demote anything yet, but I'm fairly comfortable with the global catalogue.  DHCP has already been changed and verified working (as I'm running the new setup), and the 2008 Servers are global catalogues already.

I didn't realize the scripts only if it's not part of a policy.  That means everything must be working correctly then!

If your scripts are part of a policy then they will be stored insde your policies the scripts folder is for login scripts that are not part of group policies.

Well I think your all done then!
Well done for sticking with it! We got there in the end.

Well all of our scripts are part of policies so yeah, seems like it's done.

Awesome!!!!  That was great!  You have so much more patience then I would have had!

Absolutely perfect step by step instructions and very clear solution.  Worked perfectly!

Comes with the teritory!
At least your all back up and running!