resolving missing domain controllers in Windows domain

Hi,

The saga of the trashed office network continues.

I have a DC running windows server 2003 that needs to be upgraded.  In order to do this I need to remove the references to the old secondary domain controllers that have vanished.

I have run DCDIAG and have some issues that I think need to be resolved before installing the new 2012 R2 domain controller.

I would think I need to remove the missing DCs and make the remining one the GC and update the FSMO roles.
What do you think?

Thanks.

Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\Documents and Settings\mcioffi>dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site\OHM
      Starting test: Connectivity
         ......................... OHM passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site\OHM
      Starting test: Replications
         [Replications Check,OHM] A recent replication attempt failed:
            From TESLA to OHM
            Naming Context: DC=ForestDnsZones,DC=PRO_DC_3,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2015-06-18 10:48:55.
            The last success occurred at 2014-03-31 13:45:22.
            10400 failures have occurred since the last success.
         [TESLA] DsBindWithSpnEx() failed with error 1722,
         The RPC server is unavailable..
         [Replications Check,OHM] A recent replication attempt failed:
            From MARCONI to OHM
            Naming Context: DC=ForestDnsZones,DC=PRO_DC_3,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2015-06-18 10:49:16.
            The last success occurred at 2014-06-26 11:57:03.
            8313 failures have occurred since the last success.
         [MARCONI] DsBindWithSpnEx() failed with error 1722,
         The RPC server is unavailable..
         [Replications Check,OHM] A recent replication attempt failed:
            From TESLA to OHM
            Naming Context: DC=DomainDnsZones,DC=PRO_DC_3,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2015-06-18 10:48:55.
            The last success occurred at 2014-03-31 14:06:23.
            10400 failures have occurred since the last success.
         [Replications Check,OHM] A recent replication attempt failed:
            From MARCONI to OHM
            Naming Context: DC=DomainDnsZones,DC=PRO_DC_3,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2015-06-18 10:49:16.
            The last success occurred at 2014-06-26 11:57:03.
            8313 failures have occurred since the last success.
         [Replications Check,OHM] A recent replication attempt failed:
            From MARCONI to OHM
            Naming Context: CN=Schema,CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:49:58.
            The last success occurred at 2014-06-26 11:56:20.
            8313 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From TESLA to OHM
            Naming Context: CN=Schema,CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:50:19.
            The last success occurred at 2014-03-31 13:45:21.
            10400 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From PRO_DC_3DC2 to OHM
            Naming Context: CN=Schema,CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:50:40.
            The last success occurred at 2014-07-03 06:56:45.
            8150 failures have occurred since the last success.
            [PRO_DC_3DC2] DsBindWithSpnEx() failed with error 1722,
            The RPC server is unavailable..
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From TESLA to OHM
            Naming Context: CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:48:55.
            The last success occurred at 2014-03-31 13:45:21.
            10400 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From MARCONI to OHM
            Naming Context: CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:49:16.
            The last success occurred at 2014-06-26 11:56:20.
            8313 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From PRO_DC_3DC2 to OHM
            Naming Context: CN=Configuration,DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:49:37.
            The last success occurred at 2014-07-03 06:56:03.
            8150 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From PRO_DC_3DC2 to OHM
            Naming Context: DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:48:34.
            The last success occurred at 2014-07-03 07:40:05.
            8150 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From TESLA to OHM
            Naming Context: DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:51:01.
            The last success occurred at 2014-03-31 14:18:48.
            10400 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,OHM] A recent replication attempt failed:
            From MARCONI to OHM
            Naming Context: DC=PRO_DC_3,DC=local
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2015-06-18 10:51:22.
            The last success occurred at 2014-06-26 12:20:27.
            8313 failures have occurred since the last success.
            The source remains down. Please check the machine.
         REPLICATION-RECEIVED LATENCY WARNING
         OHM:  Current time is 2015-06-18 11:30:14.
            DC=ForestDnsZones,DC=PRO_DC_3,DC=local
               Last replication recieved from MARCONI at 2014-06-26 11:57:03.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from TESLA at 2014-03-31 13:57:30.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
            DC=DomainDnsZones,DC=PRO_DC_3,DC=local
               Last replication recieved from MARCONI at 2014-06-26 11:57:03.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from TESLA at 2014-03-31 14:06:23.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
            CN=Schema,CN=Configuration,DC=PRO_DC_3,DC=local
               Last replication recieved from PRO_DC_3DC2 at 2014-07-03 06:56:45.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from MARCONI at 2014-06-26 11:56:20.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from TESLA at 2014-03-31 13:57:30.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
            CN=Configuration,DC=PRO_DC_3,DC=local
               Last replication recieved from PRO_DC_3DC2 at 2014-07-03 06:56:03.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from MARCONI at 2014-06-26 11:56:20.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from TESLA at 2014-03-31 13:57:30.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
            DC=PRO_DC_3,DC=local
               Last replication recieved from PRO_DC_3DC2 at 2014-07-03 07:40:04.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from MARCONI at 2014-06-26 12:20:27.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
               Last replication recieved from TESLA at 2014-03-31 14:18:48.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!
         ......................... OHM passed test Replications
      Starting test: NCSecDesc
         ......................... OHM passed test NCSecDesc
      Starting test: NetLogons
         ......................... OHM passed test NetLogons
      Starting test: Advertising
         ......................... OHM passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... OHM passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... OHM passed test RidManager
      Starting test: MachineAccount
         ......................... OHM passed test MachineAccount
      Starting test: Services
         ......................... OHM passed test Services
      Starting test: ObjectsReplicated
         ......................... OHM passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... OHM passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... OHM failed test frsevent
      Starting test: kccevent
         ......................... OHM passed test kccevent
      Starting test: systemlog
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:29:54
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:03
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:04
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:05
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:05
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:06
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:07
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 06/18/2015   11:30:08
            (Event String could not be retrieved)
         ......................... OHM failed test systemlog
      Starting test: VerifyReferences
         ......................... OHM passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : PRO_DC_3
      Starting test: CrossRefValidation
         ......................... PRO_DC_3 passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... PRO_DC_3 passed test CheckSDRefDom

   Running enterprise tests on : PRO_DC_3.local
      Starting test: Intersite
         ......................... PRO_DC_3.local passed test Intersite
      Starting test: FsmoCheck
         Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
         A Global Catalog Server could not be located - All GC's are down.
         ......................... PRO_DC_3.local failed test FsmoCheck
Matthew CioffiSenior DBAAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Seth SimmonsSr. Systems AdministratorCommented:
cleanup the server metadata for those missing domain controllers

Clean Up Server Metadata
https://technet.microsoft.com/en-us/library/cc816907%28v=ws.10%29.aspx

do netdom query fsmo on the existing domain controllers and make sure they reference the correct server
if any reference a domain controller that is gone, you need to seize the role(s) to an existing server

Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller
https://support.microsoft.com/en-us/kb/255504?wa=wsignin1.0

then run dcdiag again and see the results; should be a lot cleaner

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Will SzymkowskiSenior Solution ArchitectCommented:
A long with performing a meta database cleanup i would also suggest that you check your SRV records and AD Sites and Services.

In the DNS Console you need to expand the _msdcs.domain.com folder and go through all of them and make sure there are no references to domain controllers that are no longer present in your domain. If you do see SRV records that are present in this folder simply delete them.

You also need to open Active Directory Sites and Services and make sure that there are no replicating partners or DC computer objects that are referencing DC's that are no longer in your domain. If there are, then simply delete them.

I would also recommend running the following commands as well to check AD health during this process...

netdom query dc (this will show you all of the DC's that Active Directory is registering in the domain)
Readmin /replsum
Repadmin /showrepl
Repadmin /bridgeheads


As for your 2012 server into the domain you will require that your forest and domian functional level is set to 2003 before you can promote a 2012 DC. Make sure that AD health and replicaiton is correct before raising the functional level of the domain to 2003, if you haven't already.

Will.
Matthew CioffiSenior DBAAuthor Commented:
Thanks and sorry for the delay responding.

I ended up having a couple of additional issues, but worked them out.  The info here was a great help.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.