Link to home
Create AccountLog in
Avatar of richtree
richtree

asked on

How to repair Windows 2003 Activve Directory/DNS issue?

Environment:
single domain multiple site: windows 2003
domain function level: windows 2003
site Misi: 192.168.17.0
ms-dc-svr03: windows 2003 dc + dns + exchange 2003
ms-dns-svr2: windows 2000, dns only
ms-vmdc02: windows 2003 dc + dns;
ms-exch02: exchange 2003 only on windows 2003.

site Mon: 192.168.10.0
mn-dc-svr12: Windows 2003 dc + dns;


an external consultant did some changes remotely without my visual witness; but here are the changes that he claimed to make:
1) migrate all mailboxes from dc-svr03 to exch02;
2) remove exchange 2003 from dc-svr03;
3) demote dc-svr03
Original goal of his change is to retire ms-dc-svr03;


Issues:
1) domain user is unable to logon to ms-dc-svr03, not even to domain admin account who once logged onto this server;
2) ms-vmdc02 and mn-dc-svr12 is unable to resolve internal and external names;
3) domain users are unable to logon to ms-exch02; mail service could not start;
4) Outlook users are unable to connect to ms-exch02;

I guess the consultant did something wrong. For example, when he retire ms-dc-svr03, he is supposed to remove dns component with AD function. But right now I can still see DNS function in it even though there is no domain data.

Current Facts:
1) if the tcp/ip dns setting point to itself on ms-vmdc02: site Misi user is unable to login;
2) if the tcp/ip dns setting point to mn-dc-svr12: site Misi user is able to login; but Outlook still could not connect to ms-exch02;

Q#1. Is ms-vmdc02 corrupted? How to verify?
Q#2. Is mn-dc-svr12 still good? How to verify?
Q#3. What options do I have to correct the AD/DNS issue?

Thanks a lot.
SOLUTION
Avatar of Ajay Sharma
Ajay Sharma
Flag of India image

Link to home
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
See answer
Avatar of richtree
richtree

ASKER

Hi akicute555,
Thanks a lot for your response. Here is the info:

from ms-vmdc02: it shows it owns all 5 FSMO roles.
from mn-dc-svr12: 'ERROR' shows up in 'Operations master' field when it tries to query RID/PDC/Infrastructure master.
Before the change, ms-dc-svr03 is the primary DNS. But now ms-dc-svr03 has DNS snap-in with no content. Which might indicate the DNS was NOT removed when removing AD role.
Q#4. how to transfer (or force) these roles to mn-dc-svr12?
Q#5. how to verify mn-dc-svr12 is still a good domain controller?
Q#6. if mn-dc-svr12 is a good dc, how to rebuild exchange 2003 and restore the data (files such as priv1.edb) from ms-exch02?
SOLUTION
Link to home
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ChiefIT:
Right now:
ms-vmdc02 is NOT able to resolve internal and external hosts.
ms-dns-svr2 is able to resolve internal and external hosts.
mn-dc-svr12 is able to resolve internal and external hosts; also able to authenticate users.

There is no DCdiag program installed. Where to get it? How to install it?
Thanks.
SOLUTION
Link to home
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
Hi ChiefIT, here is the dcdiag info.
dcdiag /v:

Domain Controller Diagnosis

Performing initial setup:
   * Verifying that the local machine ms-vmdc02, is a DC.
   * Connecting to directory service on server ms-vmdc02.
   * Collecting site info.
   * Identifying all servers.
   * Identifying all NC cross-refs.
   * Found 5 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests
   
   Testing server: misi\ms-vmdc02
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         * Active Directory RPC Services Check
         ......................... ms-vmdc02 passed test Connectivity

Doing primary tests
   
   Testing server: misi\ms-vmdc02
      Starting test: Replications
         * Replications Check
         * Replication Latency Check
            DC=ForestDnsZones,DC=appliedbusiness,DC=corp
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DomainDnsZones,DC=appliedbusiness,DC=corp
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=appliedbusiness,DC=corp
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=appliedbusiness,DC=corp
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=appliedbusiness,DC=corp
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         * Replication Site Latency Check
         ......................... ms-vmdc02 passed test Replications
      Test omitted by user request: Topology
      Test omitted by user request: CutoffServers
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC ms-vmdc02.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=appliedbusiness,DC=corp
            (NDNC,Version 2)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=appliedbusiness,DC=corp
            (NDNC,Version 2)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=appliedbusiness,DC=corp
            (Schema,Version 2)
         * Security Permissions Check for
           CN=Configuration,DC=appliedbusiness,DC=corp
            (Configuration,Version 2)
         * Security Permissions Check for
           DC=appliedbusiness,DC=corp
            (Domain,Version 2)
         ......................... ms-vmdc02 passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\ms-vmdc02\netlogon
         Verified share \\ms-vmdc02\sysvol
         ......................... ms-vmdc02 passed test NetLogons
      Starting test: Advertising
         The DC ms-vmdc02 is advertising itself as a DC and having a DS.
         The DC ms-vmdc02 is advertising as an LDAP server
         The DC ms-vmdc02 is advertising as having a writeable directory
         The DC ms-vmdc02 is advertising as a Key Distribution Center
         The DC ms-vmdc02 is advertising as a time server
         ......................... ms-vmdc02 passed test Advertising
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Domain Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role PDC Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Rid Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         ......................... ms-vmdc02 passed test KnowsOfRoleHolders
      Starting test: RidManager
         * Available RID Pool for the Domain is 7103 to 1073741823
         * ms-vmdc02.appliedbusiness.corp is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 6603 to 7102
         * rIDPreviousAllocationPool is 6603 to 7102
         * rIDNextRID: 6606
         ......................... ms-vmdc02 passed test RidManager
      Starting test: MachineAccount
         Checking machine account for DC ms-vmdc02 on DC ms-vmdc02.
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp
         * SPN found :LDAP/ms-vmdc02
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp/appliedbusiness
         * SPN found :LDAP/b903ab73-3c86-42db-b3d0-298d4a253334._msdcs.appliedbusiness.corp
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/b903ab73-3c86-42db-b3d0-298d4a253334/appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp/appliedbusiness
         * SPN found :GC/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         ......................... ms-vmdc02 passed test MachineAccount
      Starting test: Services
         * Checking Service: Dnscache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: RpcSs
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... ms-vmdc02 passed test Services
      Test omitted by user request: OutboundSecureChannels
      Starting test: ObjectsReplicated
         ms-vmdc02 is in domain DC=appliedbusiness,DC=corp
         Checking for CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp in domain DC=appliedbusiness,DC=corp on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp in domain CN=Configuration,DC=appliedbusiness,DC=corp on 1 servers
            Object is up-to-date on all servers.
         ......................... ms-vmdc02 passed test ObjectsReplicated
      Starting test: frssysvol
         * The File Replication Service SYSVOL ready test
         File Replication Service's SYSVOL is ready
         ......................... ms-vmdc02 passed test frssysvol
      Starting test: frsevent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/07/2011   20:17:17
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/07/2011   20:17:17
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/07/2011   20:17:17
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/07/2011   20:17:19
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test frsevent
      Starting test: kccevent
         * The KCC Event log test
         Found no KCC errors in Directory Service Event log in the last 15 minutes.
         ......................... ms-vmdc02 passed test kccevent
      Starting test: systemlog
         * The System Event log test
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 10/08/2011   09:33:27
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 10/08/2011   09:33:49
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test systemlog
      Test omitted by user request: VerifyReplicas
      Starting test: VerifyReferences
         The system object reference (serverReference)

         CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp and backlink

         on

         CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp

         are correct.
         The system object reference (frsComputerReferenceBL)

         CN=ms-vmdc02,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=appliedbusiness,DC=corp

         and backlink on

         CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp are correct.
         The system object reference (serverReferenceBL)

         CN=ms-vmdc02,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=appliedbusiness,DC=corp

         and backlink on

         CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp

         are correct.
         ......................... ms-vmdc02 passed test VerifyReferences
      Test omitted by user request: VerifyEnterpriseReferences
      Test omitted by user request: CheckSecurityError
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : appliedbusiness
      Starting test: CrossRefValidation
         ......................... appliedbusiness passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... appliedbusiness passed test CheckSDRefDom
   
   Running enterprise tests on : appliedbusiness.corp
      Starting test: Intersite
         Skipping site Edmonton, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site Vancouver, this site is outside the scope provided by

         the command line arguments provided.
         Skipping site Montreal, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site Calgary, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site misi, this site is outside the scope provided by

         the command line arguments provided.
         ......................... appliedbusiness.corp passed test Intersite
      Starting test: FsmoCheck
         Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
         A Global Catalog Server could not be located - All GC's are down.
         PDC Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003f9
         Time Server Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003f9
         Preferred Time Server Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003f9
         KDC Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003f9
         ......................... appliedbusiness.corp failed test FsmoCheck
      Test omitted by user request: DNS
      Test omitted by user request: DNS
ms-dns-svr2 has ip 192.168.117.4

dcdiag /test:dns


Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: misi\ms-vmdc02
      Starting test: Connectivity
         ......................... ms-vmdc02 passed test Connectivity

Doing primary tests
   
   Testing server: misi\ms-vmdc02

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : appliedbusiness
   
   Running enterprise tests on : appliedbusiness.corp
      Starting test: DNS
         Test results for domain controllers:
           
            DC: ms-vmdc02.appliedbusiness.corp
            Domain: appliedbusiness.corp

                 
               TEST: Delegations (Del)
                  Error: DNS server: ms-dc-svr03.appliedbusiness.corp. IP:192.168.117.5 [Broken delegated domain _msdcs.appliedbusiness.corp.]
                 
               TEST: Records registration (RReg)
                  Network Adapter [00000001] Intel(R) PRO/1000 MT Network Connection:
                     Error: Missing CNAME record at DNS server 192.168.117.4 :
                     b903ab73-3c86-42db-b3d0-298d4a253334._msdcs.appliedbusiness.corp
                     
                     Error: Missing DC SRV record at DNS server 192.168.117.4 :
                     _ldap._tcp.dc._msdcs.appliedbusiness.corp
                     
                     Error: Missing PDC SRV record at DNS server 192.168.117.4 :
                     _ldap._tcp.pdc._msdcs.appliedbusiness.corp
                     
               Error: Record registrations cannot be found for all the network adapters
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 192.168.117.5 (ms-dc-svr03.appliedbusiness.corp.)
               1 test failure on this DNS server
               Delegation is broken for the domain _msdcs.appliedbusiness.corp. on the DNS server 192.168.117.5
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: appliedbusiness.corp
               ms-vmdc02                  PASS PASS PASS FAIL PASS FAIL n/a  
         
         ......................... appliedbusiness.corp failed test DNS
ASKER CERTIFIED SOLUTION
Link to home
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
Thank you so much for your responses. The issue is resolved now by the consultant.But I do not know the details. He ran dcpromo a few times on ms-dc-svr03 and run replication a few times.

Thanks again.
STILL, be aware of the metadata cleanup and proper replications across domain controllers, REVIEW his/her work by running DCdiag on all DCs. If this is wrong and replications continue to be a problem, then you will eventually tombstone a DC.
Hi ChiefIT,

Thanks a lot for your advice.
Would you please review the current diag and let me know any issues and how to fix it?

Domain Controller Diagnosis

Performing initial setup:
   * Verifying that the local machine ms-vmdc02, is a DC.
   * Connecting to directory service on server ms-vmdc02.
   * Collecting site info.
   * Identifying all servers.
   * Identifying all NC cross-refs.
   * Found 5 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Misi\ms-vmdc02
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         * Active Directory RPC Services Check
         ......................... ms-vmdc02 passed test Connectivity

Doing primary tests
   
   Testing server: Misi\ms-vmdc02
      Starting test: Replications
         * Replications Check
         * Replication Latency Check
            DC=ForestDnsZones,DC=appliedbusiness,DC=corp
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DomainDnsZones,DC=appliedbusiness,DC=corp
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=appliedbusiness,DC=corp
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=appliedbusiness,DC=corp
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=appliedbusiness,DC=corp
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         * Replication Site Latency Check
         ......................... ms-vmdc02 passed test Replications
      Test omitted by user request: Topology
      Test omitted by user request: CutoffServers
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC ms-vmdc02.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=appliedbusiness,DC=corp
            (NDNC,Version 2)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=appliedbusiness,DC=corp
            (NDNC,Version 2)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=appliedbusiness,DC=corp
            (Schema,Version 2)
         * Security Permissions Check for
           CN=Configuration,DC=appliedbusiness,DC=corp
            (Configuration,Version 2)
         * Security Permissions Check for
           DC=appliedbusiness,DC=corp
            (Domain,Version 2)
         ......................... ms-vmdc02 passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\ms-vmdc02\netlogon
         Verified share \\ms-vmdc02\sysvol
         ......................... ms-vmdc02 passed test NetLogons
      Starting test: Advertising
         The DC ms-vmdc02 is advertising itself as a DC and having a DS.
         The DC ms-vmdc02 is advertising as an LDAP server
         The DC ms-vmdc02 is advertising as having a writeable directory
         The DC ms-vmdc02 is advertising as a Key Distribution Center
         The DC ms-vmdc02 is advertising as a time server
         The DS ms-vmdc02 is advertising as a GC.
         ......................... ms-vmdc02 passed test Advertising
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Domain Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role PDC Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Rid Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp
         ......................... ms-vmdc02 passed test KnowsOfRoleHolders
      Starting test: RidManager
         * Available RID Pool for the Domain is 7603 to 1073741823
         * ms-vmdc02.appliedbusiness.corp is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 6603 to 7102
         * rIDPreviousAllocationPool is 6603 to 7102
         * rIDNextRID: 6606
         ......................... ms-vmdc02 passed test RidManager
      Starting test: MachineAccount
         Checking machine account for DC ms-vmdc02 on DC ms-vmdc02.
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp
         * SPN found :LDAP/ms-vmdc02
         * SPN found :LDAP/ms-vmdc02.appliedbusiness.corp/appliedbusiness
         * SPN found :LDAP/b903ab73-3c86-42db-b3d0-298d4a253334._msdcs.appliedbusiness.corp
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/b903ab73-3c86-42db-b3d0-298d4a253334/appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp
         * SPN found :HOST/ms-vmdc02
         * SPN found :HOST/ms-vmdc02.appliedbusiness.corp/appliedbusiness
         * SPN found :GC/ms-vmdc02.appliedbusiness.corp/appliedbusiness.corp
         ......................... ms-vmdc02 passed test MachineAccount
      Starting test: Services
         * Checking Service: Dnscache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: RpcSs
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... ms-vmdc02 passed test Services
      Test omitted by user request: OutboundSecureChannels
      Starting test: ObjectsReplicated
         ms-vmdc02 is in domain DC=appliedbusiness,DC=corp
         Checking for CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp in domain DC=appliedbusiness,DC=corp on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp in domain CN=Configuration,DC=appliedbusiness,DC=corp on 1 servers
            Object is up-to-date on all servers.
         ......................... ms-vmdc02 passed test ObjectsReplicated
      Starting test: frssysvol
         * The File Replication Service SYSVOL ready test
         File Replication Service's SYSVOL is ready
         ......................... ms-vmdc02 passed test frssysvol
      Starting test: frsevent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/08/2011   10:47:23
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/08/2011   10:55:23
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C5
            Time Generated: 10/08/2011   10:57:19
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test frsevent
      Starting test: kccevent
         * The KCC Event log test
         Found no KCC errors in Directory Service Event log in the last 15 minutes.
         ......................... ms-vmdc02 passed test kccevent
      Starting test: systemlog
         * The System Event log test
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:12
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:12
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:13
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:23
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:23
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test systemlog
      Test omitted by user request: VerifyReplicas
      Starting test: VerifyReferences
         The system object reference (serverReference)

         CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp and backlink

         on

         CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp

         are correct.
         The system object reference (frsComputerReferenceBL)

         CN=ms-vmdc02,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=appliedbusiness,DC=corp

         and backlink on

         CN=ms-vmdc02,OU=Domain Controllers,DC=appliedbusiness,DC=corp are correct.
         The system object reference (serverReferenceBL)

         CN=ms-vmdc02,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=appliedbusiness,DC=corp

         and backlink on

         CN=NTDS Settings,CN=ms-vmdc02,CN=Servers,CN=Misi,CN=Sites,CN=Configuration,DC=appliedbusiness,DC=corp

         are correct.
         ......................... ms-vmdc02 passed test VerifyReferences
      Test omitted by user request: VerifyEnterpriseReferences
      Test omitted by user request: CheckSecurityError
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : appliedbusiness
      Starting test: CrossRefValidation
         ......................... appliedbusiness passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... appliedbusiness passed test CheckSDRefDom
   
   Running enterprise tests on : appliedbusiness.corp
      Starting test: Intersite
         Skipping site Edmonton, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site Vancouver, this site is outside the scope provided by

         the command line arguments provided.
         Skipping site Montreal, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site Calgary, this site is outside the scope provided by the

         command line arguments provided.
         Skipping site Misi, this site is outside the scope provided by

         the command line arguments provided.
         ......................... appliedbusiness.corp passed test Intersite
      Starting test: FsmoCheck
         GC Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003fd
         PDC Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003fd
         Time Server Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003fd
         Preferred Time Server Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003fd
         KDC Name: \\ms-vmdc02.appliedbusiness.corp
         Locator Flags: 0xe00003fd
         ......................... appliedbusiness.corp passed test FsmoCheck
      Test omitted by user request: DNS
      Test omitted by user request: DNS

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests
   
   Testing server: Misi\ms-vmdc02
      Starting test: Connectivity
         ......................... ms-vmdc02 passed test Connectivity

Doing primary tests
   
   Testing server: Misi\ms-vmdc02

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
   
   Running partition tests on : DomainDnsZones
   
   Running partition tests on : Schema
   
   Running partition tests on : Configuration
   
   Running partition tests on : appliedbusiness
   
   Running enterprise tests on : appliedbusiness.corp
      Starting test: DNS
         Test results for domain controllers:
           
            DC: ms-vmdc02.appliedbusiness.corp
            Domain: appliedbusiness.corp

                 
               TEST: Basic (Basc)
                  Warning: adapter [00000001] Intel(R) PRO/1000 MT Network Connection has invalid DNS server: 192.168.117.5 (ms-dc-svr03.appliedbusiness.corp.)
                  Warning: adapter [00000001] Intel(R) PRO/1000 MT Network Connection has invalid DNS server: 192.168.100.2 (<name unavailable>)
                 
               TEST: Delegations (Del)
                  Error: DNS server: ms-dc-svr03.appliedbusiness.corp. IP:192.168.117.5 [Broken delegated domain _msdcs.appliedbusiness.corp.]
                 
               TEST: Records registration (RReg)
                  Network Adapter [00000001] Intel(R) PRO/1000 MT Network Connection:
                     Error: Missing CNAME record at DNS server 192.168.117.4 :
                     b903ab73-3c86-42db-b3d0-298d4a253334._msdcs.appliedbusiness.corp
                     
                     Error: Missing DC SRV record at DNS server 192.168.117.4 :
                     _ldap._tcp.dc._msdcs.appliedbusiness.corp
                     
                     Error: Missing GC SRV record at DNS server 192.168.117.4 :
                     _ldap._tcp.gc._msdcs.appliedbusiness.corp
                     
                     Error: Missing PDC SRV record at DNS server 192.168.117.4 :
                     _ldap._tcp.pdc._msdcs.appliedbusiness.corp
                     
                     Error: Missing A record at DNS server 192.168.117.5 :
                     ms-vmdc02.appliedbusiness.corp
                     
                     Error: Missing CNAME record at DNS server 192.168.117.5 :
                     b903ab73-3c86-42db-b3d0-298d4a253334._msdcs.appliedbusiness.corp
                     
                     Error: Missing DC SRV record at DNS server 192.168.117.5 :
                     _ldap._tcp.dc._msdcs.appliedbusiness.corp
                     
                     Error: Missing GC SRV record at DNS server 192.168.117.5 :
                     _ldap._tcp.gc._msdcs.appliedbusiness.corp
                     
                     Error: Missing PDC SRV record at DNS server 192.168.117.5 :
                     _ldap._tcp.pdc._msdcs.appliedbusiness.corp
                     
               Error: Record registrations cannot be found for all the network adapters
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 192.168.117.5 (ms-dc-svr03.appliedbusiness.corp.)
               2 test failures on this DNS server
               Name resolution is not functional. _ldap._tcp.appliedbusiness.corp. failed on the DNS server 192.168.117.5
               Delegation is broken for the domain _msdcs.appliedbusiness.corp. on the DNS server 192.168.117.5
               
            DNS server: 192.168.100.2 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.168.100.2
               Name resolution is not functional. _ldap._tcp.appliedbusiness.corp. failed on the DNS server 192.168.100.2
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: appliedbusiness.corp
               ms-vmdc02                  PASS WARN PASS FAIL PASS FAIL n/a  
         
         ......................... appliedbusiness.corp failed test DNS
Looks like your FRS and system event logs show errors, as seen below. You might delete these logs and watch them for errors. You should make absolute sure that you are replicating between DCs good.....

Starting test: frsevent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/08/2011   10:47:23
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 10/08/2011   10:55:23
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x800034C5
            Time Generated: 10/08/2011   10:57:19
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test frsevent
      Starting test: kccevent
         * The KCC Event log test
         Found no KCC errors in Directory Service Event log in the last 15 minutes.
         ......................... ms-vmdc02 passed test kccevent
      Starting test: systemlog
         * The System Event log test
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:12
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:12
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:13
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:22
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:23
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 10/08/2011   15:14:23
            (Event String could not be retrieved)
         ......................... ms-vmdc02 failed test systemlog
any other issue?
Sorry to say this, but the contractor didn't fix anything... Do you see the DNS metadata left within DNS on the DCdiag /test:DNS test? This will cause replication problems. The five steps above should still be followed or you will eventually tombstone a server and have domain problems.
Thanks a lot. I will follow it through and post it separately.