Link to home
Start Free TrialLog in
Avatar of FocIS
FocIS

asked on

Active Directory not functioning 100%, can't add new DC

Please don't post the first google search results you find, we're looking for a little more personal of a diagnosis and answer.

The scope of this question is aimed at being able to add an additional domain controller to our existing domain.

This is a single domain in a single forest.  We have two domain controllers (GC) that seem to be functional for the most part but some things seem haunted.  The domain is at "Server 2008" functional level.  One domain controller is server 2008, the other is server 2008 R2.

When trying to dcpromo a member server up to a DC, it asks for credentials over and over with "The operation failed because a domain controller could not be contacted for the domain domain.com that contained an account for this computer.  make the computer a member of a workgroup then rejoin the domain before retrying the promotion - access is denied".  Behind the scenes, the debug log shows "Failed to find a DC for domain domain.com"

Note that it doesn't necessarily matter WHICH server we try to dcpromo up, ALL of them have the same issue as above... we have disjoined/renamed/rejoined one of them several times with no change.

I have attached a text document with ipconfig /all from each server, dcdiag's from each, repadmin's from each, the dcpromo debug log, and netdom query fsmo.

Some other weird symptoms that may or may not be related:
- a newly created VM server fully a domain member refuses to get any group policies applied
- sometimes users who remote desktop to server.domain.com get error messages locally about the server not containing a workstation entry for their local machine (which it shouldnt), other times it just says "the logon attempt failed".  BUT if they remote desktop to the public ip address, it works.  these workstations are NOT on the local network of the servers, they are across the internet coming in public.
- one GPO repeatedly fails as 'invalid entry', while it worked fine for years
- every 15 minutes or so, many kerberos errors in the system event logs, regarding preauth failure
diags.txt
Avatar of Jeff Perry
Jeff Perry
Flag of United States of America image

First have you removed the server you are trying to promote from the domain and added it back again?

Second after step 1 from the same server run :

dcdiag /test:dcpromo /DnsDomain:<your domain> /ReplicaDC

Open in new window

First place to start:
DNS 0 on the DC should point to self in the NIC settings.
DNS 1 on the DC should be _blank_ meaning no IP there.

In an Active Directory integrated setup (AD/DNS/DHCP) the DC will use AD to figure out where the DCs/DNS/DHCP servers are.

Remove the secondary DNS IP on the DC's NIC.

Second: Is replication healthy between the two existing DCs? It looks like it is not.

Please check the Event Logs on both servers for more information on what is failing.

Philip
I think you need to step back to ensure DNS resolution is working correct and it is able to be ping first.

and are they on the same site?
Avatar of FocIS
FocIS

ASKER

jeff - thanks for the reply

we have disjoined/renamed/rejoined that server several times but no affect on the dcpromo

the dcdiag results from that server 3 are here:

C:\>dcdiag /test:dcpromo /dnsdomain:domain.com /replicadc
   Starting test: DcPromo
      The DNS configuration is sufficient to allow this computer to be promoted
      as a replica domain controller in the domain.com domain.

      Messages logged below this line indicate whether this domain controller
      will be able to dynamically register DNS records required for the
      location of this DC by other devices on the network. If any
      misconfiguration is detected, it might prevent dynamic DNS registration
      of some records, but does not prevent successful completion of the Active
      Directory Domain Services Installation Wizard. However, we recommend
      fixing the reported problems now, unless you plan to manually update the
      DNS database.

      DNS configuration is sufficient to allow this domain controller to
      dynamically register the domain controller Locator records in DNS.

      The DNS configuration is sufficient to allow this computer to dynamically
      register the A record corresponding to its DNS name.

      ......................... abcsrvr passed test DcPromo

C:\>
Avatar of FocIS

ASKER

justin - thanks for the reply

as far as i can tell dns appears to be well.

ping domain.com  - resolves to the internal ip of the main DC

there are two sites, "datacenter" and "ssc" - although "ssc" is now empty.  all servers in the equation are absolutely in the "datacenter" site.

how else should i test dns beyond what's in the diagnostics txt attachment?

dc1 has dc1 as primary dns, dc2 as secondary
dc2 has dc2 as primary dns, dc1 as secondary
server3 has dc1 as primary, dc2 as secondary
Avatar of FocIS

ASKER

phillip - good suggestions, i'll remove the secondary dns from the nic on each DC right now, will wait 15-30 mins and try again

as you saw in the attached txt file, it appears that replication IS WORKING fine as there are no errors and each dc repadmin indicated success
Are both your current domain controllers Global Catalog servers?
Avatar of FocIS

ASKER

jeff - both are both marked as GC yes
check with DNS health see if any issue.

remember AD / DNS are chicken and Egg. they are rely on each others

DCDIAG /TEST:DNS

Justin
Avatar of FocIS

ASKER

from my own googling, i think the next step might be to delete the partitions in DNS and let them be recreated, but that seems like such a dangerous thing to do i haven't yet considered it.
Avatar of FocIS

ASKER

justin - here are the results from dcdiag /test:dns on each GC:

----------------
server1:
----------------

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: abc-Cleveland-Datacenter\SC01
      Starting test: Connectivity
         ......................... SC01 passed test Connectivity

Doing primary tests

   Testing server: abc-Cleveland-Datacenter\SC01

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : domain

   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:

            DC: sc01.domain.com
            Domain: domain.com


               TEST: Forwarders/Root hints (Forw)
                  Error: Forwarders list has invalid forwarder: 8.8.8.8 (<name u
navailable>)

               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure
domain.com.

         Summary of test results for DNS servers used by the above domain contro
llers:

            DNS server: 8.8.8.8 (<name unavailable>)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 8.8.8.8

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: domain.com
               sc01                         PASS PASS FAIL PASS WARN PASS n/a

         ......................... domain.com failed test DNS

----------------
server2:
----------------
Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = ts10
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: abc-Cleveland-Datacenter\TS10
      Starting test: Connectivity
         ......................... TS10 passed test Connectivity

Doing primary tests

   Testing server: abc-Cleveland-Datacenter\TS10

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...
         ......................... TS10 passed test DNS

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : domain

   Running enterprise tests on : domain.com
      Starting test: DNS
         Test results for domain controllers:

            DC: ts10.domain.com
            Domain: domain.com


               TEST: Dynamic update (Dyn)
                  Warning: Failed to delete the test record dcdiag-test-record i
n zone domain.com

               ts10                         PASS PASS PASS PASS WARN PASS n/a
         ......................... domain.com passed test DNS

Doing primary tests
   
   Testing server: abc-Cleveland-Datacenter\SC01
      Starting test: Replications
         ......................... SC01 passed test Replications
      Starting test: NCSecDesc
         ......................... SC01 passed test NCSecDesc
      Starting test: NetLogons
         ......................... SC01 passed test NetLogons
      Starting test: Advertising
         ......................... SC01 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... SC01 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... SC01 passed test RidManager
      Starting test: MachineAccount
         ......................... SC01 passed test MachineAccount
      Starting test: Services
         ......................... SC01 passed test Services
      Starting test: ObjectsReplicated
         ......................... SC01 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... SC01 passed test frssysvol
      Starting test: frsevent
         ......................... SC01 passed test frsevent
      Starting test: kccevent
         An Information Event occured.  EventID: 0x40000456
            Time Generated: 02/24/2014   13:05:45
            Event String: Promotion of this domain controller to a global         An Warning Event occured.  EventID: 0x80000709
            Time Generated: 02/24/2014   13:05:45
            Event String: The partition         An Warning Event occured.  EventID: 0x80000709
            Time Generated: 02/24/2014   13:05:45
            Event String: The partition         An Warning Event occured.  EventID: 0x80000709
            Time Generated: 02/24/2014   13:08:59
            Event String: The partition         An Warning Event occured.  EventID: 0x80000709
            Time Generated: 02/24/2014   13:08:59
            Event String: The partition         ......................... SC01 failed test kccevent

Are you sure?

Philip
Avatar of FocIS

ASKER

phillip - i was basing that off of:

repadmin /syncall from server 1
-----------------------------------------------------------
CALLBACK MESSAGE: The following replication is in progress:    
      From: a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com    
      To  : ed6f34ed-a7b8-4dac-94b3-5df27303d73e._msdcs.domain.com
CALLBACK MESSAGE: The following replication completed successfully:    
      From: a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com    
      To  : ed6f34ed-a7b8-4dac-94b3-5df27303d73e._msdcs.domain.com
CALLBACK MESSAGE: SyncAll Finished.SyncAll terminated with no errors.

--------------------------------------------------------------
repadmin /syncall from server 2
--------------------------------------------------------------
CALLBACK MESSAGE: The following replication is in progress:
    From: ed6f34ed-a7b8-4dac-94b3-5df27303d73e._msdcs.domain.com
    To  : a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com
CALLBACK MESSAGE: The following replication completed successfully:
    From: ed6f34ed-a7b8-4dac-94b3-5df27303d73e._msdcs.domain.com
    To  : a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.

but - i'm here for your help so, tell me what you'd like to see or have done
Have a look in the SC1 Event Logs for the replication errors indicated.


Starting test: kccevent
         An Information Event occured.  EventID: 0x40000456
            Time Generated: 02/24/2014   13:05:45
            Event String: Promotion of this domain controller to a global         An Warning Event occured.  EventID: 0x80000709

Philip
Please check the following reg key on both existing DC's for value (6).

HKEY_Local_Machine\System\CurrentControlSet\Services\NTDS\Parameters\ Global Catalog Partition Occupancy  

This is from the referenced link - section "Requirements for Global Catalog Readiness"

http://technet.microsoft.com/en-us/library/how-global-catalog-servers-work(v=ws.10).aspx

Your kcc event warning from your SC01 server is what pointed me to this.

It looks as though you have redacted your log maybe I can't tell for sure.

Try removing the gc setting for sc01, and see if there is a difference in the promo.
confirm all the FSMO role are on correct location

netdom query fsmo
Does Secure Update is enabled in domain DNS zone properties ?

Are you able to view correct NS records in DNS zones with correct IP address ?

If you deleted host record from DNS and run ipconfig /registerdns, does it registers records in DNS ?

Are you able to view CNAME records in _Msdcs.domain.com zone and if you ping them, are they able to resolve to their actual IP address ?
If not here you need to start 1st by copying CNAME record from AD sites\servers\servername\ntds setting properties \ general tab for each DC

verify your SRV records as per below article
http://support.microsoft.com/kb/816587

Also you can try with uninstalling and reinstalling DNS service on both domain controllers one by one

Also you can rename netlogon.dns file located at %systemroot%\system32 and then restart netlogon service on both DCs and check if this resolves your problem

Try resetting Domain Controller account password with netdom utility one by one
http://support.microsoft.com/kb/325850

Also can you post output for repadmin /showrepl here please

Mahesh
Avatar of FocIS

ASKER

sorry i haven't responded in a few hours, here are the updates:
-------------------
phillip - i see what you mean there, here is the full text of that event viewer log - which by the way is repeating every 15 minutes:

The partition DC=ForestDnsZones,DC=domain,DC=com should be hosted at site CN=abc-Cleveland-Datacenter,CN=Sites,CN=Configuration,DC=domain,DC=com, but has not been instantiated yet. However, the KCC could not find any hosts from which to replicate this partition.

--------------------
jeff - on sc01 (the one i tried to promote to GC, has always been a DC), "Global Catalog Partition Occupancy" does not exist.  However, in the same key, "Global Catalog Promotion Complete" is set to "1".  

On ts10 (has always been a GC), the keys are identical to sc01 (no "occupancy" key, "complete" set to 1)

earlier today sc01 was not a GC, i put the checkmark there and that caused the event viewer log at just after 1pm.  The only things i'm changing in my posts are the name of the domain to "domain" and the name of the company to "abc".

promoting a third server yesterday, when sc01 was not a GC, failed the same way... i just added sc01 to be a GC just to see if it would work.  i can remove it and try again but it will certainly fail

---------------------
justin - i've included the results of netdom query fsmo, in the original attachment when i opened the question.  they are all on ts10 as expected
---------------------
mahesh:
in dns on my forward lookup, domain.com, "dynamic updates" is set to "nonsecure and secure" on both servers (ts10 and sc01)

NOTE: in the "name servers" tab i did find a reference to an old non-existing server, i removed that just now.

"view correct ns records in dns zones with correct ip addresses" - they look correct to me - references are to ts10 and sc01 with the correct ip addresses

delete and registerdns: i deleted the A record for ts10 and the A record for sc01 - and then ran registerdns from each server.  3 minutes later they both came back.

cname records in msdcs: i see the two guid's which are referenced in my repadmin results, listed as cnames.  
ping a194d60b-0df4-4f0d-9e9d-95278a32dfb9 - can't find host
ping a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com = cant find host
(not sure if they should resolve?)  


pinging the fqdn sc01.domain.com does of course reply with the correct ip address.

note:  in AD sites and services, the guid listed for each server's ntds settings are IDENTICAL to the cnames, except in ntds they are all caps (not that that matters)

verify SRV records: dns gui looks perfect.  nslookup shows expected on both dns servers.

rename netlogon.dns:  i found it in system32\config - on each server, renamed and bounced the netlogon service on each server.   trying to promote a 3rd server failed with the same results as the original post.

reset DC account passwords one at a time:  this looks very logical and reasonable but first can you please explain the dangers and things NOT to do?  this seems like it could have very drastic consequences if something goes wrong here.


uninstalled the dns role from sc01, rebooted sc01.  reinstalled dns role on sc01, rebooted sc01.  attempted to promote server3, fails the same way.

uninstalled the dns role from ts10, rebooted ts10.  reinstalled dns role on ts10, rebooted ts10.  attempted to promote server3, fails the same way

repadmin /showrepl - as seen from ts10:
C:\>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
abc-Cleveland-Datacenter\TS10
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: ed6f34ed-a7b8-4dac-94b3-5df27303d73e
DSA invocationID: 56db89cb-58c9-4578-8c33-509db054a724

==== INBOUND NEIGHBORS ======================================

DC=domain,DC=com
    abc-Cleveland-Datacenter\SC01 via RPC
        DSA object GUID: a194d60b-0df4-4f0d-9e9d-95278a32dfb9
        Last attempt @ 2014-02-24 20:25:40 was successful.

CN=Configuration,DC=domain,DC=com
    abc-Cleveland-Datacenter\SC01 via RPC
        DSA object GUID: a194d60b-0df4-4f0d-9e9d-95278a32dfb9
        Last attempt @ 2014-02-24 19:53:25 was successful.

CN=Schema,CN=Configuration,DC=domain,DC=com
    abc-Cleveland-Datacenter\SC01 via RPC
        DSA object GUID: a194d60b-0df4-4f0d-9e9d-95278a32dfb9
        Last attempt @ 2014-02-24 19:53:25 was successful.
=========
repadmin /showrepl as seen from sc01:
C:\>repadmin /showrepl

repadmin running command /showrepl against server localhost

abc-Cleveland-Datacenter\SC01
DC Options: IS_GC
Site Options: (none)
DC object GUID: a194d60b-0df4-4f0d-9e9d-95278a32dfb9
DC invocationID: 5dffe497-0f5f-4913-a328-edcdd2672f67

==== INBOUND NEIGHBORS ======================================

DC=domain,DC=com
    abc-Cleveland-Datacenter\TS10 via RPC
        DC object GUID: ed6f34ed-a7b8-4dac-94b3-5df27303d73e
        Last attempt @ 2014-02-24 20:28:31 was successful.

CN=Configuration,DC=domain,DC=com
    abc-Cleveland-Datacenter\TS10 via RPC
        DC object GUID: ed6f34ed-a7b8-4dac-94b3-5df27303d73e
        Last attempt @ 2014-02-24 20:26:45 was successful.

CN=Schema,CN=Configuration,DC=domain,DC=com
    TRN-Cleveland-Datacenter\TS10 via RPC
        DC object GUID: ed6f34ed-a7b8-4dac-94b3-5df27303d73e
        Last attempt @ 2014-02-24 19:48:58 was successful.

---------------------------

as a side note - attempting to dcpromo server3 (or any other server) it hangs at examining dns configuration, for a good 5 minutes or so.  in my experience this isn't typical

another side note - just before dcpromo pops up the authentication box (which it shouldn't) the step just before that shows "checking if group policy management needs to be installed" (it blinked by for 0.5 seconds before the password prompt)

the same failures promoting server3 happen on randomserver4 too

SIDE NOTE:  after "additional domain controller options", a warning popup "a delegation for this dns server cannot be created because the authoritative parent zone cannot be found or it does not run windows dns server.  if you are integrating with an existing dns infrastructure, you should manually create a delegation to this dns server in the parent zone to ensure reliable name resolution from outside the domain domain.com.  otherwise, no action is required.  continue y/n"
Avatar of FocIS

ASKER

UPDATE: for fun i just tried to dcpromo an "old" server and it actually worked - but not on the most recent two servers we joined to the domain...  

weird since they're all in the same forest/domain/site, same physical switch in the same lan in the same cabinet with the same dns servers

hmmmmmm
It looks like the ForestDnsZones and DomainDnsZones are corrupted or having issue. You will need to delete the application partitions and recreate it

http://eventid.net/display.asp?eventid=1801&eventno=5096&source=NTDS%20KCC&phase=1

Before you delete the application partitions, make sure you take the backup of the server and proceed.
Troubleshoot replication:

Ultrasound - Monitoring and Troubleshooting Tool for File Replication Service (FRS) http://bit.ly/1fA8er7

Microsoft IT Environment Health Scanner: http://bit.ly/1civDbT

Those are two good places to start.

Philip
ping a194d60b-0df4-4f0d-9e9d-95278a32dfb9._msdcs.domain.com

This must resolves to ip address of corresponding DC.

If this failed means they are pointing to non existent DC and hence your current DC server cannot be identified

Just go to Ad sites and services\sitename\servers\ntds settings properties, on general tab you could find correct GUID, just copy that one and replace _msdcs guids with that one and then try to ping, it should work then
follow this process for both DCs

Since you are running DC promo in single domain single forest, you can simply ignore delegation warning message. It is by design. You can click yes to continue

Also run ntdsutil command to view metadata to find out any orphaned servers in active directory, if you find any one just remove that.
Ntdsutil will provide you every command that needs to be run by typing ? and hitting enter

Resetting DC password as mentioned in KB article will not make any harm, it is standard method and will help to reset DC secure channel which might helps some times to resolve issues.

Just check FRS events for 13568 - if it found then probably your FRS is gone in Journal wrap. But I don't think you will find that, let us know if you found that
Then you need to follow below KB article
http://support.microsoft.com/kb/290762

Are you able to view netlogon and Sysvol shared if you run net share command through cmd on both DCs ?

Mahesh
Avatar of FocIS

ASKER

justin - you suggested deleting the application partition and recreating it.  i have uninstalled and reinstalled dns on both domain controllers, does that take care of it?  i see the link but it still seems kind of dangerous... i'd rather hold off on this step but i'm not ruling it out

phillip - those tools look interesting, i will run them later today and post back results

mahesh - pinging that guid as found in dns does NOT resolve from either dns server, but the GUID from dns DOES EXACTLY MATCH the one found in NTDS - what should i do there if they already match but dont resolve?

ntdsutil - looking in both sites, list servers in site, only the expected servers are there

net share - from each DC i see netlogon and sysvol... i've also followed a previous KB to  make sure the permissions on those shares are correct

one thing to note which i think may really help - the servers i've been trying to dcpromo (server3 and server4) are both relatively new to the domain, they don't get group policies but they should... and an "old" server was able to dcpromo just fine last night.  besides the age difference in the servers, there are differences in physical versus vmware:
sc01 (current dc/gc): vmware
ts10 (current dc/gc): vmware
server3 (attempting to dcpromo): vmware
server4 (attempting to dcpromo): vmware
ow2 (just tried dcpromo last night and worked): physical

not saying it makes a difference, but might help troubleshooting

suggestions i haven't tried yet:
- delete the partitions in dns and recreate
- reset machine account passwords on the now three dc's
- two tools suggested by phillip
If you have ADC on physical hardware still running can you just try to ping guid._msdcs.com for all Domain controllers from that server please

Mahesh
Avatar of FocIS

ASKER

mahesh - this was my bad - i pasted back in the guid including "domain.com" instead of the actual domain name.  both guid's do resolve properly from both dc's (all three in fact, now)
I am suspecting that some odd issue exists with VMware, not sure what

Please download Portqueryui tool from Microsoft and run it from new VM member servers (future DCs) to existing DCs to identify any network port issues \ latencies \ errors even if may be all VMs are in same subnet.

because You have promoted physical server successfully to ADC, it indicates that active directory is working fine hopefully

Mahesh
Avatar of FocIS

ASKER

mahesh - the portqueryui gave a ton of output i'll genericify and attach as text files here - nothing jumps out at me as odd but there's a ton of data maybe you can have a quick glance?
server3-to-sc01.txt
server3-to-ts10.txt
Avatar of FocIS

ASKER

as a side note, gpupdate /force from one of the two future dc's shows this:

C:\>gpupdate /force
Updating Policy...

User Policy update has completed successfully.
Computer policy could not be updated successfully. The following errors were encountered:

The processing of Group Policy failed because of lack of network connectivity to
 a domain controller. This may be a transient condition. A success message would
 be generated once the machine gets connected to the domain controller and Group
 Policy has succesfully processed. If you do not see a success message for several hours, then contact your administrator.

To diagnose the failure, review the event log or run GPRESULT /H GPReport.html from the command line to access information about Group Policy results.

-----------------------

all attempts to access the domain controllers from the server that generated that error are successful - but i suspect the reason for this gpupdate getting blocked is the same reason for dcpromo getting blocked

when i view the created gpreport.html, nothing pops out as an error or invalid
Avatar of FocIS

ASKER

update:  i remember making a change to 'access this computer from the network' while trying to resovle this issue - basically we added "domain administrators" and "builtin\administrators".  the policy was applied to the main dc/gc ts10.  

however, researching the gpupdate error above leads me to believe something might be wrong with the 'access this computer from the network' policy for the DC's

can you (or anyone) confirm what exactly should be in that domain controller policy?

right now i have:
a random SID
administrators
domain admins
domain users
enterprise domain controllers
limited
Firewall ports are correctly opened.

Does your srv3 points to existing DC in his preferred DNS server

Are you able to locate srv records from srv3 (server being promoted to ADC) with nslookup command
open cmd
nslookup <Enter>
set type=all <enter>
_ldap._tcp.dc._msdcs.domain.com <enter> --- Replace domain.com with your domain name

If this is failed then there must be some issue with DNS

Mahesh
Avatar of FocIS

ASKER

the only dns entry on server3 is the ip of ts10 (a dc/gc)

the nslookup results in:

> set type=all
> _ldap._tcp.dc._msdcs.domain.com
Server:  ts10.domain.com
Address:  172.16.1.110

_ldap._tcp.dc._msdcs.domain.com      SRV service location:
          priority       = 0
          weight         = 100
          port           = 389
          svr hostname   = sc01.domain.com
_ldap._tcp.dc._msdcs.domain.com      SRV service location:
          priority       = 0
          weight         = 100
          port           = 389
          svr hostname   = ts10.domain.com
_ldap._tcp.dc._msdcs.domain.com      SRV service location:
          priority       = 0
          weight         = 100
          port           = 389
          svr hostname   = ow2.domain.com
sc01.domain.com      internet address = 172.16.1.73
ts10.domain.com      internet address = 172.16.1.110
ow2.domain.com  internet address = 172.16.1.70
below groups There must be in "Access this computer from network in default domain controller policy under user rights, deviation might cause issues

everyone
administrators
authenticated users
Enterprise Domain Controllers
Pre-Windows 2000 Compatible access

mahesh
Add below in above policy...its missing
authenticated users
everyone
Pre-Windows 2000 Compatible access

Then reboot all DCs one by one and then try to build ADC on VM

Mahesh
Avatar of FocIS

ASKER

pre-windows 2000 compatible access was not listed, is now.  there are more things listed in there but all the ones you mentioned are included.

dcpromo on one of the future dc's still fails the same
Avatar of FocIS

ASKER

when on ts10 (one of the dc's) start > admin tools  > security configuration management .. this has different entries than 'default domain controller security policy'

i can't edit the "scm" entry on the start menu, so editing the default domain controller security policy - all of the ones (and more) that you mentioned are in there
Please reboot all Dcs one by one and check in Default Domain Controller policy on each server weather above changes are reflecting under GPMC

You must reboot the DC servers once in order to take it effective and also GPO changes  should get replicated to all DCs

Mahesh
Please open default domain controller policy from GPMC on each server after reboot and check if all having same settings

If here you are facing issues, then its likely GPO Sysvol issue and you might need to follow below KB article
http://support.microsoft.com/kb/290762

Mahesh
Here is the Default Domain Controllers GPO setting:
User generated imageNote the EVERYONE

Philip
Avatar of FocIS

ASKER

At this point all domain controllers do see the same, correct settings for access this computer from the network, as seen from GPMC - no change on the dcpromo from the future vm's
One more tool for you to use that's an update to the replication tool mentioned earlier:

http://bit.ly/1cMnos8 (AD Replication Status Tool from MSFT).

Philip
Is your Sysvol is replicating properly

I mean if you create test.txt file under \\domain.com\sysvol\domain.com on one server is it getting replicated to another DCs without any issues

Can you please check event id 13568 on DCs in FRS events if exists please

Mahesh
Avatar of FocIS

ASKER

Do we still think there's a replication issue though?  I haven't seen anything to indicate that yet, in fact repadmin indicates success

i ran thru the gui of the ad replication status tool, for both forest and domain, it identified all 3 DC's, as GC's, with zero errors and all 'operation completed successfully'
Avatar of FocIS

ASKER

mahesh - i created test.txt at:
\\domain.com\sysvol\domain.com\

it immediately appeared in:
\\sc01\c$\Windows\SYSVOL\sysvol\domain.com\
\\ts10\c$\Windows\SYSVOL\sysvol\domain.com\
\\ow2\c$\Windows\SYSVOL\sysvol\domain.com\
Ok

FocIS,

I am now running out of ideas, not sure where is the exact issue lies

Still i suspect that it is some thing to do with VMware, but not sure what exactly, may be on network front....

I suggest you to call MS support to resolve this issue, they might help you to isolate and identify issue as appropriate

Mahesh
Something I noticed - your output from repadmin /showrepl doesn't reference the DomainDNSZones or ForestDNSZones partitions which I would expect.  That coupled with the error - "The partition DC=ForestDnsZones,DC=domain,DC=com should be hosted at site CN=abc-Cleveland-Datacenter,CN=Sites,CN=Configuration,DC=domain,DC=com, but has not been instantiated yet. However, the KCC could not find any hosts from which to replicate this partition." - to me indicates that these partitions don't exist in your AD for some reason.

Can you look at each of your zones and see what each is set to for replication?
"All DNS servers in this domain" equates to DomainDnsZones.
"All DNS servers in this forest" equates to ForestDnsZones.
"All domain controllers in this domain..." equates to the domain NC.

Also, in answer to one question you had - No, uninstalling DNS from a DC does not remove the DomainDnsZones and ForestDnsZones partitions from replication to that DC.
http://blogs.technet.com/b/askpfeplat/archive/2012/03/09/so-you-think-you-removed-dns-from-your-server.aspx

If these application partitions indeed do not exist, you can create them by following the steps in
http://technet.microsoft.com/en-us/library/cc739505(v=ws.10).aspx
You can run the following commands:
dnscmd ServerName /CreateBuiltinDirectoryPartitions /Domain
dnscmd ServerName /CreateBuiltinDirectoryPartitions /Forest

or from the DNS Management console, right-click on a server and choose "Create Default Application Directory Partitions".  If the zones already exist you will get an error message.
Avatar of FocIS

ASKER

footech - sorry i missed your comment from two weeks ago

in AD sites/services i see:
datacenter > servers > ow2 > ntds replicates from sc01
datacenter > servers > sc01 > ntds replicates from ow2
abc-ssc > servers > no servers or settings exist here

in DNS, i did right-click > create default AD partitions > it returned an error as you guessed, "the specified directory partition already exists"

just now i went thru every single folder in domain.com in dns, and verified that only the expected two dc/gc's exist where they should as names and ip addresses.  i did find at the root of domain.com one "same as parent folder" with the ip address of a previously removed domain controller from the "abc-ssc" site.  i just deleted that A record right now.

in dns, under domain.com > DomainDnsZones > _sites > datacenter > _tcp > i only see references to one of the two dc/gc's (a functional one) - is this ok?  the second functional dc/gc is also in this site physically, but not listed here or at the root of DomainDnsZones

likewise, under ForestDnsZones the same as above, the second functional dc/gc isn't listed here but is physically in the same site

<b>one thing i just remembered - there used to be a server in the abc-ssc site which was force-removed as it was permanently offline.  this server existed for many years and was an additional dc/gc.  The problems may have started at or around the time this server was force removed</b>
Avatar of FocIS

ASKER

i just manually added the missing second dc to the forestdnszones and domaindnszones, used the same exact record as the existing one - verified it did automatically replicate to the other dns server...  attempted to promote a 3rd dc and it failed with the same error as the initial post in this question
For this question - "Can you look at each of your zones and see what each is set to for replication?" - please look under DNS Management console, and look at the properties of the zone > on the General tab you should see the replication setting.  Also, please report what it says for Type.

You should have both of your DCs listed there.  The Netlogon service should take care of creating all SRV records for you everytime it starts.

You might want to run nltest /dsgetdc:yourdomain.com from the servers you are trying to promote to see if they find a DC.
Avatar of FocIS

ASKER

Thanks for the clarification, i was looking in the wrong spot

in DNS, properites for the zone:
replication: all domain controllers in this domain
type: active directory-integrated

those are true as seen from each of the two dns servers

NLtest from a working DC:
C:\>nltest /dsgetdc:domain.com
           DC: \\onlinew2.domain.com
      Address: \\172.16.1.70
     Dom Guid: d59209b3-bb9d-41d9-82de-10d0ddf8c1e4
     Dom Name: domain.com
  Forest Name: domain.com
 Dc Site Name: Cleveland-Datacenter
Our Site Name: Cleveland-Datacenter
        Flags: PDC GC DS LDAP KDC TIMESERV GTIMESERV WRITABLE DNS_DC DNS_DOMAIN DNS_FOREST CLOSE_SITE FULL_SECRET WS
The command completed successfully

The result of that command as seen from the machine we're trying to dcpromo is exactly the same
Avatar of FocIS

ASKER

I've just opened a case with MS, will post the results here when done
ASKER CERTIFIED SOLUTION
Avatar of FocIS
FocIS

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
That is an interesting fix.  I'm curious what setting is the actual culprit.  Maybe a signing or LMCompatibility setting.
Glad you were able to find a fix.
As a rule of thumb one should _never_ touch the default policies. Period.

Whatever changes need to be made and enforced should be done in a separate GPO whether at the domain level or focused via OU linking.

See Jeremy Moskowitz's content and books for the best AD/GP methodologies out there.

Philip
Avatar of FocIS

ASKER

A lot of good people put some good effort into this thread, but for this specific issue, the fix was provided by a microsoft support case and detailed in this comment.