Re-promoting a Windows Server 2019 Domain Controller DC3 in a set of 3.

Fred Marshall
Fred Marshall used Ask the Experts™
on
I have a system with 3 Domain Controllers (Server 2019).  One of them got out of sync (my best description) and wasn't replicating - while the other 2 seem to be working fine.
The problem seemed to be RPC errors but all 3 DCs were running together in a test lab before being deployed.
This one was deployed a month or so after the first two, having been powered down during the interim.
The failure would cause me to suspect the real network they are now in somehow - but "how?" is elusive.
I have wireshark captures taken during replication attempts but I don't see much that jumps out.

Eventually, I was advised to take the "bad" DC off the domain and re-promote it.  
I've proceeded with thatprocess and have tried a lot of suggested things but so far no luck in promoting this DC.
Fortunately, this DC wasn't in the mainstream of operations and, without replication, had less configuration data on it than the others.
So, taking a step back is of little concern.

I think I did a pretty good job of bringing the DC down but I'm unsure of its status.  There's a first time for everything...
I still see remnants of its name on the others for example.
So, I think the first step is to confirm that stuff is removed as required in preparation for promotion.
I've run a bunch of tools but don't know when to be happy or sad re: the results.  Just no experience.
So, I'm wasting calendar time by running down blind alleys.

I *did* try to promote this DC but the process won't complete.
The process ends with what appears to be a need for further DNS configuration.

I could use some real help.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Distinguished Expert 2017

Commented:
Your issue is "The problem seemed to be RPC errors but all 3 DCs were running together in a test lab before being deployed.
This one was deployed a month or so after the first two, having been powered down during the interim."

The object has been tombstoned.

You need to do it in the following stages.
1) make sure you have a local account on this DC.
2) disjoin it from the domain
3) do metadata cleanup to remove all references to this DC including deleting its object from the ad computers...
4) addict it as a new member/DC.
Distinguished Expert 2017

Commented:
Use dcdiag to make sure the AD is clean and has no reference to the DC #3

Author

Commented:
arnold:  Thank you!  What dcdiag test would you recommend exactly for this purpose?

arnold: I did all those things - even though I feel unsure about step 3 being 100% effective.

When I try to promote the DC, here is what I get:
 DNS delegation not found
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Distinguished Expert 2017

Commented:
Check which name servers are configured on this system? It could still have configuration remnants I.e. Points to itself for name resolutions.

ipconfig /all | find /i "name server"

dcdiag shoukd be run on the DCs #1 and #2.
#3 is no longer a DC, sounds missed that in step 2 prior to disjoining from the domain to demote the DC.

often when disjoin, rejoin, it says that it located an existing object and whether you want to use it..

Author

Commented:
dcdiag shoukd be run on the DCs #1 and #2.
Yes, but which of the many commands do you recommend?
Distinguished Expert 2017
Commented:
You are looking at the health of the AD
On dc#1
dcdiag /v
dcdiag /s:dc#2 /v

You are mainly looking to make sure there are no reference to the decomed dc#3

Before rejoining .....dc#3
Lee W, MVPTechnology and Business Process Advisor
Most Valuable Expert 2013

Commented:
Why three DCs?  Except in RARE circumstances, you should never need more than 2 per site.  Do you have multiple sites?  

It's unclear how you removed the DC from the network.  If it got "out of sync" by more than 60 days, then you needed to treat it as a failed DC and remove it as such.  For the most part, deleting the "failed" DC from the Domain Controllers OU should be largely sufficient, but you would still need to clean up some items in DNS and likely AD Sites and Services.

If it got corrupted and AD was no longer replicating, there may have been other ways to resolve the issues, but if you

As for DCDIAG Before promoting a second (or subsequent DC), I always run DCDIAG /C /E /V from all DCs (in a small environment).  Then I resolve any unexplained issues before attempting to promote another system to be a DC.

Author

Commented:
Lee W:  Just FYI

There are 3 sites that are all fairly tightly coupled with fiber.  There are disaster recovery plans in place that the 3 DCs support.
We originally thought of having 2 DCs (one per site again).  So, except for the objectives, the system is operationally fine less one DC.  
We have rarely had the fiber system go down but it did happen once - and recently.

The network was brought up entirely off site.
Then, our integration process left the last one out too long.
And, just now, we learned that we needed to treat it as a failed DC so have removed it.
And here we are....

arnold and Lee W: Thank you both for the guidance re: dcdiag!  That saves a lot of time sorting through the tools.  I don't mind figuring it out myself but in this case there are a lot of words being used that aren't immediately familiar re: meaning.  That's a big deal in a practical sense.  So thanks for being translators!

Author

Commented:
OK, so those have been run and I'll analyze them in the morning.
As I mentioned, I originally had all 3 working together and then moved #1 and #2 out into the real world.
However, #3 was in the original configurations and was destined to be reunited with the others.
So now I still see #3 in the configurations of #1 and #2 and need to remove it.

Author

Commented:
As I look through the DCs and the logs, I notice this redundancy:
Current info from what we are callingI imagine it should be removed but I need to decide which one?
If I were to guess, I'd say the second, longer entry.
It seems like I did this before but it's still there (if I did).
Distinguished Expert 2017

Commented:
Does each DC have multiple IPs?
i.e. DC2 how many IPs does it have one ipv4 and one ipv6?

You can check each connection's properties to see what it is.

Author

Commented:
Does each DC have multiple IPs?
Not ipv4.  Only one.

i.e. DC2 how many IPs does it have one ipv4 and one ipv6?
Yes.  One each.

You can check each connection's properties to see what it is.
If you mean in NIC properties, right?

I have come to just let ipv6 exist with no changes.
I did read that turning off ipv6 solved a DC re-promoting problem.... but I couldn't tell how old the solution was.
Is that still recommended - or recommended at times?

The SAME THING is true of #1 but the display doesn't show the redundancy...

Author

Commented:
#3 (DC1-RAY) is NOT any longer listed in the NIC DNS entries on #1 nor #2.  Yet:

On #1 I get:

DNS server: x.x.1.2 (DC1-RAY)

               1 test failure on this DNS server

               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server x.x.1.2               [Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
               Name resolution is not functional. _ldap._tcp.net.rayfed.com. failed on the DNS server x.x.1.2
               [Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]

Open in new window


On #2 I get the same thing.

I *think* I've done everything to get rid of these references, yet they are still there....
Distinguished Expert 2017

Commented:
The error is not significant it merely means that the localhost 127.0.0.1 could not be looked up by ip.
The fix to avoid the warning is to open DNS management add a reverse zone 127.0.0 and to add a PTR record 1 that points to localhost.

It might be using the name server records that point to the localhost versus LAN IPs.
Distinguished Expert 2017

Commented:
In ntds you can select each entry from the right pane of the image and get properties it will tell you the type of relationship, ..

Author

Commented:
I can go so far as to figure out that you mean: Active Directory Sites and Services / Sites / [site] / Servers / DCxyz / NTDS Settings.  Right?
But truthfully, shorthand is tough for me right now because of my limited language in this community.
Now "type of relationship" is completely lost on me.  What type am I looking for and where?
Distinguished Expert 2017

Commented:
In the image where you have two connections from dc2 reflected. Select one and right-click select properties. Within it will show you info. You can then do the same for the other vonnection and it will indicate the distinction between the two.

On newer servers, I think since 2012 the IPv4/IPv6 should not be disabled. Those suggestions for IPv6 disabling dealt. With issues when newer 2008, 2012 were being added to existing 2003 AD ....

Author

Commented:
arnold:  Thank you again for the quick reply!
So the ipv4/ipv6 question is settled (again)  :-)

"it will indicate the distinction between the two"
Well, that may be obvious to you but it sure isn't obvious to me.
I asked "What am I looking for and where?"
I can look through the 4 tabs that are presented and they look pretty much the same.
I can open both of them up side-by-side in Attribute Editor and scroll down through the long list of Attributes and, again, I see nothing much useful.  The difference is their names which we can see in the NTDS display on the right.  If there's another difference of note, I've not found it.

Anyway, I followed the route of adding a PTR record to 127.0.0.1.  We'll see how that affects things.

Thanks again!
Distinguished Expert 2017

Commented:
Sorry, initially I had the impression that dc#3 us the issue. In your most recent comments including the images, dc3 but you reference with issues in dc1.

I am uncertain what state/stage you are in.
Metadata cleanup to remove the DC you unjointed.
The old object deleted from ad?

Recheck the network configuration on the demoted ad to make sure name server records point to existing DCs.
The link to the ..... That the name server/DC can be reached.

nslookup -q=SRV _ldap._tcp.dc._msdcs.youraddomain

Since you have multiple sites the above query instead of DC which reflects all DCs in the environment would be narrowed to DCs at the soecific site if ...

Try to rejoin this system as another DC.

Author

Commented:
I'm sorry about the numbering.  There's a difference between the question here #1, #2, #3 and the actual names DC1=#3, DC2=#1, DC3=#2.  I was rather hoping to not have to confuse with that.

I'm nearing the end of the cleanup I believe which includes the old object deleted from AD.
And, I have the impression that a pristine system of 2 operating DCs is rather important before embarking on rejoining #3.
Thus the DCDIAG reports / discussion on #1 and #2.

I *may* be ready to begin the rejoin process.
Distinguished Expert 2017
Commented:
it Does not need to have major existing errors as an addition would add complexity to problem isolation if not push problems...

Make sure your name server settings on this DC pointing to the other two DCs.

Author

Commented:
arnold:  OK.  Thanks!

Author

Commented:
Thanks all!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial