Solved

Demoting Last DC

Posted on 2013-05-20
29
1,543 Views
Last Modified: 2013-06-13
Last week I had a virtual SBS 2011 server become corrupt due to a hard drive redundancy issue.  On that server I have many roles installed including AD.  That server acted as our primary domain GC server.  I also had a backup domain controller that was also a GC.  When I restored the virtual server from a previous export approximately two months ago it could no longer speak to the AD domain that was still running.  To get the computers to talk to the restored server which held file shares I had to shutdown the backup DC and rejoin some of the computers to the "new" domain.  I just attempted to demote the backup dc that's off the network and re-promote it as the backup GC for the "new domain."  I ran dcpromo and stated that this domain was the last in the forest.  I went through the prompts and it came up to a screen stating: "The operation failed because: This Active Directory Domain Controller is not the last AD DC in the domain.  The server is unwilling to process the request."

Does anyone know how I can demote the backup domain controller or how to fix this problem?

Thanks in advance
0
Comment
Question by:ollybuba
  • 10
  • 6
  • 4
  • +4
29 Comments
 
LVL 11

Expert Comment

by:apathy42
ID: 39182647
Easiest way - reinstall Windows.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 39182653
Bring up the DC you're having a problem with off network.  Then delete all OTHER DCs from the Domain Controllers OU in Active Directory Users and Computers.  Then use DCPROMO to demote.

If that fails, run DCPROMO /FORCEREMOVAL

NOTE: You should NOT be using a second DC if you don't understand the proper way to restore a failed DC in a multi-DC environment.  Doing what you did could have severely corrupted your Active Directory.  It likely did not because your restored copy was SO old, but had this happened a week after your last "export" you could be having serious problems now if that were restored.
0
 
LVL 56

Expert Comment

by:Cliff Galiher
ID: 39182701
Based on what you've described, joining machines to the "new" domain, I wonder of the health of your AD environment.

I agree completely with leew that managing a multi-DC environment is not an easy proposition. For example, exports and snapshots should *NEVER* be done on a DC. Period. Full stop.

Where leew and I may differ is based on your description, I believe that AD corruption may have occurred. He is a bit more optimistic than I.

The steps he outlines are basically what I'd recommend. Since you are treating the SBS server as the authoritative server, I'd not even worry about your other DC. That is contingent on if I understand that you've been joining machines back to the SBS version of the split domain. Don't bother trying to demote the server because it sees itself as the only DC so a demote does nothing, not even with a forceremoval. Simply take the machine off the network and clean up any old references to it in the SBS domain.

If, however, corruption has occurred, plan on rebuilding. In most instances, a network small enough to still be managed by SBS is more easily rebuilt than trying to clean up the AD corruption would be.

-Cliff
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 39182746
Hi,

We have dealt with this exact issue:
http://bit.ly/KJ8lOE

How to deal with the second DC after restoring SBS. It's all in there.

Philip
0
 

Author Comment

by:ollybuba
ID: 39187531
Now I am getting numerous errors in the event logs.  One of which is MSExchange ADAccess 2102.  I am also unable to connect to the Exchange Management Console and receive the error:

Initialization failed

The following error occurred while attempting to connect to the specified Exchange server 'XXX.XXX.local':

The attempt to connect to http://XXX.XXX.local/PowerShell using "Kerberos" authentication failed: Connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure.  For more information, see the about_Remote_Troubleshooting Help topic.
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39191127
I just attempted to demote the backup dc that's off the network and re-promote it as the backup GC for the "new domain."

The problem is that the trust relationship between your SBS and other DC has been broken.
Most likely the DC's are not replicating at the moment.

You can confirm that by running repadmin /showrepl or replmon on both DC's.
If replication is not OK, then you'd find that the server last replicated some time back.
If they are in sync then the replication should be done today, maximum a few hours ago.

First thing you can try is to configure the DC's for an authoritative restore.
http://blogs.technet.com/b/sbs/archive/2011/03/31/how-to-perform-an-authoritative-system-state-restore-in-sbs-2008-2011-standard.aspx

Should that task not restart the replication AND you are happy that your SBS server is working 100%, except for possible replication errors, and all your workstations are fine.

The easiest option is to run the METADATA CLEANUP process on the SBS server and removing the other DC from the topology. METADATA cleanup is the task to remove servers from AD directory when you cannot use dcpromo.
http://support.microsoft.com/kb/2647882

Then you can safely format the backup DC and re-install and promote it again.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 39191662
The problem is that the trust relationship between your SBS and other DC has been broken.
Most likely the DC's are not replicating at the moment.

Which is why I'm optimistic.  If the trust relationship was broken (tombstone life was exceeded) then no replication should have occurred and the non-SBS DCs data should not be corrupt as it wouldn't accept changes (and vice versa).  If it was restored within the tombstone life, then yes, you're pretty much in deep trouble.
0
 

Author Comment

by:ollybuba
ID: 39191818
TombstoneLifetime = 180 Do I still continue with the authoritative restore?  It has only been two months at the most.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 39191846
If your tombstone life is 180, then cliff is right.  Start rebuilding.  You've pretty much corrupted your active directory by restoring that VM.

You can try your continued recovery efforts, but I expect you've seriously corrupted things.
0
 

Author Comment

by:ollybuba
ID: 39191880
How do you know what to restore?
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 39191927
Would you please read my blog post?

If the AD on the restored SBS is reasonably accurate, then _keep_ it! DCPromo the existing DC OUT of the domain.

Clean up DNS of references to the second DC (_msdcs stub and _msdcs.domain.local) and DSSite.

Once you have cleaned up the AD/DNS situation DCPromo that second DC BACK into the domain.

As the blog post mentions, that is what we had to do when restoring an SBS server back (even a recent one) because of the way SBS, AD, and DNS are set up.

Philip
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 39191933
In general or in light of this information?

If you still have a copy of the virtual hard disk that was the SBS server or if you made backups of everything before you started your recovery efforts, then you might not have to rebuild from scratch.  At this point, if you had the SBS backup, I'd force-demote the other DC and THEN restore the SBS server like you did.  With the other DC force-demoted, it won't even be in the domain.  You then remove it from the SBS server's Active Directory and rejoin it.  YOU WILL LOSE ALL AD CHANGES SINCE THE SBS BACKUP.
0
 

Author Comment

by:ollybuba
ID: 39192109
MPECSInc I did read your article and thank you for the information.  It seems that active directory is working properly on my SBS except for exchange.  When I turn on the backup dc and have it connected to the network exchange works properly.  Would it make sense to try and restore the Microsoft Exchange Security Groups to the SBS server?


In addition

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\Server1
DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL
Site Options: (none)
DSA object GUID: 622acd30-c9f4-4506-b95d-c7553f84af00
DSA invocationID: 622acd30-c9f4-4506-b95d-c7553f84af00

==== INBOUND NEIGHBORS ======================================

DC=domain,DC=local
    Default-First-Site-Name\DCBACKUP via RPC
        DSA object GUID: d7602463-9d4e-4051-902a-b5a56d2cf2e6
        Last attempt @ 2013-05-23 12:54:28 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        201 consecutive failure(s).
        Last success @ 2013-04-14 10:06:23.
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 38

Expert Comment

by:Philip Elder
ID: 39192217
Please post an IPConfig /ALL from the restored SBS

Philip
0
 

Author Comment

by:ollybuba
ID: 39192260
Could this be because I don't have 10.0.10.22 set as another DNS server?


C:\Windows\system32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : Server1
   Primary Dns Suffix  . . . . . . . : domain.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : domain.local

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Ada
pter
   Physical Address. . . . . . . . . : 00-15-5D-01-6C-04
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.0.10.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.0.10.254
   DNS Servers . . . . . . . . . . . : 10.0.10.2
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{43EA2928-0256-481F-84C4-B172B58369F5}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 39192293
Where is IPv6?

DNS0 should point to self on a DC and that's it.

Philip
0
 

Author Comment

by:ollybuba
ID: 39192470
Windows IP Configuration

   Host Name . . . . . . . . . . . . : Server1
   Primary Dns Suffix  . . . . . . . : domain.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : domain.local

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Ada
pter
   Physical Address. . . . . . . . . : 00-15-5D-01-6C-04
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::d821:2574:93f3:fdde%10(Preferred)
   Link-local IPv6 Address . . . . . : fe80::f255:7011:7b1b:23c0%10(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.0.10.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.0.10.254
   DHCPv6 IAID . . . . . . . . . . . : 234886493
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-19-26-00-B2-00-15-5D-01-6C-04

   DNS Servers . . . . . . . . . . . : fe80::f255:7011:7b1b:23c0%10
                                       10.0.10.2
                                       10.0.10.22
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{43EA2928-0256-481F-84C4-B172B58369F5}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 39192485
What is 10.0.10.22 list5ed in DNS servers?

That should not be there. DNS on the NIC should only point to itself.

What service pack level is Exchange at?

Philip
0
 

Author Comment

by:ollybuba
ID: 39192583
SP2
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 39192606
Save and clear the App and SYS logs (Event Logs).

Reboot SBS (with DC offline).

Once SBS comes up, does Exchange come back online?

If not, run the Connect to the Internet, Internet Address, and then the Fix My Network Wizards.

If you had a GoDaddy or other third party certificate then run the Third Party Trusted Certificates wizard and choose the option to install a certificate located on the server.

Does this help straighten out the errors?

Philip
0
 

Author Comment

by:ollybuba
ID: 39202589
Exchange is functionally running fine.  The only problems I am still seeing are errors in the event logs in relation to not being able to contact the second dc.  Time for METADATA CLEANUP?
0
 
LVL 27

Expert Comment

by:Steve
ID: 39208119
@Ollybuba

You're in a really difficult situation here and I'm afraid this is a tough one to resolve due to the complexity of what has happened.
I agree with some of the details mentioned by the guys above and I also suspect you may end up having to start again from scratch.

I'm happy to help you fix it in the meantime though as it may be possible to fix, but it is a complicated procedure and I do not know what level you are at and if the tasks are within your comfort zone.

Have a look at the below and consider if you feel it's worth trying. There will be downtime involved and you need to be sure you are happy using utils like ADSIedit & NTDSutil. If this isn't your kind of thing you may be looking at wiping your domain and starting again.


In my opinion, this is worth considering:

Initially, you need to give up on fixing both servers and just fix your AD on one. this is critical.
Normally I'd recommend assessing which server to fix based on several criteria, but you don't really have a choice due to having an SBS box.

you therefore have to concentrate on getting the SBS box working WITHOUT the 2nd DC.

This means turning off the 2nd DC and giving up on it. You can wipe that and create a new DC with it once everything is working.
*Note: when I say 'turn off' I really mean 'off'. DO NOT TURN  THE 2nd DC BACK ON WHILE CONNECTED TO YOUR NETWORK or you will be right back to where you started.

With the 2nd DC out of the frame, you need to clean up your AD on the SBS. This involves manually removing the 2nd DC from AD so your network is based entirely on the SBS as a DC.

You also need to make sure your SBS is functioning OK. ensure it is not referring to the other DC for DNS queries and make sure other applications like exchange are not configured to use the other DC/DNS server.


These guides take you through the removal of a failed DC.
http://www.petri.co.il/delete_failed_dcs_from_ad.htm#

http://www.experts-exchange.com/Software/Server_Software/File_Servers/Active_Directory/A_677-How-to-Remove-a-Failed-Domain-Controller.html

http://technet.microsoft.com/en-us/library/cc781245(v=WS.10).aspx

Checking which DC exchange is using:
http://exchangeserverpro.com/how-to-use-a-specific-domain-controller-in-exchange-2010-management-shell/
0
 

Author Comment

by:ollybuba
ID: 39216375
C:\Windows\system32>cd..

C:\Windows>ntdsutil
ntdsutil: metadata cleanup
metadata cleanup: connections
server connections: connect to server <server1>
Binding to <domain> ...
DsBindWithSpnExW error 0x6ba(The RPC server is unavailable.)
server connections: connect to server <server1>
Binding to <server1> ...
DsBindWithSpnExW error 0x6ba(The RPC server is unavailable.)
server connections: connect to server server1
Binding to server1 ...
Connected to server1 using credentials of locally logged on user.
server connections: quit
metadata cleanup: select operation target
select operation target: list domains
Found 1 domain(s)
0 - DC=domain,DC=local
select operation target: list sites
Found 1 site(s)
0 - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=local
select operation target: select site 0
Site - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=local
No current domain
No current server
No current Naming Context
select operation target: list sites
Found 1 site(s)
0 - CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=local
select operation target: list servers in site
No current site
No current site
No current domain
No current server
No current Naming Context
select operation target:
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39216416
I believe you are running the commands on the wrong server.
You need to perform the METADATA cleanup on the server that has the valid copy of your AD database. i.e. should be run on the server that you want to keep.
Reason: You want to remove the 'failed' DC from the most up to date version of the AD database.

Alternatively, if this is a valid DC then it could require a Windows firewall change.
http://blogs.dirteam.com/blogs/paulbergson/archive/2010/11/01/windows-dcdiag-generating-error-0x6ba-quot-the-rpc-server-is-unavailable-quot.aspx

If it is the case of the firewall being blocked then I'd suggest a review of your firewall settings as DC's should be able to find other DC's. Should you not change the windows firewall configuration then your DC's won't be able to replicate/communicate properly.
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39216430
Explanation of what the DsBindWithSpnEx function is trying to perform.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms675963(v=vs.85).aspx
0
 

Author Comment

by:ollybuba
ID: 39216740
When running metadata cleanup and I select the site, domain, server to remove will this also delete the site/domain?

I'm know I'm suppose to get a warning message but I attached the one I received.
Server-Remove-Confirmation.jpg
0
 
LVL 26

Accepted Solution

by:
Leon Fester earned 500 total points
ID: 39217473
No, only the specified domain controller will be deleted.

Have a look at the "Advanced optional syntax with the SP1 or later versions of Ntdsutil.exe" near the end of this link http://support.microsoft.com/kb/216498 and you'll see that it uses the "DN of the NTDS settings object of the server that is being demoted.

It also shows you how to find the DN of the NTDS settings object of the server that is being demoted.

P.S. You cannot delete a domain or site while an object exists below that level. i.e. you cannot remove a site if a server still exists in the site. You cannot delete a domain while a site exists in that domain. You cannot delete a root domain while a child domain exists.

BTW, since AD 2008, you could perform the metadata cleanup by deleting the computer object from ADUC. Maybe try that option first if the computer account still exists in AD.

http://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

This article will help you understand what HashTables are and how to use them in PowerShell.
Marketers need statistics and metrics like everybody else needs oxygen. In this article we explain how to enable marketing campaign statistics for Microsoft Exchange mail.
This tutorial will give a short introduction and overview of Backup Exec 2012 and how to navigate and perform basic functions. Click on the Backup Exec button in the upper left corner. From here, are global settings for the application such as conne…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now