Server 2003 Active Directory tombstoned

https://support.microsoft.com/en-us/help/216498/how-to-remove-data-in-active-directory-after-an-unsuccessful-domain-controller-demotion

Everything you need is in there. Metadata cleanup is what you were looking for,

I am not sure why you bring it back again, or what do you mean by reintroduced?

Are you saying that you have introduced new DC with same hostname as retired 2003 DC and now its not able to replicate?

simply shut down the server permanently and do forceful metadata cleanup
Then introduce the new DC with different hostname than retired 2003 DC and it will work

Mahesh.

Thanks for the feedback.  I re-introduced the server that had been in a failed state for a period longer than tombstone period.  Probably should have just left it dead, Anyway it has been shutdown again.

 I guess I was looking for slightly clearer instructions on the SEIZE requirements since there are two domains in one forest and there is some replication between DC's in the different domains for some forest functions ??  The dead server definitely holds some of the FSMO roles since FSMO queries are reporting as ERROR.

There are two DCs (DC-A and DC-B) in the normal functioning domain (DOMAIN1)  and one DC (DC-C) in the other domain (DOMAIN2) with the 2nd DOMAIN2 DC (DC-D) in the tombstone state.  What roles do I need DC "C" to seize ?

Thanks in advance !

I don't know how many domains you have in forest
in forest root domain, you should have total 5 FSMO roles
schema
domain naming master
PDC master
RID master
Infrastructure Master

in all other domains, you should have below 3 FSMO roes
PDC master
RID master
Infrastructure Master

FSMO seize procedure:
https://support.microsoft.com/en-in/help/255504/using-ntdsutil.exe-to-transfer-or-seize-fsmo-roles-to-a-domain-controller

You need to check which role is unavailable and only need to seize that FSMO master

Mahesh

Two domains - call them DOM1 and DOM2.  DOM1 is ROOT DOMAIN.

DOM1 has two DC's - DC1 and DC2 - all functioning
DOM2 had two DC's - DC3 and DC4 - DC4 is the failed server.

NETDOM QUERY FSMO on ROOT DOMAIN DC1 shows all 5 roles on functioning DC1 server  (note however that one role title is DOMAIN ROLE OWNER , not DOMAIN NAMING MASTER mentioned above)

NETDOM QUERY FSMO on DC3 in the other domain shows DOMAIN ROLE OWNER and SCHEMA OWNER points to same server as above (DC1).  The other three roles PDC,  RID and INFRASTRUCURE OWNER all point to the failed DC4 in this domain.  This is where I assume I have to seize the three roles to DC3 ???

Please note DC1, DC2 and DC3 are GC servers

Please provide as many details as possible please as this is a production domain and want to be as careful as possible with this very old environment.  Thanks !

Ran:  dcdiag /v /e     See attached file.  Clearly see FWTHR_BDC2 as the failed server.  This is my DC4 from above.

To be exact:  

DOM1=MAIN
DOM2=FWTHR
DC1= BDC_SRVR1 (MAIN)
DC2=PRT_SRV (MAIN)
DC3=FWTHRDC (FWTHR)
DC4=FWTHR_BDC2 (FWTHR)

Also a 5th DC=BDC2 in MAIN

Also concerned with a message within the above log:    "A recent replication attempt failed: From FWTHRDC to BDC_SRVR1"   These two systems are fine however I have noticed I cannot log into FWTHRDC with current MAIN administrator password.  Must use old administrator password which confirms replication between MAIN and FWTHRDC not functioning because of ACCESS DENIED.  Not sure how to update MAIN\ADMINISTRATOR password on FWTHRDC to clear ACCESS DENIED ??? This appears to be happening SINCE the tombstoned (and now offline failed DC FWTHR_BDC2) was temporarily fixed and reintroduced but never successfully replicated)

repadmin /showrepl results:

repadmin running command /showrepl against server localhost

Default-First-Site-Name\PRT_SRV
DC Options: IS_GC
Site Options: (none)
DC object GUID: 6e1b6323-f039-43d0-b024-6fdf54bff805
DC invocationID: 00f14728-c33d-42d1-acce-ae1cf62b5e9a

==== INBOUND NEIGHBORS ======================================

DC=main,DC=inc
    Default-First-Site-Name\BDC_SRVR1 via RPC
        DC object GUID: a5ac8ec6-f1d3-4057-92b9-97dc57644f20
        Last attempt @ 2017-05-23 12:41:10 was successful.
    Default-First-Site-Name\BDC2 via RPC
        DC object GUID: c82aa03c-fca2-404b-94dd-16f57e7e29b6
        Last attempt @ 2017-05-23 12:42:25 was successful.

CN=Configuration,DC=main,DC=inc
    Default-First-Site-Name\FWTHR_BDC2 via RPC
        DC object GUID: 891557ca-ac11-4c15-954d-1aa17140397e
        Last attempt @ 2017-05-23 11:57:41 failed, result 5 (0x5):
            Access is denied.
        10925 consecutive failure(s).
        Last success @ 2016-02-23 06:49:12.
    Default-First-Site-Name\BDC2 via RPC
        DC object GUID: c82aa03c-fca2-404b-94dd-16f57e7e29b6
        Last attempt @ 2017-05-23 11:57:41 was successful.
    Default-First-Site-Name\FWTHRDC via RPC
        DC object GUID: aa392e5e-3e42-44ab-a685-8124843f12d4
        Last attempt @ 2017-05-23 11:57:41 failed, result 5 (0x5):
            Access is denied.
        171 consecutive failure(s).
        Last success @ 2017-05-17 19:57:31.
    Default-First-Site-Name\BDC_SRVR1 via RPC
        DC object GUID: a5ac8ec6-f1d3-4057-92b9-97dc57644f20
        Last attempt @ 2017-05-23 11:57:41 was successful.

CN=Schema,CN=Configuration,DC=main,DC=inc
    Default-First-Site-Name\FWTHR_BDC2 via RPC
        DC object GUID: 891557ca-ac11-4c15-954d-1aa17140397e
        Last attempt @ 2017-05-23 11:57:41 failed, result 5 (0x5):
            Access is denied.
        10924 consecutive failure(s).
        Last success @ 2016-02-23 06:49:12.
    Default-First-Site-Name\BDC_SRVR1 via RPC
        DC object GUID: a5ac8ec6-f1d3-4057-92b9-97dc57644f20
        Last attempt @ 2017-05-23 11:57:41 was successful.
    Default-First-Site-Name\BDC2 via RPC
        DC object GUID: c82aa03c-fca2-404b-94dd-16f57e7e29b6
        Last attempt @ 2017-05-23 11:57:41 was successful.
    Default-First-Site-Name\FWTHRDC via RPC
        DC object GUID: aa392e5e-3e42-44ab-a685-8124843f12d4
        Last attempt @ 2017-05-23 11:57:41 failed, result 5 (0x5):
            Access is denied.
        139 consecutive failure(s).
        Last success @ 2017-05-17 19:57:31.

DC=fwthr,DC=inc
    Default-First-Site-Name\FWTHR_BDC2 via RPC
        DC object GUID: 891557ca-ac11-4c15-954d-1aa17140397e
        Last attempt @ 2017-05-23 11:57:41 failed, result 1256 (0x4e8):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
        10925 consecutive failure(s).
        Last success @ 2016-02-23 07:01:57.
    Default-First-Site-Name\BDC_SRVR1 via RPC
        DC object GUID: a5ac8ec6-f1d3-4057-92b9-97dc57644f20
        Last attempt @ 2017-05-23 11:57:42 was successful.
    Default-First-Site-Name\FWTHRDC via RPC
        DC object GUID: aa392e5e-3e42-44ab-a685-8124843f12d4
        Last attempt @ 2017-05-23 12:40:19 failed, result 5 (0x5):
            Access is denied.
        830 consecutive failure(s).
        Last success @ 2017-05-17 20:08:16.

Source: Default-First-Site-Name\FWTHRDC
******* 829 CONSECUTIVE FAILURES since 2017-05-17 20:08:16
Last error: 5 (0x5):
            Access is denied.

Source: Default-First-Site-Name\FWTHR_BDC2
******* 10925 CONSECUTIVE FAILURES since 2016-02-23 07:01:57
Last error: 1256 (0x4e8):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.


C:\Program Files\Support Tools>
dcdiag.txt

You are correct

If DC4 holding all 3 roles, you must seize them on to DC3

ur setup is simple, I don't see any issues in seizing roles

DOM1 (root domain) don't have any issues

what about the current rep failure between FWTHRDC and DCs in the ROOT DOMAIN ??  Do I need to fix this BEFORE seizing the roles ?

you need to go step by step

1st fix the issue exists in child domain by seizing the roles

after that you can move to parent domain

any reboots required ?  I assume I am seizing them to DC3 (only domain left in other domain)

you will only seize, PDC, RID and Infrastructure - child domain FSMO roles

No need to touch domain master and schema master which are part of root domain

still waiting on thoughts re GC role ?  The one and only DC in the domain FWTHR is now RID, PDC, IM and has GC role.  Technotes say this is not good.  What should I do in this case ?

no need, this is the only DC left with child domain, keep it as GC, in fact you must keep it as GC
there is no problem, built new ADC and make it also as GC

only issue appears to be that administrator on one domain no longer has rights as administrator on the other domain.  Help !

For example - I log onto domain MAIN as administrator and try to open DHCP administrator.  It can access DHCP servers from domain I logged into but not other domain.  Says access denied.  Same with DNS.

In fact when I try to log onto FWTHDC DC as MAIN administrator it doesn't let me in also. Says invalid password.

have you added root domain "domain admins" group in built-in administrators group in child domain?

also root domain admin must be added to local administrators group on DHCP server in child domain, by default it don't have that

DOMAIN ADMINS is already a part of built-in administrators group - both ways - both domains.  Always was there.  Nothing changed.

Any time I want to do something where the rights were available on one administrator, however logged in as the other, the only way I can get access is if I log in as the actual administrator for that domain. Always getting access denied. Not just DHCP admin.

Something strange - on CHILD ADU&C when logged on as CHILD administrator,  I can access and  select either domain via CONNECT TO DOMAIN.  

On ROOT ADU&C, if I am logged in as ROOT administrator, it does not allow me to select child - says LOGON FAILURE or BAD USER NAME or PASSWORD. If I log on as CHILD ADMINISTRATOR to ROOT ADU&C I can CONNECT TO EITHER DOMAIN.

One more bit of trivia - if I log into CHILD DC as ROOT administrator - does not let me login using old or new password. Keeps saying can not log you in - make sure user name and password are correct.

have you checked DNS delegation in parent domain pointing towards child domain?
OR if you have secondary zone created in parent dns server pointing to child domain?

Check if one of either above exists and you removed failed DC NS record from there

It seems that you are able to resolve parent from chil but not child from parent because of no existent server

you have to have conditional forwarder in child dns pointing to parent domain

Just an FYI - metadata clean up has not been done as yet.

Re DNS - can you be more specific ?  DNS is NOT AD integrated.

Also, not 100% sure this is parent/child relationship.  They are two domains in one forest with trust between them.

from any DC, I can ping FWTHR.INC and MAIN.INC - the full domain names.

DNS is installed on one of the DC's (FWTHRDC) but I believe it was there before AD role was introduced. Other two DNS servers are non AD servers.

This was working for 10 years as is before failed DC.

SUDDENLY - DNS and DHCP see ALL servers.  I can now log into FWTHRDC using domain\administrator from other domain !!

Still get NOT ACCESSIBLE (either permissions or incorrect name) to access \\FWTHR,INC\NETLOGON frmo PC logged on as MAIN\ADMINISTRATOR.

I think I may be having a single PC issue on my end.  Will reboot and advise

all looks normal right now.  Will give it a day before closing.  Thanks for all the help.  Still need to do meta data clean up

Thanks Mahesh - went above and beyond !

Thanks Mahesh - went above and beyond !