Link to home
Start Free TrialLog in
Avatar of midustouch
midustouchFlag for Singapore

asked on

Reconnect child domain to AD forest after tombstone period without demoting child domain DCs

Hi everyone,
Having a really hard time digging up the correct information and steps to take from Internet...

Scenario:
At my main forest DC, I can't see my child domain DCs in AD Sites and Services.
At my disconnected child domain, in AD Sites and Services, I can see all my child domain DCs

We shall call this disconnected child domain site "PITA"

Attempted "repadmin /regkey * +allowDivergent" at both Parent Forest and disconnected child domain without removing lingering objects
Result - At my disconnected child domain "AD Sites and Services", there is now site "PITA" and "PITACNFCNF:601a7e30-7274-4e13-84fe-5eadbc4b03b5"

- Have placed AD back into STRICT replication.

Limitations and other information:
1. AD Tombstone lifetime of 60 days has been exceeded.
2. Avoid at all costs - demote domain controllers at this site (it can be done, but it's too costly a move)
3. Mixed environment with Windows 2008, 2008 R2, 2003, 2003R2 DCs
4. Parent Forest DC is a Windows 2008 SC machine.
5. Disconnected child domain DC is a Windows 2008 Full Install machine
6. Disconnected child domain has Windows 2003 and Windows 2003 R2 Peer DCs

Question:
How do I reconnect this disconnected child domain back to the main AD forest WITHOUT demotion of child domain controllers

Thank you all in advance for any assistance rendered.
ASKER CERTIFIED SOLUTION
Avatar of ARK-DS
ARK-DS

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Since you have 2003 and 2008 domain you can use adrestore from sysinternals to try and restore tombstoned objects. This is the only option I see without taking the machine offlineYou should read the following article before you attempt this.
http://technet.microsoft.com/en-us/magazine/2007.09.tombstones.aspx?pr=blog
and here is the link to adrestore
 http://technet.microsoft.com/en-us/sysinternals/bb963906.aspx
  Good luck and let me know how it goes
Avatar of midustouch

ASKER

@ark-ds;
Thanks for the links.
Will give your suggestions a try on Monday - is the first link still usable for a forest with W2K8 servers?

@sfossupport;
1. Why would I want to restore tombstoned objects? Right now I have lingering objects problem and duplicate sites with possibly security permissioning issues,,,
2. Why would I want to do an adrestore, the rest of my forest is ok. It's just the child domain is an island now that's all.
Schema partition doesn't contain or can have lingering objects.
Adrestore will not restore membership of the user..........Just for info only
 If its crossed the tombstone there is no way you can get the adc working,only option is to demote & promote.
You can try removing lingeringobject using repadmin /removelingeringobjects command,but can't guarantee whether its going to be successfull,but demoting & repromting is the only option since its crossed tombstone period.
http://technet.microsoft.com/en-us/library/cc949136(WS.10).aspx
As you said its too costly but in that case repadmin /removelingering object doesn't remove lingering object cleanly & there is issue with replication afterwords too,so its recommended demote/promote.
References:
http://blogs.technet.com/glennl/archive/2007/07/26/clean-that-active-directory-forest-of-lingering-objects.aspx 
http://blogs.dirteam.com/blogs/jorge/archive/2006/05/08/Lingering-objects.aspx 
@Awinish

Will ARK-DS suggestion to rejoin domain using "http://support.microsoft.com/kb/887430" work?

I've already tried steps in http://technet.microsoft.com/en-us/library/cc949136(WS.10).aspx prior to posting this question.

The problem now is that in AD Sites and Services in my parent forest, I don't see any of my domain controllers in my child domain site "PITA"

In my child domain PITA, in AD Sites and Services, I see PITA (nothing in it) and in addition another site PITACNFCNF:601a7e30-7274-4e13-84fe-5eadbc4b03b5 with all the child domain DCs

This command was attempted at child site:

repadmin /removelingeringobjects dc1.parent.forest 601a7e30
-7274-4e13-84fe-5eadbc4b03b5 DC=parent,DC=forest /advisory_mode
DsReplicaVerifyObjectsW() failed with status 8453 (0x2105):
    Replication access was denied.

Open in new window

This command was attempted at parent forest site:

repadmin /removelingeringobjects dc1.pita.parent.forest a
c3fc2b8-759a-4bf1-8fba-619509fd989a DC=PITA,DC=parent,DC=forest
DsBindWithCred to dc1.pita.parent.forest failed with status -2146892976 (0x800903
50):
    The system detected a possible attempt to compromise security. Please ensure
 that you can contact the server that authenticated you.

<edited by SouthMod 03/08/2010>

Open in new window

Retried @ parent forest site with correct authentication (/u /pw parameters):

repadmin /removelingeringobjects dc1.pita.parent.forest a
c3fc2b8-759a-4bf1-8fba-619509fd989a DC=PITA,DC=parent,DC=forest /advisory_mode /u:PITA\domainadmin /pw:*
Password:
DsReplicaVerifyObjectsW() failed with status 8524 (0x214c):
    The DSA operation is unable to proceed because of a DNS lookup failure.

Open in new window

(I made a mistake editing code snipped in ID: 27554103. Missed out changing actual LDAP names.)

(Can't seem to find where I can edit that post in this thread)

I suspect it's a security/authentication issue where the PITA site has been "blacklisted" by the rest of the good domain controllers at parent domain and other child domains caused by tombstone lifetime exceeded.

Since I can see my DCs in my PITA site (although the DCs are in a CNF container in my Sites and Services) and there's no problem so far in my PITA site for local users authentication to local domain.

The last response (D: 27554331) after I used parameters /u and /pw of "DSA operation is unable to proceed because of a DNS lookup failure" is confusing me. Is it at my parent site lookup or at the child domain site?

I've already tried moving my DC objects from PITACNF container to PITA but it doesn't get replicated out. (even in LOOSE mode), so I've moved it back.

Is there any way to force the PITACNF container to be replicated out, then delete the PITA container and rename PITACNF to PITA?
With regards to the option of demoting and promoting, I have 5 DCs in the child domain, if I demote one and promote it back, can I retain the current domain name?

If so, then that's a viable option.

I'm avoiding demote and promote as I'm assuming I'd have to create a new domain and the problem with that is that I have a sharepoint team service (WSS) portal in the child domain and almost a hundred workstations - it's alot of extra work.
If I have the child domain DSA GUID (gleaned from my good sites), can I manually add it back to my DNS servers and edit the sites and services using ADSIedit?
With regards to the option of demoting and promoting, I have 5 DCs in  the child domain, if I demote one and promote it back, can I retain the  current domain name?

Are you talking about same name of domain controller,if yes, you can demote one of the dc, clean up all the entry from dns as well as from ADSIEDIT.MSC,if it is graceful demotion means w/o using dcpromo /forceremoval.

If its force removal of dc,perform metadata cleanup of that dc & promote it.

But before promoting make sure old entries are removed properly & replicated to all the dc's,then promote the new dc with same name & IP.

CNF means conflict there is object in ad with same name,look for adsiedit to remove & also in lost & found container.

You have to remove lingering object from child domain, because mostly lingering object sits into domain directory partition & you can run same into parent domain too.

Domain directory partition stores info to its domain specific.

The only common directory partition is schema,configuration & conditional application directory partition.

Yes,you can make the GUID entry but i'll not recommend for changes into adsiedit,because changes done into that will be permnanent.

http://activedirectoryutils.codeplex.com/releases/view/13664
http://www.aspfree.com/c/a/IIS/The-Importance-of-a-Domain/6/
@Awinish - Thanks alot! I'm glad you know about "CNF"

OK let me reiterate to make sure I understood completely.

How to fix the orphaned child domain by graceful demote and promote:
- I have 5 DCs in my orphaned domain, called PITA.
- I gracefully demote one DC in PITA domain
- Cleanup all entries in DNS and ADSIEDIT
- Join the domain controller back (at this point it should appear in my AD Sites and Services PITA container?)
- Once this DC is back in my forest, let replication complete to all forest and other child sites
- Repeat for the rest of the DCs

Removal of Lingering Objects:
1. Once all DCs are out of the CNF container, look for all instances and remove them
2. Places to look; Domain directory partition, and ?? schema, configuration & conditional appliction directory partition

I'm not clear about #2. Can help clarify further?

Lastly:
A. Is there anything else I need to check or fix before attempting the above?
B. What if a graceful demote doesn't work? ADSIEdit only option then? If so where and how.

Thanks alot.
While removing lingering object using repadmin you can specify domain & configuration directory partition & configuration partition..

Windows 2003 introduced a new directory partition type—the application directory partition. The application directory  partition stores dynamic application-specific data in the Active Directory but  rather than being replicated to all domain controllers in a domain or tree, the  data is replicated only to domain controllers specified by the Administrator.  Application directory partitions can contain any type of object apart from  security principals (users, groups, and computers). http://www.aspfree.com/c/a/IIS/The-Importance-of-a-Domain/6/


If graceful demotion doesn't work,you can use dcpromo /forceremoval & perform metadata cleanup. Clean up the demoted ADC from AD & DNS(each record of dns from _msdcs folder,name server,cname etc). Give time to replicate to other dc's.

http://www.petri.co.il/delete_failed_dcs_from_ad.htm
After metadata cleanup,normall all the entries are not removed,verify in dns under each in  _msdcs folder,in adsiedit.msc  etc.

Yes,above steps are fine.
Is all your dc's are windows 2003 & above or 2000 also?

You can safely remove the cnf object & also check in adsiedit if its exists.

http://blogs.technet.com/janelewis/archive/2006/10/30/unravelling-cnf.aspx
http://blogs.technet.com/ad/archive/2008/06/06/conflict-resolution-lingering-objects-printers.aspx
You must make sure parent domain dc's are fine & replication is proper along with dns name resolution.
Dc's in child Child domain is also proper except problem one & dns is proper.

You can use dcdiag as well as repldiag tool from below sites.

http://activedirectoryutils.codeplex.com/releases/view/13664 
Yes all my DCs are Windows 2003 and above. No Windows 2000.
Yes; replication is working properly at forest level and the other child sites (other than the orphaned)

I don't think force removal is an option here.
1. That would mean I would have to rejoin domain for all workstations.
2. It will break my WSS installation as my predecessor used the original first domain administrator (of child site) to create it.

To your question "Are you talking about same name of domain controller,if yes, you can demote one of the dc"
- I want to retain the DOMAIN/NETBIOS name - these steps (without force removal) will let me keep the existing PITA domain right?

In this case, wouldn't it be safer for me to create another new DC at PITA domain? Test if that replicates correctly to forest/other sites and test if it is placed into the correct AD sites and services container "PITA"
I'm confused,

You have five child domain with single dc in each child domain?

Means parents domain abc domain child domain as 1.abc.com,2.abc.com etc.

Dc in child domain as server.1.abc.com etc

Thats why if you remove single dc,you don't have any more dc's in child domain & all the domain system will be disconnected.
Let me clarify. (sorry for the confusion)

1. I have 5 DCs in my orphaned child domain. I wish to keep the current netbios/domain name PITA.
2. This orphan domain is a child domain of a forest/parent which has other child domains not related to this one.

Means parents domain abc domain, child domain as pita.abc.com, otherchild1.abc.com, otherchild2.abc.com

I need to make sure I can still logon to PITA(netbios) pita.abc.com(fqdn AD) as the domain. Need to retain the existing relationships between workstations and domain controler @ orphaned child domain.

(Reference ID ID: 27564709):- I'm suggesting create another new DC in PITA domain to test AD sites and services container placement and replication so that nothing further has to be changed.
Got you,but as your child domain pita.abc.com conntains lingering object then it will be back on new dc too & block replication on that too.

ok we're back to square one then.

How will it work if I demote 1 DC from the existing 5 DC?

won't replication be blocked too?
Do you have healthy system state backup which is not older as tombstone life.

Its always recommended at least there should be min two dc's in domain & check replication & dc health always.

Anways,if you create new domain all the machines hav to be rejoined even though you use same name.
No I don't have a system state backup. lolz.
I want to force the replication through. Unlock "blocked" replication.
You have 5 dc in your child domain named as pita.abc.com & one is infected with lingering object.

is it all giving you error of llingering object or others dc's in child domain is working fine.


I have 5 DCs in your child domain (yes domain name is pita.abc.com).
I think the whole domain is affected.

This child domain is disconnected from my parent domain and forest.

I'm not receiving any lingering object errors anywhere.

Parent domain = fine (RPC error to orphan domain)
All other child domains = fine (RPC error to orphan domain)
Orphaned child domain = fine but is refused connection by any other DC in forest outside it's domain.

I think you should try to remove lingering object in your child domain & lets see if its works using repadmin /removelingeringobjects.
Reference ID: 27566820

I tried that already and I'm getting those errors I posted in my code snippets above.

It's a catch22
- I can't connect from my child domain to the parent cuz it's blocked.
- I can't connect from my parent to child - RPC connection failed, ie; kerberos authentication failed.
Check dns whether its resolving & also reset secure channel & try to make one of your dc's to talk to parents domain.

http://support.microsoft.com/kb/260575

If lingering object is badly present then demoting the domain & re-promoting will be only option.

Catch u later have some work.

RPC failure is related to firewall & see the ping & you are able to telnet AD ports.
It's not firewall issue, I can RDP into the affected domain.

I suspect there's a simpler solution to this problem.

There are no lingering object errors anywhere.

There is no blocking internally between sites. (VPN)
Reference  ID: 27565671 -
"Got you,but as your child domain pita.abc.com conntains lingering object then it will be back on new dc too & block replication on that too."

I don't think it's a lingering object problem, it's something else.

The new DC at orphaned site may not be connectable to outside domain anyway due to forest GC sync issues. (burflag)
Demote or existing DC in child domain resulted in "The directory service is missing mandatory configuration information, and is unable to determine the ownership of floating single-master operation roles."

NTDS Site Settings is not available in the PITACNF folder.
I've logged a case with Microsoft Platform Support Group for this issue. Will update post when I have more information or the solution.
I apologise,i just got time to log now.

midustouch,you can do share the findings & resolution for others to learn.

Its disappointing,our solutions didn't help you.

Lets get the review & results from MS>
@Awinish

Had already tried most of the stuff you guys suggested before posting. :-D

Tried to demote domain controller in orphan site, but unsuccessful. Only /forcedemote will work and I can't do that for this site.

Let's see how MS is going to solve this. ;-)
Avatar of ARK-DS
ARK-DS

Hello...

I am so sorry, I could not log in earlier. To be frank, i had no guts to read whole of the thread as it has gone quiete long now.  

As far as I understand now, You have 5 DCs in the child domain. And the child domain is cut off form the parent domain.

A question: Do you see parent domains DCs in child domain's dssite.msc? (I need to know this to help you). + you must have connectivity between parent and child domain DCs (Pinging should work ).

I dont think at all that there is a need to demote and promote the DCs in the child domain. As I suggested before, please follow the article http://support.microsoft.com/kb/887430 to create a replication link between parent and child domains.

"" See, we already have a orphaned child domain, doing this can only help us, it can not do anything bad "".

If you have any further queries, you can e-mail me your queries on kohliarun@hotmail.com

@Awinish: Schema Partition can have lingering obbjects + demotion and promotion is not at all a resolution to lingering objects and tombstoned DCs (I would just consider it a workaround that always works).

Regards,

Arun.
@Arun read below

Lingering  objects are objects that exist on one or more DCs that do not exist on other  DCs hosting the same partition. They may be introduced in any partition except  the schema.

http://blogs.technet.com/glennl/archive/2007/07/26/clean-that-active-directory-forest-of-lingering-objects.aspx

We can use work around but if they don't get removed by repadmin tool,then only option is demote & promote learned,faced lingering obj issue in my previous project & they didn't get removed using repadmin tool too.
@ARK-DS

Problem is almost resolved.  Been talking to Microsoft these last two weeks - we are at the last stage of monitoring replication and cleaning up.

And yes, you are right about KB http://support.microsoft.com/kb/887430. It's the first step required to rejoin the orphaned domain to parent. Unfortunately, I didn't get confirmation that this would work with Windows 2008 servers and in Windows 2003 Forest/Domain level installations - if I had, it would have saved a call to Microsoft. :-)  AND Microsoft's KB article was updated (APPLIES TO doesn't mention anything other than Windows 2000 Servers!!) See attached image kb887430.jpg

However, the whole process isn't as simple and involves many steps.

Will post further once everything is resolved.

kb887430.jpg
Hello midustouch,

I apologize not being able to respond in a timely manner as it would have saved your call to MS. As I have done this quiete a few times before when I was a part of MS team.
Anyways I am happy that the issue got resolved.

@Awinish: I opposed your saying that schema partition CAN NOT have lingering objects. I can oppose this blog as well because there are numerous MS articles that talk about removing lingering objects and all of them say that you can remove them from all partitions (including schema) "just an example: http://support.microsoft.com/kb/870695 : see step 3 of this article"

Just a situation for you to understand it better:
Your organization is a show company and you are the CEO. You want AD to store the shoe size of every employee of your company so that you can gift them a pair of nice shoes on any specific eve. So you add an attribute to schema.
Now you pack one of your DCs just to feel that your AD will be secure as u have a backup DC packed :-).
After some time, you decide not to include "the shoe attribute" in AD and you remove it (DEFUNC it). Now, after 6 months, you got your packed DC out and made it replicate.
Now, as per me, Other DCs will complain about the deleted attrbute in schema which is still there on the packed Dc.

Hope this makes things clear ...

Regards,

Arun.
Nice Example Arun,but seriously,never heard lingering object in schema & reading through blogs also got it confirmed.
Schema & configuration parition is common in single forest with multiple domain.
Shoe example is nice..:)
The above case might happen with different forest & adding attribute to schma is not something anyone perform frequently,adding attribute,shutdown dc & reconnect to just introduce lingering object in schema,but acc to my practical experience i have never heard or faced issue with lingering object in schema till now, even had same question discussed with one of MCS guy, they also never heard such case,but i donno you had actually faced something or just don't want to accept the fact.
@ Awinish: as this thread is about something else, lets communicate of e-mail? I know what I am talking about and I can make you understant that...

kohliarun@hotmail.com.

Regards,

Arun.
@Awinish;

I'm afraid ARK-DS is right. After reconnecting the orphan domain, lingering object problems started.

willing be continuing and hopefully finishing it by this coming Monday or Tuesday.

@ARK-DS

Great to know you used to be MS Support. Lolz. Small world.
There is no easy or one shot answer to my question. But expert here is correct.

Microsoft needs to update their KB articles to reflect coverage to newer server OSes.

Thing to take away from this is: if the KB article exists for a problem but the newer OS is not listed within the "applies to" list, assume that it will still work.

Hopefully this won't come back to bite MS in the future though.

Good effort.
Too many steps to fix the problem.

Closed it and awarded points to ARK-DS since the first link was exactly what MS TS did when I logged the call with them.

Anyone else need help with a similar problem mail me at webmaster@midus-fx.com