Solved

disconnected DC needs to be reconnected...

Posted on 2008-10-20
40
2,082 Views
Last Modified: 2012-08-14
We had trouble with a VPN tunnel a while back and as a result we have a 2003 DC that has been out of sync for a really long time at a remote site.  The site has a very slow internet connection, and now has a vpn tunnel again, but it's been several months (maybe 6 or more... predates my time here) since it has replicated to the primary.  I need to find a way to make it replicate again if possible WITHOUT demoting and repromoting it.  accessing the site psysically will not be feasible since it is (literally) in a very remote village in Alaska (insert Palin joke here).  If we demote and repromote the DC then all the computers on the domain will be orphaned and someone will have to go to each one and reconnect them to the domain... I don't want to do that... i doubt you do either.... it's pretty cold that far north here in alaska this time of year... so please... don't suggest demoting and repromoting... as I am avoiding it...
0
Comment
Question by:cymrich
  • 14
  • 14
  • 6
  • +1
40 Comments
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
The disconnected domain controller is running Windows Server 2003, and an authoritative domain controller running Windows Server 2003 is available in this site or a neighboring site: Reconnect the domain controller, and immediately follow the instructions in Use Repadmin to remove lingering objects.

The about is from the following article titled "Reconnecting a Domain Controller After a Long-Term Disconnection":
http://technet.microsoft.com/en-us/library/cc786630.aspx

Use Repadmin to remove lingering objects:
http://technet.microsoft.com/en-us/library/cc785298.aspx

This should work for you.  Since you stated the DC was taken down before your time, I doubt if you kinow whether it was done via the recommended method MS discusses in the article.  If it was, then you are even better off. :)  In either case, this should get you set without the dcpromo issue, which I wouldn't have suggested anyway.
0
 

Author Comment

by:cymrich
Comment Utility
yeah, I read that before posting this... and it was right under something about "assuming" the DC not reaching tombstone end of life... since I suspect it has reached end of life I wasn't sure.  Also, it's not very clear on what "reconnecting" the DC means.  Is there a command?  do I just plug it back into the network?  Since I am working remotely and can't afford to have it get FUBARd and lock me out ,I am not willing to try things randomly without knowing what it means exactly.  i.e. I'm trying to be extremely cautious.
0
 

Author Comment

by:cymrich
Comment Utility
Here is the top paragraph I was refering to... how do I determine if it has reached this limit?  

Assuming that a domain controller has not been disconnected for longer than the maximum safe period for disconnection (tombstone lifetime minus end-to-end replication latency), reconnecting the domain controller to the replication topology requires no special procedures. By default, the Knowledge Consistency Checker (KCC) on a domain controller runs five minutes after the domain controller starts, automatically incorporating the reconnected domain controller into the replication topology.
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
here is the article on determining your tombstone liftetime.

http://technet.microsoft.com/en-us/library/cc784932.aspx
0
 
LVL 59

Accepted Solution

by:
Darius Ghassem earned 500 total points
Comment Utility
If the system has been tombstoned then the above fixes will not work. You will have to do a dcpromo then a metadata cleanup on the DC. There isn't anyway around it. Once the Server is tombstoned then you must demote then re-promote. You will most likely will have to use dcpromo /forceremoval then do a metadata cleanup.
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Do you have someone at that location with admin rights to that DC?
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
0
 
LVL 59

Expert Comment

by:Darius Ghassem
Comment Utility
Sorry left this link out. You will have to demote.

http://www.petri.co.il/delete_failed_dcs_from_ad.htm
0
 

Author Comment

by:cymrich
Comment Utility
I remembered an error in the event logs I was looking at a week or so ago... after seeing the error below where it lists the tombstone lifetime does that mean that I am pretty much out of luck and need to go buy some severe weather clothing?  

This is the replication status for the following directory partition on the local domain controller.
 
Directory partition:
DC=ForestDnsZones,DC=XXXXXXX,DC=net
 
The local domain controller has not recently received replication information from a number of domain controllers.   The count of domain controllers is shown, divided into the following intervals.
 
More than 24 hours:
1
More than a week:
1
More than one month:
1
More than two months:
1
More than a tombstone lifetime:
1
Tombstone lifetime (days):
60
 Domain controllers that do not replicate in a timely manner may encounter errors. It may miss password changes and be unable to authenticate. A DC that has not replicated in a tombstone lifetime may have missed the deletion of some objects, and may be automatically blocked from future replication until it is reconciled.
 
To identify the domain controllers by name, install the support tools included on the installation  CD and run dcdiag.exe.
You can also use the support tool repadmin.exe to display the replication latencies of the domain controllers in the forest.   The command is "repadmin /showvector /latency <partition-dn>".

For more information, see Help and Support Center at
0
 
LVL 59

Expert Comment

by:Darius Ghassem
Comment Utility
Go a head and buy some warm clothes because you are taking a trip. Follow the instructions above and you shouldn't have any issues. Make sure to follow the link I posted.
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Well, looks like you will have to go with dariousq's method and the one you were not wanting to do.  Back to my other question, do you have someone at the location who can act as an admin on that box?  How is your winter survival skills...up to date? ;-)  Might invest in some good snow shoes.
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Dang,..... Dariusq had the same warped sense of humour. ;-)
0
 
LVL 59

Expert Comment

by:Darius Ghassem
Comment Utility
I guess we do. I was thinking when he said Alaska that he might be in the world's deadliest IT career.
0
 

Author Comment

by:cymrich
Comment Utility
here and I was expecting some good Palin jokes... damn...

There is nobody up there that has admin access... there is one guy that I would be willing to give access to, but this is far above his level of knowledge with computers and I'm not sure I would want ot put him through this...

so... just to double check my suspicions... once I demote and repromote this I will have to go re-add every PC to the domain again won't I?  
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
You know...you could use the link I supplied to ship them a DC and then have them ship the one back.  You can then set that user up with rights to add workstations to the domain.  That would negate having to do anything with that tombstoned DC on that end of things.
0
 

Author Comment

by:cymrich
Comment Utility
lol... well, this all came about because a previous employee installed a satelite internet system up there to replace the supposedly 256k DSL connection (that effectively works like a 56K dial up)... the vpn tunnels wouldnt work through the satelite system because they have their own routers and it was blocking all the traffic.  Add to this the fact that they (the satelite people) could not figure out how to do a 1:1 NAT through their own routers, and then the pix firewall died, and we eventually arrive to where we are now.  

For added fun, occasionally the wind blows hard enough up there that it blows the dish out of allignment and the only guy that knew how to allign it is the one that set it all up... so eventually I am going to end up with a crash course in dish allignment I'm sure.  

All this while dodging polar bears and trying to keep from getting hypothermia...
0
 

Author Comment

by:cymrich
Comment Utility
We had thought of doing that... sending a new DC up there and sending the old one back... it brings us back to the question about the PCs though... will they have to be re-added to the domain when we do that?  Cause that would mean someone would have to go to every PC 1 at a time...
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
I am thinking yes since they have not been connected to the domain just like the DC and thus are tombstoned.  The other possibility is that they can be connected since unlike the DC, they do not need to replicate.  The good thing is you can try it without corrupting your AD like you could with a tombstoned DC coming back.  It is also easier to give a user permissions to add and remove workstations from the domain and then take that away when not needed.
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Let's find out what stage you are in:

Phantoms, Tombstones and the Active Directory Infrastructure:
http://support.microsoft.com/kb/248047
0
 
LVL 59

Expert Comment

by:Darius Ghassem
Comment Utility
If the computers were added before the tombstone then you might not will have to add the computer back to the domain. Thats a maybe though. You could also use a remote admin tool like logmein to get into the server before ice skating up there to see if you can fix the issue DC issue.
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
That's a good idea on the Logmein.  I use it here, but not for my servers.  The only problem is he will have to give the person up there the admin password to get on the server and install the program.
0
 

Author Comment

by:cymrich
Comment Utility
the admin password has changed a couple times since it got tombstoned so giving out the one that the tombstoned server thinks it is wouldn't be a big issue...

ChiefIT:  I haven't had a chance to finish reading that yet but it looks more like it's just explaining phantoms and tombstones... does it eventually tell you how to check these things?  
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Tombstones should be seen in DCdiag /v or with the ADSIutil.

I am going to do some research on how you might come up with a disaster plan>

This is what I am thinking. Only the DC is in the tombstoned state. If so, you can use the ADSIutility to remove the tombstoned object off the local server>>Demote then repromote the remote server>>then replicate from the local server to the remote server. The GUID (SID) for each comptuer on the remote site is probably still good. It will be the computers you join after the fact that will not be within the local site's active directory databse. So, those may have to rejoin the domain with those.

At this time, I want to give you a article that may help you start setting up a disaster recovery plan:
http://technet.microsoft.com/en-us/magazine/cc162459.aspx

In the meantime, I am going to look into this and get back with you.


0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Which is what the links already supplied tell him.  The problem, Chief, is stated on up in that this is a VERY remote site.  Doing a DCPROMO is not a real option since this means either cymrich traveling there or the local person doing this.
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
I am seriously looking for alternatives to performing the Ididerog for an AD restore. I do realize this is a remote DC.

Sorry for not seeing the disaster recovery link Patrick:

Have you ever heard of ADrestore: It is a command line GUI shell that allows you to browse and and restore AD tombstones. If anything can do this without a hitching up the huskies, ADrestore can.

http://technet.microsoft.com/en-us/sysinternals/bb963906.aspx
0
 

Author Comment

by:cymrich
Comment Utility
lol... hitching up the huskies... i love these references...

Never heard of ADrestore... will look into it tomorrow morning when things are quieter
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
No problem Chief. :-)  That's a good idea as well.  I came across that one as well, but the challenge of Cymrich trying to do this with a non-tech up there is as much a challenge as it is getting the Huskies to mush all the way.  Plus, when reading through the linked article that Mark (who has a lot of good tools) has on the site I didn't see any mention about restoring a tombstoned DC.  Have you used it for such?  From reading up on other articles, it sounds like this can be very touchy with a DC due to replication issues.

This could be a good time to use a paid Microsoft call and get their input.  Normally I go for a DIY approach, but with the logistics of this one, I would probably go with the paid tech call.  I have used that one time when I had an Exchange 2007 issue that I spent 4 months on.  One morning session on the phone and it was fixed.  I think the paid tech call would be cheaper than prepping a DC and then sending it up and having the other sent back (or just leave it up there).  
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
WOW, it is the logistics that are killing us here:

M$ support:
You know M$ support might be a good idear. I never thought of that because I use to work for M$ support.

I am with you fellars in thinking this has to be done remotely. We could provide a "Step-By-Step" to demote and force replicate with this local server.

Between Me, you, and Dariusq, I think we could team up and come up with a great disaster recovery plan to help out in this situation. I know of Dariusq's excellent work and I am really growning fond of your expertise Patrick.

So, what do you think> M$ or teamwork to provide a No-Brainer disaster recovery plan?

I think either plan is better than march of the penguins.
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Heh heh...yeah...I hear you Chief.  I tend to go for the DIY over paying someone else.  There are times that it pays for itself, like my Exchange 2007 issue.  I am just afraid of that distance factor and totally killing their remote sites ability to function.  I guess it would be how comfortable Cymrich would be on this.  The teamwork sounds cool.  I like this site simply because of the exchange of ideas.  While I've been a member for a while, I just never started giving input because of the amount of work I have for my real job.  However, I'm finding the more I answer the more I get better acquainted with the things I work with....if that makes any sense. :)

Cymrich:  Do you have remote capabilities on that machine?  Meaning, can it be connected to the Internet through the sattelite connection?  Or is that down?  Can you get Logmein installed like Darius suggested?  That would allow you do try the various restores suggested between all of us.
0
 

Author Comment

by:cymrich
Comment Utility
sorry about the lack of update... this got put on hold for a while.  I can remote in through remote desktop via a vpn tunnel thats setup through a very slow DSL.  so yeah... i can install things.  I think we've pretty much decided that we will need to take a new server up there... I have one of their laptops here in my office and any time it sees the domain it wont authenticate to anything anymore... so I have to hook it up to an external wireless and connec tot e-mail through the internet and such.  That basically means that even if the DC is somehow miraculously revived we still have to touch every PC in the place to set them straight too.  A coworker and I are going to do rock paper scissors to see who has to go...

the trick to this is making sure you take your polar bear repellant... i.e. anyone that moves slower than you...  probably not going to make it up there till early december though and so they should all be sleeping by now...
0
 

Author Comment

by:cymrich
Comment Utility
*cough* no penguins in the north pole fyi... just south :P
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
You can't blame Chief.  I think he spends most of his time on a ship.  Having spent some time in the Navy, I can understand how being out to sea so much would addle his mental abilities when it comes to wildlife locations. ;-)
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
LOL:

It's funny, I deal with wildlife all day. You know, I thought the little penguins were up North as well as the large Emporer Penguins on the South pole. Learn something new every day, I guess.

Here is another option:
Have one of your guys/gals up north logon to this post and we can guide him/her with a step-by-step no brainer. This includes bringing up the PCs. You do have a server that is capable of being repaired. If we can repair it, it may save you the trip.

If you decide to go up North, tell Santa I say Howdy.

 

0
 

Author Comment

by:cymrich
Comment Utility
ok... so... I was out for vacation the last week (absolutely no relation to the WoW expansion being released... none whatsoever... really...DING! 80!) while I was away there was a power outage in the remote location.  apparently it was long enough to exceed the capacity of the UPS.  when it came back up it pretty much had stopped working as a domain controller completely and nobody could log into their PCs.  The other people int he office had them unplug their network cables, log in, and then plug them back in to get them into their PCs.  Then they demoted and repromoted the DC... now the DC still doesn't work, but it seems to think it does.  I removed a PC from the domain yesterday and tried to readd it and it gives and error that dns could not resolve the name (if I use the netbios name) or no domain controllers are available (with the full domain name).  I'm thinking when it was demoted they didnt do a metadata cleanup... but since it's connection to the anchorage office is so slow it could be related to that... any thoughts?
0
 

Author Comment

by:cymrich
Comment Utility
ok... so... I was dead on with the metadata cleanup.  I had to demote the DC, do a metadata cleanup, delete everying from DNS, and then promote it again.  So far it appears things are working and I've added 2 of the PCs back to the domain successfully.  working on more.  
0
 

Author Comment

by:cymrich
Comment Utility
I did manage to do this all remotely... since the power outage FUBARd everything it was a moot point to not try cause it wasn't even marginally working anymore.  I'm going to go ahead and award the points to Darius since he suggested doing exactly what I did even though I was avoiding it due to the remote issue.  Thank you all for the suggestions and assistance!  
0
 

Author Closing Comment

by:cymrich
Comment Utility
I was avoiding doing it that way but in that end a power outage forced me to do exactly what he described to fix it.
0
 
LVL 59

Expert Comment

by:Darius Ghassem
Comment Utility
@cymrich

I'm glad everything is working and you didn't have to bring the sled out.
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
So, instead of snow day, we can Siesta.

Well done dariusq!!
0
 
LVL 4

Expert Comment

by:Patrick49er
Comment Utility
Great job Darius!  Glad everything worked out finally for you, Cymrich.  Things got really crazy here and so I didn't get a chance to log back in; not to mention my workstation's hard drive decided to crap out without any warning signs.  Man I hate setting my workstation back up.  All those tools you use and then have to remember all the ones you use to install on the freshly installed hard drive! :-(
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Setting up a Microsoft WSUS update system is free relatively speaking if you have hard disk space and processor capacity.   However, WSUS can be a blessing and a curse. For example, there is nothing worse than approving updates and they just have…
In this article, we will see the basic design consideration while designing a Multi-tenant web application in a simple manner. Though, many frameworks are available in the market to develop a multi - tenant application, but do they provide data, cod…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles from a Windows Server 2008 domain controller to a Windows Server 2012 domain controlle…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now