disconnected DC needs to be reconnected...

yeah, I read that before posting this... and it was right under something about "assuming" the DC not reaching tombstone end of life... since I suspect it has reached end of life I wasn't sure.  Also, it's not very clear on what "reconnecting" the DC means.  Is there a command?  do I just plug it back into the network?  Since I am working remotely and can't afford to have it get FUBARd and lock me out ,I am not willing to try things randomly without knowing what it means exactly.  i.e. I'm trying to be extremely cautious.

Here is the top paragraph I was refering to... how do I determine if it has reached this limit?  

Assuming that a domain controller has not been disconnected for longer than the maximum safe period for disconnection (tombstone lifetime minus end-to-end replication latency), reconnecting the domain controller to the replication topology requires no special procedures. By default, the Knowledge Consistency Checker (KCC) on a domain controller runs five minutes after the domain controller starts, automatically incorporating the reconnected domain controller into the replication topology.

here is the article on determining your tombstone liftetime.

http://technet.microsoft.com/en-us/library/cc784932.aspx

Do you have someone at that location with admin rights to that DC?

Take a look at this post on server newsgroups:

http://www.servernewsgroups.net/group/microsoft.public.windows.server.active_directory/topic12873.aspx

Sorry left this link out. You will have to demote.

http://www.petri.co.il/delete_failed_dcs_from_ad.htm

I remembered an error in the event logs I was looking at a week or so ago... after seeing the error below where it lists the tombstone lifetime does that mean that I am pretty much out of luck and need to go buy some severe weather clothing?  

This is the replication status for the following directory partition on the local domain controller.
 
Directory partition:
DC=ForestDnsZones,DC=XXXXXXX,DC=net
 
The local domain controller has not recently received replication information from a number of domain controllers.   The count of domain controllers is shown, divided into the following intervals.
 
More than 24 hours:
1
More than a week:
1
More than one month:
1
More than two months:
1
More than a tombstone lifetime:
1
Tombstone lifetime (days):
60
 Domain controllers that do not replicate in a timely manner may encounter errors. It may miss password changes and be unable to authenticate. A DC that has not replicated in a tombstone lifetime may have missed the deletion of some objects, and may be automatically blocked from future replication until it is reconciled.
 
To identify the domain controllers by name, install the support tools included on the installation  CD and run dcdiag.exe.
You can also use the support tool repadmin.exe to display the replication latencies of the domain controllers in the forest.   The command is "repadmin /showvector /latency <partition-dn>".

For more information, see Help and Support Center at

Go a head and buy some warm clothes because you are taking a trip. Follow the instructions above and you shouldn't have any issues. Make sure to follow the link I posted.

Well, looks like you will have to go with dariousq's method and the one you were not wanting to do.  Back to my other question, do you have someone at the location who can act as an admin on that box?  How is your winter survival skills...up to date? ;-)  Might invest in some good snow shoes.

Dang,..... Dariusq had the same warped sense of humour. ;-)

I guess we do. I was thinking when he said Alaska that he might be in the world's deadliest IT career.

here and I was expecting some good Palin jokes... damn...

There is nobody up there that has admin access... there is one guy that I would be willing to give access to, but this is far above his level of knowledge with computers and I'm not sure I would want ot put him through this...

so... just to double check my suspicions... once I demote and repromote this I will have to go re-add every PC to the domain again won't I?  

You know...you could use the link I supplied to ship them a DC and then have them ship the one back.  You can then set that user up with rights to add workstations to the domain.  That would negate having to do anything with that tombstoned DC on that end of things.

lol... well, this all came about because a previous employee installed a satelite internet system up there to replace the supposedly 256k DSL connection (that effectively works like a 56K dial up)... the vpn tunnels wouldnt work through the satelite system because they have their own routers and it was blocking all the traffic.  Add to this the fact that they (the satelite people) could not figure out how to do a 1:1 NAT through their own routers, and then the pix firewall died, and we eventually arrive to where we are now.  

For added fun, occasionally the wind blows hard enough up there that it blows the dish out of allignment and the only guy that knew how to allign it is the one that set it all up... so eventually I am going to end up with a crash course in dish allignment I'm sure.  

All this while dodging polar bears and trying to keep from getting hypothermia...

We had thought of doing that... sending a new DC up there and sending the old one back... it brings us back to the question about the PCs though... will they have to be re-added to the domain when we do that?  Cause that would mean someone would have to go to every PC 1 at a time...

I am thinking yes since they have not been connected to the domain just like the DC and thus are tombstoned.  The other possibility is that they can be connected since unlike the DC, they do not need to replicate.  The good thing is you can try it without corrupting your AD like you could with a tombstoned DC coming back.  It is also easier to give a user permissions to add and remove workstations from the domain and then take that away when not needed.

Let's find out what stage you are in:

Phantoms, Tombstones and the Active Directory Infrastructure:
http://support.microsoft.com/kb/248047

If the computers were added before the tombstone then you might not will have to add the computer back to the domain. Thats a maybe though. You could also use a remote admin tool like logmein to get into the server before ice skating up there to see if you can fix the issue DC issue.

That's a good idea on the Logmein.  I use it here, but not for my servers.  The only problem is he will have to give the person up there the admin password to get on the server and install the program.

the admin password has changed a couple times since it got tombstoned so giving out the one that the tombstoned server thinks it is wouldn't be a big issue...

ChiefIT:  I haven't had a chance to finish reading that yet but it looks more like it's just explaining phantoms and tombstones... does it eventually tell you how to check these things?  

Tombstones should be seen in DCdiag /v or with the ADSIutil.

I am going to do some research on how you might come up with a disaster plan>

This is what I am thinking. Only the DC is in the tombstoned state. If so, you can use the ADSIutility to remove the tombstoned object off the local server>>Demote then repromote the remote server>>then replicate from the local server to the remote server. The GUID (SID) for each comptuer on the remote site is probably still good. It will be the computers you join after the fact that will not be within the local site's active directory databse. So, those may have to rejoin the domain with those.

At this time, I want to give you a article that may help you start setting up a disaster recovery plan:
http://technet.microsoft.com/en-us/magazine/cc162459.aspx

In the meantime, I am going to look into this and get back with you.

Which is what the links already supplied tell him.  The problem, Chief, is stated on up in that this is a VERY remote site.  Doing a DCPROMO is not a real option since this means either cymrich traveling there or the local person doing this.

I am seriously looking for alternatives to performing the Ididerog for an AD restore. I do realize this is a remote DC.

Sorry for not seeing the disaster recovery link Patrick:

Have you ever heard of ADrestore: It is a command line GUI shell that allows you to browse and and restore AD tombstones. If anything can do this without a hitching up the huskies, ADrestore can.

http://technet.microsoft.com/en-us/sysinternals/bb963906.aspx

lol... hitching up the huskies... i love these references...

Never heard of ADrestore... will look into it tomorrow morning when things are quieter

No problem Chief. :-)  That's a good idea as well.  I came across that one as well, but the challenge of Cymrich trying to do this with a non-tech up there is as much a challenge as it is getting the Huskies to mush all the way.  Plus, when reading through the linked article that Mark (who has a lot of good tools) has on the site I didn't see any mention about restoring a tombstoned DC.  Have you used it for such?  From reading up on other articles, it sounds like this can be very touchy with a DC due to replication issues.

This could be a good time to use a paid Microsoft call and get their input.  Normally I go for a DIY approach, but with the logistics of this one, I would probably go with the paid tech call.  I have used that one time when I had an Exchange 2007 issue that I spent 4 months on.  One morning session on the phone and it was fixed.  I think the paid tech call would be cheaper than prepping a DC and then sending it up and having the other sent back (or just leave it up there).  

WOW, it is the logistics that are killing us here:

M$ support:
You know M$ support might be a good idear. I never thought of that because I use to work for M$ support.

I am with you fellars in thinking this has to be done remotely. We could provide a "Step-By-Step" to demote and force replicate with this local server.

Between Me, you, and Dariusq, I think we could team up and come up with a great disaster recovery plan to help out in this situation. I know of Dariusq's excellent work and I am really growning fond of your expertise Patrick.

So, what do you think> M$ or teamwork to provide a No-Brainer disaster recovery plan?

I think either plan is better than march of the penguins.

Heh heh...yeah...I hear you Chief.  I tend to go for the DIY over paying someone else.  There are times that it pays for itself, like my Exchange 2007 issue.  I am just afraid of that distance factor and totally killing their remote sites ability to function.  I guess it would be how comfortable Cymrich would be on this.  The teamwork sounds cool.  I like this site simply because of the exchange of ideas.  While I've been a member for a while, I just never started giving input because of the amount of work I have for my real job.  However, I'm finding the more I answer the more I get better acquainted with the things I work with....if that makes any sense. :)

Cymrich:  Do you have remote capabilities on that machine?  Meaning, can it be connected to the Internet through the sattelite connection?  Or is that down?  Can you get Logmein installed like Darius suggested?  That would allow you do try the various restores suggested between all of us.

sorry about the lack of update... this got put on hold for a while.  I can remote in through remote desktop via a vpn tunnel thats setup through a very slow DSL.  so yeah... i can install things.  I think we've pretty much decided that we will need to take a new server up there... I have one of their laptops here in my office and any time it sees the domain it wont authenticate to anything anymore... so I have to hook it up to an external wireless and connec tot e-mail through the internet and such.  That basically means that even if the DC is somehow miraculously revived we still have to touch every PC in the place to set them straight too.  A coworker and I are going to do rock paper scissors to see who has to go...

the trick to this is making sure you take your polar bear repellant... i.e. anyone that moves slower than you...  probably not going to make it up there till early december though and so they should all be sleeping by now...

*cough* no penguins in the north pole fyi... just south :P

You can't blame Chief.  I think he spends most of his time on a ship.  Having spent some time in the Navy, I can understand how being out to sea so much would addle his mental abilities when it comes to wildlife locations. ;-)

LOL:

It's funny, I deal with wildlife all day. You know, I thought the little penguins were up North as well as the large Emporer Penguins on the South pole. Learn something new every day, I guess.

Here is another option:
Have one of your guys/gals up north logon to this post and we can guide him/her with a step-by-step no brainer. This includes bringing up the PCs. You do have a server that is capable of being repaired. If we can repair it, it may save you the trip.

If you decide to go up North, tell Santa I say Howdy.

 

ok... so... I was out for vacation the last week (absolutely no relation to the WoW expansion being released... none whatsoever... really...DING! 80!) while I was away there was a power outage in the remote location.  apparently it was long enough to exceed the capacity of the UPS.  when it came back up it pretty much had stopped working as a domain controller completely and nobody could log into their PCs.  The other people int he office had them unplug their network cables, log in, and then plug them back in to get them into their PCs.  Then they demoted and repromoted the DC... now the DC still doesn't work, but it seems to think it does.  I removed a PC from the domain yesterday and tried to readd it and it gives and error that dns could not resolve the name (if I use the netbios name) or no domain controllers are available (with the full domain name).  I'm thinking when it was demoted they didnt do a metadata cleanup... but since it's connection to the anchorage office is so slow it could be related to that... any thoughts?

ok... so... I was dead on with the metadata cleanup.  I had to demote the DC, do a metadata cleanup, delete everying from DNS, and then promote it again.  So far it appears things are working and I've added 2 of the PCs back to the domain successfully.  working on more.  

I did manage to do this all remotely... since the power outage FUBARd everything it was a moot point to not try cause it wasn't even marginally working anymore.  I'm going to go ahead and award the points to Darius since he suggested doing exactly what I did even though I was avoiding it due to the remote issue.  Thank you all for the suggestions and assistance!  

I was avoiding doing it that way but in that end a power outage forced me to do exactly what he described to fix it.

@cymrich

I'm glad everything is working and you didn't have to bring the sled out.

So, instead of snow day, we can Siesta.

Well done dariusq!!