Server 2003 Domain Controllers Exceeded Tombstone

Hi,
We have two domain controllers at a school and they were shut down for the summer.  Now they have exceeded the tombstone limit or whatever and won't talk to each other or replicate.  No workstations can access shares and even the DHCP on one DC isn't working and workstations can't even obtain IP addresses.  I have tried everything and one error just leads to another one when I do anything with ntdsutil or repadmin.  I've tried to remove lingering objects but get "DSA object not found" on the server I'm doing it on.  So I have w2003server which had all the fsmo roles and w2003-teachers.  Can somebody help to get these things talking to each other please?  

P.S. In an effort to just remove AD from w2003-teachers, I seized the FSMO roles because the installation couldn't transfer stuff to w2003server but it still wouldn't let me uninstall AD so now both servers, while not talking, have all the fsmo roles.  Prolly gonna have to deal with that once I get them back online.
Netdiag.txt
Replication.txt
Jasonhill630Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mike KlineCommented:
So you have the one DC that didn't communicate and you seized the FSMO roles off that DC.    For all intents and purposes that DC should not be brought back online at this point.

How did you try and remove AD?  

You can use dcpromo /forceremoval

Then you have to cleanout that DC using metadata cleanup (from a good DC)

http://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx

Once that has replicated you can join the machine back to the domain and promote it.  

You could also just clean the metadata and rebuild that dead machine and promote it again.

Thanks

Mike
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JaniLSCommented:
Do you have System State backups from the period before they were shut down?

Now that you have both DCs thinking they are the FSMO role holders, I would choose the DC that was the FSMO holder and do an authoritative restore with the other DC offlined.
You will need the Active Directory Recovery Mode password when you boot it up to do restore.

After the newly restored DC has went through the restore and the event viewer shows active directory is running you can do an unauth restore on the other DC. Once the other DC is restored you can online the second DC and allow them to sync.

Before you went through seizing roles and whatnot, you may have been able to change the bios date to 60 days ago and let them sync then move it forward to current date time and let them sync again?
0
Jasonhill630Author Commented:
Thanks for your response.  I couldn't communicate with either and doing dcpromo said I couldn't remove it because it wasn't the infrastructure master.  Now even though it is, it's like it still sees it because even if I say w2003-teachers is the last DC in the domain, it still tells me that it isn't.  Right now on each server, they both think they have all the fsmo roles.  It errored out on the metadata cleanup too.  It's like everything I did brought up another error I was trying to work around.  So should I try dcpromo /forceremoval still?
0
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

Mike KlineCommented:
On the box that you sezied the FSMO roles on yes.  

How many DCs did you have to start with...sounds like 2.
0
JaniLSCommented:
You may have to go through the some extreme measures which many experts here will walk you through but can you tell us if these are the only two DCs in AD and if your have System State backups from the period before they were offlined?
0
Jasonhill630Author Commented:
Yes, just two.
0
Mike KlineCommented:
ok then do it on the box that first had problems and that is the same box that you seized the FSMO roles off.

Thanks

Mike
0
Jasonhill630Author Commented:
I don't have the system state backups.  They don't follow the backup solution, exactly.  And yes, I read a bunch of scenarios where doing the authoritative restore after the tombstone created a pretty big mess.  Let me see how this forceremoval thing goes.

Thanks
0
Jasonhill630Author Commented:
I removed AD from w2003-teachers now.  What's the syntax for removing the metadata now?  I run that on w2003server, right?
0
Mike KlineCommented:
Do that using ntdsutil   http://www.petri.co.il/delete_failed_dcs_from_ad.htm

By the way once you get to 2008 and you need to do this metadata cleanup is much easier.  You just need to delete the old DC in ADUC....for now use the steps in the blog above from your good DC.

Thanks

Mike
0
Jasonhill630Author Commented:
Now just reinstall AD on teachers?
0
Mike KlineCommented:
So just to make sure we are on the same page

2 DCs oriingally Teachers and Other

FSMO roles seized off teachers to other
dcpromo /forceremoval on teachers
metadata cleanup of Teachers on Other

Check sites and services/AD/DNS make sure Teachers is gong (spot check)

Thanks

Mike
0
Jasonhill630Author Commented:
Other had FSMO roles first.  Seized roles ON Teachers but since they never communicated, Teachers and other both still showed themselves as having FSMO roles.  Did dcpromo /forceremoval on teachers.  Metadata cleanup on other and removed DC Teachers.  Teachers no longer showing up in ADUAC but is still showing up in sites and services and still has SRV records in DNS on other.  Ldap and kerberos...and A record.
0
Mike KlineCommented:
Damn I had it backwards, I might have had you take down others.  You can delete teachers in sites and services and the DNS records can be deleted.
0
Jasonhill630Author Commented:
Well since Others still had the roles and Teachers was acting like a lil bish, it was probably better we demoted Teachers, right?
Deleted Teachers in Sites and services but A record and SRV records still exist.  Delete all manually?
Oops...reread your last.  I'll delete them.
0
Mike KlineCommented:
So now you the teachers box should be part of a workgroup, can you add it back to the domain?
0
Jasonhill630Author Commented:
So I deleted a bunch of DNS records.  A lot of SRVs, CNAMEs, and DNS Server records.  Anything else to clear out before rejoining?
0
Mike KlineCommented:
No just make sure to go through the steps in the petri article, steps after the ntdsutil part.
0
Jasonhill630Author Commented:
Error:  The domain controller for the domain could not be contacted.
0
Mike KlineCommented:
Are you pointing to the DC for DNS?
0
Jasonhill630Author Commented:
It's using Teachers and Other for DNS server.  Forgot about that. Should I remove DNS from Teachers and just point to Other for DNS??  How does it still have DNS installed on Teachers anyway?  I have teachers as the primary dns and others for secondary in DHCP coming from router.
0
Mike KlineCommented:
it shouldn't have any zones, yes point it to current DC/DNS server.
0
Jasonhill630Author Commented:
I'm retarded.  Nevermind about the DHCP thing.  It has static.
Changed DNS server to other and it's joining.
0
Jasonhill630Author Commented:
Dcpromo completed successfully.  Rebooting.
0
Mike KlineCommented:
ok, can you check replication repadmin /showreps from teacher.

Thanks

Mike
0
Jasonhill630Author Commented:
Default-First-Site-Name\W2003-TEACHERS
DC Options: (none)
Site Options: (none)
DC object GUID: e2fc5599-fd58-4ab2-b4eb-777f3ac535b3
DC invocationID: 7488e742-b100-43b5-b71b-17b97a5b7b62

==== INBOUND NEIGHBORS =====================================

DC=annunciationbvm,DC=org
    Default-First-Site-Name\W2003SERVER via RPC
        DC object GUID: 9d867b4e-d732-467d-8756-c5fbd3d1b9ef
        Last attempt @ 2013-09-11 12:33:07 was successful.

CN=Configuration,DC=annunciationbvm,DC=org
    Default-First-Site-Name\W2003SERVER via RPC
        DC object GUID: 9d867b4e-d732-467d-8756-c5fbd3d1b9ef
        Last attempt @ 2013-09-11 12:45:31 was successful.

CN=Schema,CN=Configuration,DC=annunciationbvm,DC=org
    Default-First-Site-Name\W2003SERVER via RPC
        DC object GUID: 9d867b4e-d732-467d-8756-c5fbd3d1b9ef
        Last attempt @ 2013-09-11 12:32:57 was successful.

DC=DomainDnsZones,DC=annunciationbvm,DC=org
    Default-First-Site-Name\W2003SERVER via RPC
        DC object GUID: 9d867b4e-d732-467d-8756-c5fbd3d1b9ef
        Last attempt @ 2013-09-11 12:48:12 was successful.

DC=ForestDnsZones,DC=annunciationbvm,DC=org
    Default-First-Site-Name\W2003SERVER via RPC
        DC object GUID: 9d867b4e-d732-467d-8756-c5fbd3d1b9ef
        Last attempt @ 2013-09-11 12:48:29 was successful.
0
Jasonhill630Author Commented:
Everything looks good.  I read somewhere that the global catalog and infrastructure master shouldn't be on the same DC.  Is that true?
0
Mike KlineCommented:
That is good to see, schema/config/domain partitions all replicating and your DNS zones (application partitions) full success.

Just for fun...can you run netdom query fsmo...make sure the FSMOs are on the box you seized them too.

Thanks

Mike
0
Jasonhill630Author Commented:
Ran query.  FSMO roles are all on the "other" server where they were originally.  Good stuff.
0
Mike KlineCommented:
In a single domain environment the GC doesn't do anything, in this case both DCs should be GCs so then the IM doesn't do anything.

If you were in a multi-domain environment and all DCs were not GCs then you would separate your IM/GC

Microsoft recommends all DCs as GCs in most cases these days.  More on that here   http://blogs.technet.com/b/askds/archive/2011/09/30/friday-mail-sack-super-slo-mo-edition.aspx#gc

Thanks

Mike
0
Jasonhill630Author Commented:
Sweet.  Well, I appreciate your help with this.  I'm working remotely and the computer teacher will be in at 14:00cst.  I'll have her check out everything on the workstations and make sure we're all good before I close this.  Thanks a lot!  Been working on this for a day just trying to narrow down the problem and then battling error messages with every util I ran.
0
Mike KlineCommented:
Great and glad to help out.   We never like things going down and having issues like this but think about how much you learned going through all this.  

Just in time before she comes in :)

Thanks

Mike
0
Life1430Sr EngineerCommented:
Great discussion I like it when such prompt replies are given by Author and Expert ..
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2003

From novice to tech pro — start learning today.