Main DC not recoverable - next step?

Here is another resource for seizing Operation Masters - http://www.adminprep.com/articles/default.asp?action=show&articleid=80

It is a demo so you can watch and learn at the same time.

Brian

alimu, thanks I had spelled it wrong! shows what sorting these things out at 2:30am can do to you! I have changed the roles by seizing them, and the new server seems to slowly be realising it. Unfortunately the samba servers are still really confused, but that is another question I guess.
How can I now remove all the data in AD about the old dead server OLDSERVER? I saw something about some maintenance or something but cannot find it again.

OK, running dcdiag again gives a few problems:
 Starting test: frsevent
    There are warning or error events within the last 24 hours after the
    SYSVOL has been shared.  Failing SYSVOL replication problems may cause
    Group Policy problems.
    ......................... NEWSERVER failed test frsevent
[snip]
 Starting test: systemlog
    An Error Event occured.  EventID: 0xC00009C9
       Time Generated: 08/07/2005   12:17:54
       Event String: The server could not bind to the transport
    An Error Event occured.  EventID: 0x825A0011
       Time Generated: 08/07/2005   12:18:14
       Event String: Time Provider NtpClient: An error occurred during
    An Error Event occured.  EventID: 0xC25A001D
       Time Generated: 08/07/2005   12:18:14
       Event String: The time provider NtpClient is configured to
    An Error Event occured.  EventID: 0x825A0011
       Time Generated: 08/07/2005   12:18:48
       Event String: Time Provider NtpClient: An error occurred during
    An Error Event occured.  EventID: 0xC25A001D
       Time Generated: 08/07/2005   12:18:48
       Event String: The time provider NtpClient is configured to
    An Error Event occured.  EventID: 0xC0001F60
       Time Generated: 08/07/2005   12:21:05
       Event String: The browser service has failed to retrieve the
    An Error Event occured.  EventID: 0xC25A001D
       Time Generated: 08/07/2005   12:25:34
       Event String: The time provider NtpClient is configured to
    An Error Event occured.  EventID: 0xC25A001D
       Time Generated: 08/07/2005   12:47:22
       Event String: The time provider NtpClient is configured to
    ......................... NEWSERVER failed test systemlog
[snip]
      Starting test: FsmoCheck
         Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
         A Global Catalog Server could not be located - All GC's are down.
         ......................... domain.mydomain.net failed test FsmoCheck
[snip]

I changed NEWSERVER to a GC in AD sites and services, and now it passes FsmoCheck, but there are a lot of errors about userenv in the application log:
Windows cannot query for the list of Group Policy objects. Check the event log for possible messages previously logged by the policy engine that describes the reason for this.
and
Windows cannot access the file gpt.ini for GPO CN={5CFEF03E-F84B-452E-B9C7-XXXXXXXXXX},CN=Policies,CN=System,DC=domain,DC=mydomain,DC=net. The file must be present at the location <\\domain.mydomain.net\SysVol\domain.mydomain.net\Policies\{5CFEF03E-F84B-452E-B9C7-XXXXXXXXXXX}\gpt.ini>. (Access is denied. ). Group Policy processing aborted.  (The names were changed and XXXs were put in SIDs to protect the innocent :)

So I'm not repeating what others have already said here I'm using PAQs where possible :)

Check your win32 time service is running, if yes have a look at this PAQ,
Netman66 takes a user through troubleshooting the ntp issues you're getting here:
http:/Q_21450017.html#14171197

For "the server could not bind to the transport"
Do you have the File and Print Service component installed?
See: http://support.microsoft.com/?kbid=314872

For: "The browser service has failed to retrieve the"
Can you look at your system log, find and post the corresponding error so that we have all the details?

For: gpt.ini error, this is possibly to do with SMB signing so take a look at: http://support.microsoft.com/?kbid=839499

(sorry for sending you off to links, but they say it better than I ever could :)  )

Let me know how you go, and if you post the systemlog event requested above, I'll check it out asap.

Hi, sorry for the delay getting back, I have been running around fixing the "small stuff".

#####
OK, the requested system log event:
The browser service has failed to retrieve the backup list too many times on transport \Device\NetBT_Tcpip_{BLA_BLA_BLA}. The backup browser is stopping.
#####

#####
This might be related?:
C:\>browstat status
Status for domain MYDOMAIN on transport \Device\NetBT_Tcpip_{BLA_BLA_BLA}
    Browsing is active on domain.
    Master browser name is: server2
        Master browser is running build 3790
    3 backup servers retrieved from master server2
        \\server2
        \\(an old sql server - not DC)
        \\(an old 2k server - not DC)
    Unable to retrieve server list from server2: 64
#####
Why is it unable to retreive server list? (just a recap: server 2 is the old DC that did not crash, one of the 2 old ones)

#####
This was in the Directory Service log:
EVENT ID: 1307
SOURCE  : NTDS KCC
The Knowledge Consistency Checker (KCC) has detected that attempts to establish a replication link with the following domain controller has consistently failed.  
Attempts:
8
Domain controller:
CN=NTDS Settings,CN=OLDSERVER,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=microlinkhouse,DC=microlink,DC=ne
Period of time (minutes):
125
The Connection object for this domain controller will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this domain controller resumes, the temporary connection will be removed.
Additional Data
Error value:
1722 The RPC server is unavailable.
#####
I dont think this one is so serious, it is looking for the dead old server, which will never come back - I found something on removing this, but any suggestions welcome.


#####
Another wierd error:
This is the replication status for the following directory partition on the local domain controller.
Directory partition:
DC=DomainDnsZones,DC=mydomain,DC=mydomain,DC=net
The local domain controller has not received replication information from a number of domain controllers within the configured latency interval.
Latency Interval (Hours):
24
Number of domain controllers in all sites:
1
Number of domain controllers in this site:
1
#####
I guess this is also trying to sync with the dead OLDSERVER?

Group policy editing is a bit wierd, when I edit a GPO, it gives masses of warnings about truncating [string] for various GP entries, the [string] is the description of the various settings, ie This setting forces all users to get a clue. Default DISABLED. Is this because I have not yet installed SP1? Maybe that allowed for longer description fileds?
I have tried to just delete unneeded GPOs like login script, and make new ones, but the new ones do not seem to get applied.

Another thing is that I cannot remote desktop to the server any more. I will try to reboot tonight and see if that fixes it (it is installing SP1 now, so will need a boot anyway)

I think this is almost closed, as we seem to have a "just-about-functional" setup ATM, just these last bits.
Thanks

Apologies for taking so long to respond- have been in meetings all day.

Looks like replication's having problems because it thinks OLDSERVER still exists.

EVENT ID: 1307
SOURCE  : NTDS KCC
...CN=OLDSERVER
--> says your server's having trouble replicating to OLDSERVER.

There's a section at the bottom of this KB (last blue section) on what to do after forcible removal of a DC
http://support.microsoft.com/?kbid=332199
perhaps checking through these items would wipe out the remnants of OLDSERVER in AD and make sure nothing's trying to get to it anymore.

Perhaps check that your servers aren't pointing to OLDSERVER for DNS also - this is often something that isn't caught after decommissioning a DC.
Will also see if I can get hold of one of our other Experts to give this the once-over.

Thanks guys - looks like it is back on track
I have made NEWSERVER a GC, by going into sites and services, and ticking the box for GC in NEWSERVER. No errors seem to be happening now. Its really funny about the spelling of that command, I came in the next day seeing that, and it got me started on a fix - its crazy how its usually always the simple things!

thanks for the points, fantastic that it's all sorted.
...and Netman66 - thanks for the help, knew it'd be worth you having a look-in :)