Techdivision
asked on
SBS Migration Hell
Have been trying to perform a migration overnight of SBS 2003 to SBS 2011. SBS 2003 server crashed last night whilst trying to migrate mailboxes over, now steadfastly refuses to boot. Hangs on 'Applying Computer Settings'.
Can boot into safe mode - only thing I can see that might help is that the Kerberos Key Distribution Service is hanging on startup. Therefore, no-one can log in, and Exchange can't start.
Can anybody help me? Am in a tricky spot, getting bombarded with text messages from angry directors who can't get e-mail. Ran dcdiag, it came back with:
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = SERVER
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\SE
Starting test: Connectivity
......................... SERVER passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\SE
Starting test: Advertising
......................... SERVER passed test Advertising
Starting test: FrsEvent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... SERVER passed test FrsEvent
Starting test: DFSREvent
......................... SERVER passed test DFSREvent
Starting test: SysVolCheck
......................... SERVER passed test SysVolCheck
Starting test: KccEvent
A warning event occurred. EventID: 0x800004A4
Time Generated: 10/12/2013 14:01:05
Event String:
A thread in Active Directory Domain Services is waiting for the comp
letion of a RPC made to the following directory service.
A warning event occurred. EventID: 0x800004D0
Time Generated: 10/12/2013 14:01:05
Event String:
Active Directory Domain Services attempted to perform a remote proce
dure call (RPC) to the following server. The call timed out and was cancelled.
......................... SERVER passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... SERVER passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... SERVER passed test MachineAccount
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=ForestDnsZones,DC=xyz,D
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=xyz,D
......................... SERVER failed test NCSecDesc
Starting test: NetLogons
......................... SERVER passed test NetLogons
Starting test: ObjectsReplicated
......................... SERVER passed test ObjectsReplicated
Starting test: Replications
[Replications Check,SERVER] A recent replication attempt failed:
From TW2 to SERVER
Naming Context: DC=ForestDnsZones,DC=xyz,D
The replication generated an error (1256):
The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
The failure occurred at 2013-10-12 13:46:19.
The last success occurred at 2013-10-12 00:46:18.
13 failures have occurred since the last success.
^C
It then hangs, so I have to Ctrl-C it. I know nothing about AD - will this service sort itself out if I just leave it for a while? This is rather time critical though, that's the problem.
Old server is called TW2, new (SBS 2011) server is called SERVER. Don't want to reboot SERVER in case I never get a chance to log onto the domain again.
Can boot into safe mode - only thing I can see that might help is that the Kerberos Key Distribution Service is hanging on startup. Therefore, no-one can log in, and Exchange can't start.
Can anybody help me? Am in a tricky spot, getting bombarded with text messages from angry directors who can't get e-mail. Ran dcdiag, it came back with:
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = SERVER
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\SE
Starting test: Connectivity
......................... SERVER passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\SE
Starting test: Advertising
......................... SERVER passed test Advertising
Starting test: FrsEvent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... SERVER passed test FrsEvent
Starting test: DFSREvent
......................... SERVER passed test DFSREvent
Starting test: SysVolCheck
......................... SERVER passed test SysVolCheck
Starting test: KccEvent
A warning event occurred. EventID: 0x800004A4
Time Generated: 10/12/2013 14:01:05
Event String:
A thread in Active Directory Domain Services is waiting for the comp
letion of a RPC made to the following directory service.
A warning event occurred. EventID: 0x800004D0
Time Generated: 10/12/2013 14:01:05
Event String:
Active Directory Domain Services attempted to perform a remote proce
dure call (RPC) to the following server. The call timed out and was cancelled.
......................... SERVER passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... SERVER passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... SERVER passed test MachineAccount
Starting test: NCSecDesc
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=ForestDnsZones,DC=xyz,D
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=xyz,D
......................... SERVER failed test NCSecDesc
Starting test: NetLogons
......................... SERVER passed test NetLogons
Starting test: ObjectsReplicated
......................... SERVER passed test ObjectsReplicated
Starting test: Replications
[Replications Check,SERVER] A recent replication attempt failed:
From TW2 to SERVER
Naming Context: DC=ForestDnsZones,DC=xyz,D
The replication generated an error (1256):
The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
The failure occurred at 2013-10-12 13:46:19.
The last success occurred at 2013-10-12 00:46:18.
13 failures have occurred since the last success.
^C
It then hangs, so I have to Ctrl-C it. I know nothing about AD - will this service sort itself out if I just leave it for a while? This is rather time critical though, that's the problem.
Old server is called TW2, new (SBS 2011) server is called SERVER. Don't want to reboot SERVER in case I never get a chance to log onto the domain again.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Have called Microsoft on this one.
What was their solution?
Alan
Alan
ASKER
5 hours online, and they still haven't solved it yet!
5 hours! Yikes - just how broken is it?
ASKER
No networking - at all. Can't solve it with netsh int ip reset, can't bring up anything under 'network connections' (blank screen), no DCOM. SFC / scannow doesn't launch, Safe Mode with networking doesn't boot, Safe Mode is the only thing I can run.
In short - it's screwed! :)
In short - it's screwed! :)
That's one big mess.
Have they fixed it for you yet or is it toast?
ASKER
Funnily enough, yes. Just about. But it took 4 Microsoft Engineers 3 days to do it. I'm so glad I pay my subscription fees.
In a nutshell, I swapped out the network adapter. That gave me enough leverage to force an IP address onto the NIC using the Netsh command. The IP address didn't ping anywhere, and couldn't do anything except ping itself, but it did at least enable the RPC service (and a few other core Microsoft services) to start up on the next boot. They couldn't do anything, but they at least saw a valid IP address, as opposed to 0.0.0.0, and started.
That, in turn, gave me enough of a system to run SFC /Scannow. That gave me enough of an IP address to ping the Internet, connect and download the Microsoft Remote Support tool. I could then call the cavalry at that stage. That was a very lonely 7 hours up to that point.
Once the cavalry arrived, they nearly panicked and ran off again. :) In the end, the only way I could restore the Exchange database was to copy off the information stores (and outstanding log files) onto a Server 2003 machine I'd built especially for the occasion (a Windows XP machine would have done, apparently), and then copied the Eseutil.exe, Ese.dll, Jcb.dll, Exosal.dll, and Exchmem.dll files from the Exchange Server 2003 computer's C:\Exchsrvr\Bin folder to the new server. That gave me enough files to replay the event logs, put the Exchange DB file into a clean shutdown state, and then copy it back onto the Exchange server.
A handy tip to note for future reference...
In a nutshell, I swapped out the network adapter. That gave me enough leverage to force an IP address onto the NIC using the Netsh command. The IP address didn't ping anywhere, and couldn't do anything except ping itself, but it did at least enable the RPC service (and a few other core Microsoft services) to start up on the next boot. They couldn't do anything, but they at least saw a valid IP address, as opposed to 0.0.0.0, and started.
That, in turn, gave me enough of a system to run SFC /Scannow. That gave me enough of an IP address to ping the Internet, connect and download the Microsoft Remote Support tool. I could then call the cavalry at that stage. That was a very lonely 7 hours up to that point.
Once the cavalry arrived, they nearly panicked and ran off again. :) In the end, the only way I could restore the Exchange database was to copy off the information stores (and outstanding log files) onto a Server 2003 machine I'd built especially for the occasion (a Windows XP machine would have done, apparently), and then copied the Eseutil.exe, Ese.dll, Jcb.dll, Exosal.dll, and Exchmem.dll files from the Exchange Server 2003 computer's C:\Exchsrvr\Bin folder to the new server. That gave me enough files to replay the event logs, put the Exchange DB file into a clean shutdown state, and then copy it back onto the Exchange server.
A handy tip to note for future reference...
That sounds like you had a heap of fun. Pretty good timing on the old server to die at exactly the wrong point.
Hopefully it will all be a distant memory soon enough.
Alan
Hopefully it will all be a distant memory soon enough.
Alan
If that works, once in normal mode, set the stores to not automatically mount and then start the information store service.
If all is good at this stage, you will need to repair the stores and integrity check them before you can proceed:
eseutil /p
eseutil /d
isinteg -s <servername> -fix -test alltests
Mount stores - continue migration!
Alan