?
Solved

exchange server 2003 SP2 has dropped off domain. Won't rejoin.

Posted on 2011-05-11
23
Medium Priority
?
571 Views
Last Modified: 2012-05-11
without warning yesterday our Exch 2003 SP2 server dropped off the domain
the first we knew was users complaining that email isn't working
We have a win 2008 R2 PDC and a Win 2003 SP2 BDC

In the logs of the 2003 BDC is event id 5723 (NETLOGON):
===================================================
Event Type:      Error
Event Source:      NETLOGON
Event Category:      None
Event ID:      5723
Date:            10/05/2011
Time:            14:34:15
User:            N/A
Computer:      BHS-ADS02
Description:
The session setup from computer 'BHS-MS-001' failed because the security database does not contain a trust account 'BHS-MS-001$' referenced by the specified computer.  

USER ACTION  
If this is the first occurrence of this event for the specified computer and account, this may be a transient issue that doesn't require any action at this time. Otherwise, the following steps may be taken to resolve this problem:  

If 'BHS-MS-001$' is a legitimate machine account for the computer 'BHS-MS-001', then 'BHS-MS-001' should be rejoined to the domain.  

If 'BHS-MS-001$' is a legitimate interdomain trust account, then the trust should be recreated.  

Otherwise, assuming that 'BHS-MS-001$' is not a legitimate account, the following action should be taken on 'BHS-MS-001':  

If 'BHS-MS-001' is a Domain Controller, then the trust associated with 'BHS-MS-001$' should be deleted.  

If 'BHS-MS-001' is not a Domain Controller, it should be disjoined from the domain.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 8b 01 00 c0               ¿..À    
================================================

A similar event occurs in the exchange server logs. Curiously there aren't any events in the 2008 R2 PDC logs which relate to the Exchange server.

What we have tried (three times):
Remove Exch Server (BHS-MS-001) from Domain. <reboot>
Remove BHS-MS-001 from AD (both DCs)
Add BHS-MS-001 back in to domain <reboot>
what happens:
Server takes an age to start, especially at applying computer settings stage.
After start, still not working.

Because this is an Exchange Server is there something extra which needs to be done? I've seen mention elsewhere of trust relationships, but am unsure how to check if those are in place. Are there authentication settings which need to be verified?

Please help!

0
Comment
Question by:ComputerFundi
  • 11
  • 5
  • 4
  • +1
23 Comments
 
LVL 33

Expert Comment

by:Busbar
ID: 35736052
looks like the exchange computer account has been deleted. can you recover it.
0
 

Author Comment

by:ComputerFundi
ID: 35736069
Busbar
Please clarify terminology:
Is the exchange computer account the same as the computer account which gets created when you add the server to the domain or is it something different?

If the same, are you suggesting that the account should be recovered rather than just re-created by joining the domain?

If so, how do I go about recovering it?

Thanks
0
 
LVL 33

Expert Comment

by:Busbar
ID: 35736079
yes, this what I meant, looks like it is deleted somehow.
some poeple recover it using join-disjoin however I am reluctant to suggest that.
0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 

Author Comment

by:ComputerFundi
ID: 35736103
so what method are you suggesting to restore?

restore whole AD database from backup or some tool which allows recovery of individual objects?
0
 
LVL 33

Expert Comment

by:Busbar
ID: 35736128
yes, but you can try join/disjoin one
0
 

Author Comment

by:ComputerFundi
ID: 35736195
as I said, I have already removed from domain and re-joined three times. If that's what you mean by join-disjoin then that's not working, so I'll have to try a recovery method.
0
 
LVL 5

Assisted Solution

by:CWCertus1
CWCertus1 earned 800 total points
ID: 35736243
You will need to perform an authoritative restore of the computer object on a domain controller. This can be done on your 2003 DC if you like as long as it is authoritative.

Please see the following MS article on how to do this.
http://technet.microsoft.com/en-us/library/cc779573(WS.10).aspx

Being as you have removed it from the domain as well, it may have taken important exchange objects with it... if just restoring the computer account does not fix this, then you may need a lot more assistance
0
 
LVL 9

Expert Comment

by:rpartington
ID: 35736251
Before I removed and rejoined I would have manually copied the exc databases and log folders off to a seperate folder before removing and rejoining, but thats just me, as a belts and braces approach I would have also imaged the exc beforehand a bit ott I know but I prefer to have as many options to get out of jail as possible.

When you removed the exchange did you check each DC to see if the exchange had being removed from which ever ad container you had it in, BEFORE you rejoined it again to the domain.

Has the exc definately got one of OR both the IPS for each of the dcs in its nic dns settings.
0
 

Author Comment

by:ComputerFundi
ID: 35736290
waiting for it to complete starting and then I'll respond to these questions .... thanks to you both
0
 

Author Comment

by:ComputerFundi
ID: 35736394
Some further clarification.
The EVENTID 5723 appeared at 14:28 yesterday. at that point the computer account was still in place on at least the PDC (BDC wasn't checked). A reboot was tried, but didn't solve the problem. The exch was booted in safe mode and removed from the domain. This was at 16.02. At this point the computer acct was removed from the PDC, but not the BDC.
There have been two further attempts to simply re-join the domain, but each fails because the server takes approx 1 hour to start and then so many services are not working.

PLEASE CLARIFY:
1  Is an authoritative restore the best approach?
2 the repeated disjoin-rejoin process will have levt several 'tombstone' accts for the server in AD. Will these compromise the likely success of the authoritative restore?
4 whilst doing the restore should the exch server be off the domain?
3 CWCerus1 - are you suggesting doing the restore on the BDC so that I can keep the PDC (and the network) online. If the restore is successful will the acct replicate back to the PDC? Do I need to ensure this has happened before starting the exch server?

Thanks
0
 
LVL 9

Accepted Solution

by:
rpartington earned 1200 total points
ID: 35736606
1  Is an authoritative restore the best approach?
The issue with this is, you have removed and rejoined at least 3 times now and the authoritative restore is pointing to the old Exc SID before you removed it from the domain. (Thats not to say its not worth giving it a try, just something to keep in mind)
2 the repeated disjoin-rejoin process will have levt several 'tombstone' accts for the server in AD. Will these compromise the likely success of the authoritative restore?
If the server was completely removed correctly from both DCS not just the one, there will be no tombstone issues as AD will have no pointers to it.

One other option you may also want to consider is:
Copy the Exchange/Public folder databases and logs to a seperate location by this I mean copy their respective folders with the logs and databases to a USB or server,

Completely rebuild the exchange with the same netbios/ip address etc all the M/S patches, Join it to the domain as a member server as you had previously, install Exchange onto it so at this point in time it is identical to what it was when it was working previously APART from it has new logs and databases.
You at this point have 2 options.
If you have a backup program you can restore the message store to it.
or
Simply copy the new empty exchange database and log folders over to a differant location as a fail safe.
Then copy and paste the original database and logs from your usb drive from the old exchange build into their new respective locations. But just make sure that the EXCHANGE services are all turned off when you do this.
Turn the services back on.
You will now have to re-associate the users in AD with their mailboxes in your new ex build which is a  pretty straight forward process.
We had to do this for one client who had not taken any backups so we had no choice and it worked absolutely perfect as long as you remember to turn off the exchange services when doing the copy process and overwrite process.

Just something else to think about as having an exchange server down is no fun with everyone screaming at you asking how long before its fixed.
0
 
LVL 9

Expert Comment

by:rpartington
ID: 35736613
To clarify I meant Both the exchange and public folder databases with the below comment that I put in my previous comment but Im sure you knew this.
>>>Copy the Exchange/Public folder databases <<<
0
 
LVL 5

Expert Comment

by:CWCertus1
ID: 35736631
Power the exchange server off while you do the auth restore. I am suggesting on the BDC so the PDC can service other requests. BDC will update the PDC... (There is not really such thing as PDC/BDC since windows 2000 AD came about. The PDC emulator is a role and only 1 server per domain can have it).

Delete the tombstoned accounts before restoring the original to avoid any confusion over which is which. Whichever server is the Global Catalog server will need the account to be replicated to it before you turn exchange server back on.
0
 
LVL 5

Expert Comment

by:CWCertus1
ID: 35736656
To address rpartington's comment, if the AD restore of that single object does not fix the issue then this may be worth considering but I would not spend time rebuilding the server (if hardware is even available to do this) whilst exchange is down before trying to simply restore an object.
0
 

Author Comment

by:ComputerFundi
ID: 35737011
We're having some difficulty getting hold of the system state backup for the PDC or BDC. I won't bore with all the details but basically our Backup Exec 2010 server failed recently. We re-installed BE and now upon cataloging the recent full backups the system state portion is missing. don't ask me why ... they were definitely there before.

So I'm now starting to think that a full exchange server rebuild is required. I am current yl copying all the public and private stores and logs off the server in preparation for that. :(

These things always seem to happen in twos and threes ...
0
 
LVL 5

Expert Comment

by:CWCertus1
ID: 35738100
Is this being rebuilt onto the same hardware or is it a seperate server.

I would still take a full backup even in this state before completely blowing it away.
If the rebuild does not work because deleting the server object took vital exchange schema objects with it, you may need to start with a fresh server at some stage and you may struggle to get the stores mounted. I would either exmerge a copy of your mail out now if you have the external storage or run a full backup of that server now so you can do this in future and prepare for the worst as you don't have good backups.
0
 

Author Comment

by:ComputerFundi
ID: 35738339
We have good recent backups of the exchange server + stores + system state. It's the PDC and BDC system state backups which we are missing. proceeding with rebuild on same hardware now ...
0
 
LVL 5

Expert Comment

by:CWCertus1
ID: 35738356
good luck
0
 

Author Comment

by:ComputerFundi
ID: 35744061
luck on our side so far! ...
Win 2003 Server installed + M/updates ... OK
Exchange 2003 installed + SP2 ... OK
Joined to domain and present in AD without errors ... OK
Stores restored from tape to 1 AM on Tuesday May 10th ... OK

Remaining task is to replay the logs from 1AM to 2.45 pm on Tuesday (when server went offline) to bring the stores fully up-to-date. This is where I am struggling. Following eseutil articles on ms support ad technet hasn't brought success.
Advice would be appreciated.

A bit more detail ...
When the stores were restored from tape I also selected to restore any log files into temp directory and left the 'last restore set' box unticked.
After the restore there was one log file (E0002492.log) + restore.env in that directory. In a separate folder saved on external drive I have E0002492.log to E00024B5.log + E00.log ... 36 files all in date / time sequence with nothing missing.
So I copied all these into the temp logs directory and then ran eseutil /cc
It processed cleanly, but only the first file E0002492.log.
I tried eseutil /r E00 but this gave a -2416 error on the next file (E0002493) with a message about an attachment not found.
I've run consistency and signature checks on all the log files and all are clean and consistent to E0002493, but I can't figure out how to move forward from here.

I'm guessing that I did something wrong with the eseutil processes described above.

I think I have three options from here.
I am certain that the store backup to tape is good - to 1AM tuesday
I am certain that the transaction logs and store databases I have on the external disk are good - to 14.45 tuesday.

1 I haven't messed up the logs and someone can give me the necessary syntax to get the logs replayed and bring the stores to 14:45 tuesday.
2 I do the restore from tape again (took about 15 minutes) and do the replay logs correctly.
3 I copy the store db fiels from the external disks and then re-associate the mailboxes to the AD accounts.

I probably would have done option 3 originally (and it was one of rpartington suggestions) but I don't know how to do the re-associate other than one by one and there are 1500 mailboxes ....

one last thing:
I temporarily mounted the student store (the least crucial) to make sure it mounts successfully and I have logged in as a test student & started outlook (for 30 secs) just to ensure the folders are opened ... OK
This was at 6:30 AM so I'm sure no student would have accessed the store in that time. The store was immediately dismounted.

0
 

Author Comment

by:ComputerFundi
ID: 35745561
Update.
After loads of reading around the use of ESEutil and logs etc, I decided to copy back the database files from the point at which the server went offline. A run of eseutil /mh confirmed they were in a 'Dirty Shutdown' state. I copied back the log files (excepting res1.log and res2.log). In addition to E00.log there was also tmpE00.log and 30 odd other files.
I deleted E00.chk
I mounted each of the stores (staff, student and public).
Looking in the Application event log I could see that ESE had replayed the relevant log files successfully and the stores all mounted ok.
I then immediately dismounted each stores.
I ran eseutil /mh again and they all showed 'Clean Shutdown' <broad smile>
I ran an offline backup of the database files, logfiles and e00.chk file
I mounted the stores and happily all the mailboxes were already attached to the respective AD accounts.
<another broad smile>
I count ourselves as very fortunate to have been able to restore everything without any data loss.

For anyone who happens to read this question & comments in the future I think it might have been possible to save all this hassle and stress with the exchange server by first removing it from the domain, then doing an account reset on the machine acct in AD; and finally re-joining the machine to the domain. It would certainly have been worth a try 2 days ago ...
0
 

Author Closing Comment

by:ComputerFundi
ID: 35745578
Thank you both for your helpful suggestions. CWCertus1 - 200 for the Ad object restore soluton which would have been great if we'd had a backup of the system state!
rpartington - 300 for the clear options around the exchange recovery options.
OK?
0
 
LVL 9

Expert Comment

by:rpartington
ID: 35745667
Thanks
As long as your up and running.
Killer when you have the world on your shoulders ie an exchange down with 1500 users all wanting it backup yesterday is no fun, we have all being there, which is why I now have grey hair taking over.
0
 

Author Comment

by:ComputerFundi
ID: 35745679
you and me both ...
0

Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If something goes wrong with Exchange, your IT resources are in trouble.All Exchange server migration processes are not designed to be identical and though migrating email from on-premises Exchange mailbox to Cloud’s Office 365 is relatively simple…
Stellar Exchange Toolkit: this 5 in 1 toolkit comes loaded with mega-software tool. Here’s an introduction to tools’ usage and advantages:
In this video we show how to create an Address List in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Organization >> Ad…
In this video we show how to create an Accepted Domain in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Ac…
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question