DCPromo Exchange 2003 - Catch 22 - Help (Production Server Down)

Hi

Got a bit of a crisis on my hands here.

I have a legacy Dell server running Server 2003 (Domain Controller) / Exchange 2003 - It used to be a branch office server and was integrated into our main office when it shut.

Came in this morning - Server dead. RAID 1 had dropped off line and server crashed. Via Dell Support, got it back up and running.

However......

RAID 1 was disks 0 and 1 on the server.

We removed Disk 1, booted Disk 0 and the server came up - with windows warning us that the C:\Windows\Debug folder was corrupt. On the suspicion that other folders may be corrupt, we powered down, removed disk 0 and powered up disk 1. No errors. Great. So we'll base the rebuild on this one.

However - users calling me to say they cannot access file shares or exchange. On close investigation AD has basically told me it has blocked incoming and outgoing access due to the USN number being lower than previously had (error 2095:)

During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers.
 
 Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC.
 
 If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations.
 
 To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support.
 
 The most probable cause of this situation is the improper restore of Active Directory on the local domain controller.
 
 User Actions:
 If this situation occurred because of an improper or unintended restore, forcibly demote the DC.
 
Remote DC:
ee94897e-59e6-4915-ad01-1eed3752d40e
Partition:
DC=KeppieDesign,DC=co,DC=uk
USN reported by Remote DC:
16515155
USN reported by Local DC:
16515139


I can see what has happened. When we booted off Disk 0 it obviously sync'd with the rest of my AD controllers. When we rebooted off Disk 1, that was in the state Disk 0 was before we booted it tried to sync but it has already been sync'd - hence the error above.

Dell Tech never appreciated the implications of this when he was telling me to swap disks in and out, and to be honest neither did i until now.

Now my problem.

1. Microsoft say DCPromo it to demote it to a member server or
2. Or Restore System State.

However.

1. I cannot dcpromo as i have Exchange 2003 on it.
2. System State not included in the nightly backups. DOH.

So i thought i could move the Exchange stuff off it, un-install exchange, dcpromo, re-install exchange, move Exchange mailboxes back.

Can't - Information Store won't start because of the above.

Catch 22. Help!

What i am thinking - If I un-install exchange 2003 as it is, will it leave my database and transacation logs alone? Then i can dcpromo, then reinstall exchange? Would the re-install hook straight back into my databases and logs if i point the store to them?

I'm nervous about doing this on a production server without doing it in a lab - so looking for some assistance.

Can i just dcpromo the server with exchange on it, and re-install exchange? I read somewhere dcpromoing and exchange 2003 server can cause problems to the exchange organisation - something i don't want to do as i have 6 sites.

Help?!

Regards
Colin

colin-andersonAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

colin-andersonAuthor Commented:
I should also add, i already have a DC at this site and this server was simply a DC beacuse i could DCPromo it with exchange on it.

No Roles on it - and no network services like DNS, WINS, DHCP etc - this is all handed by the main DC.
0
chakkoCommented:
Please take a look at this article.

There is a section on how to recover the USN rollback situation.  That's what it seems you have.

http://support.microsoft.com/kb/875495

I would not uninstall exchange.  My first thought that comes to mind would be disable all exchange services.  Restore System State (Active Directory) and the restart the server and if it looks good start the exchange services (if they don't already auto-start due to the restore).  I would probably do a non-authoritative restore from a previous backup of the system state.

I would also make a full backup too before doing anything.

0
colin-andersonAuthor Commented:
Backup is okay - last ran last night.

I do not have a System State Backup - that is my problem :-(
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

chakkoCommented:
hmmm...  I think 2 options.  I am 'thinking out loud' at the moment.

1.  adventurous option -  connect the other disk and see if any AD errors.  If none then disconnect the LAN cable, reboot in DSRM modem and copy the ntds.dit (or all of the AD files) to a USB drive. swap out the disks.  Boot into DSRM modem and On the better/working disk, I would copy/backup the ntds.dit file to another folder for easy access if you need it, and then use the 'good' copy of the file from the USB drive.
Then boot the server and see if the AD file from the other disk is 'up-to-date', you need to reconnect the LAN again and check it.
If not any better then you can reboot to directory services restore mode and put the ntds.dit file back from the original copy.  Then try DCPROMO option.  Or maybe another Expert has another option.

2.  Try DCPROMO.  You say don't have any roles on that server, there is another Global Catalog server available already, right?   From looking at the internet, DCPROMO could cause problems with OWA

0
d3ath5tarCommented:
making exchange a DC is always a bad move. It's only really supported on SBS.

Personally i would not be trying to save the existing server. If anything goes wrong with the AD on it you could end up causing more issues for the other DC.....

Get an AD backup of your other DC.

Copy off the stores as a backup and disconnect the server frm the netowrk.
Run a dcpromo with the force switch on the exchange server to remove AD from it and drop it off the domain back to workgroup. If it won't demote properly just switch it off.

If it demotes properly uninstall Exchange from the server

If it doesn;t demote properly do a full rebuil dback to the same service pack position.

Run a metadata cleanup in AD to remove the old DC references to this server.
Join the machine back to the domain with its original name.
Reinstall Exchange with the disaster recovery switch (use the same paths as previous etc) - this shoudl allow the server to take up its previous Exchange position.

If a rebuild you will need to copy the stores back to the origin paths.

Do not make this a dc after reinstall, get another separate DC.

A/.
0
colin-andersonAuthor Commented:
Agreed - I saw past the idea of saving the server.

Copied the edb and stm files of the Exchange database onto another existing Exchange Server. Created a new Storage Group, then a new Store - pointing to these databases. Mounted them fine - hooray. All the mailboxes were disconnected.

Then for each user with a mailbox on that server, i used my working DC to remove exchange attributes then via system manager reconnected the mailboxes to the correct AD account.

Seems to have worked. Webmail worked but couldn't get Outlook to connect - but eventually realised that if mailboxes are moved between storage groups, you need to redo the Mail Profile from scratch. Done two machines tonight - all seems ok. Do rest tomorrow morning.

Currently trying to work out how to move my file share - which is a mixture of live files and files archived by Enterprise Vault onto a different file server.

Once i get all files off - what is best practice to decomission this server? It's 7 years old and caused me enough grief.

Do i un-install Exchange then DC Promo it back to a member server before removing it from the domain OR do i DC Promo it, then un-install exchange, then remove from domain?

If it fails i might need help re: metadata clean up - cross that bridge when we come to it.

I agree Exchange on a DC is bad. This server used to be a single server based in a branch office with limited budget and managed by a 3rd party IT company. Branch office shut, it got integrated with our main office and i've not touched it because i knew it would be a pita dealing with. One of those jobs i should have got round too.

Thanks for help so far guys -

C>

0
d3ath5tarCommented:
Uninstall exchange then try and demote.
If this was the first exchange server in the organization you will need to perform some additional functions first like moving the oab. There is an ms kb on the subject which gives full instruction.

As far as demote goes, if it demotes, yay! If it won't, switch off and do meta cleanup.

As an admin user:
Command box
Ntdsutil
Metadata cleanup
Connections
Connect to server YourDc
Quit
Select Operation target
List domains
Select domain 0 (assuming only one domain)
List sites
Select site x (where x is your AD site)
List servers in site
Select server x (where x is dead dc)
Quit
Remove selected server (accept prompt message)
Quit
Quit
Quit

:)
0
colin-andersonAuthor Commented:
Awesome advice. Can't thank you enough.

Luckily it wasn't my first Exchange server.

Probably be tomorrow before I try that. Currently restoring the backup of the goosed server onto a good server. Can't do a file copy as Enterprise Vault insists on restoring the files before copying and I have 200GB data sitting on a 120GB array :-(

Main thing is my exchange move onto another exchange server seems to have worked. Will find out for sure in the morning.

One quick Q - I have 78 emails queued on the knackered exchange system. Is there anyway to redirect these to my good exchange server or are they past recovering?

If so I suppose I could just delete with NDR and hope the sender re sends them.

Thanks again - keep you posted and reward when I can put this to bed.

C.
0
d3ath5tarCommented:
Honestly, good question. Never tried! Its bed time here in the uk, will try it on my testlab in the morning.

A/.
0
colin-andersonAuthor Commented:
Know the feeling - I'm in the UK too (Glasgow) and been at this since 8.30am. Brain is frazzled but want to get my file share on-line and remapped for the users before hitting bed. Restore still has 2 hours to go... Sigh.

Suppose I should document everything done so far meantime before I forget.

Thanks again
C

0
chakkoCommented:
sounds like a long night.  Hope you get everything 100% soon.



0
chakkoCommented:
In the Future, or if anyone reads this later.  Another idea came to mind.

Boot from the 1st disk and chkdsk it, if the chkdsk result is not bad, then try to backup the system state on that, then use the System state backup for a restore to the 2nd disk version.
0
colin-andersonAuthor Commented:
Hi folks. Almost there.  I have now successfully migrated all the Exchange data across to a new server (flowing freely and working 100%), restored all the files from backup and been working on Enterprise Vault so users can restore archived files.

I now need to remove the server from my active directory manually. Bearing in mind this is a Domain Controller with Exchange. I cannot un-install Exchange because it thinks mailboxes are still on it. I cannot start Exchange because it’s busted - If you remember, the original problem was the DC dropped out because we swapped and booted from different RAID 1 discs on the DC, knocking it off. DC then blocked all incoming connections and Exchange would not start, so  I copied the Exchange databases to a different Exchange server and updated the AD for the users concerned so they pointed to the new server and re-done their mail profiles. Wprked great.

The server is now powered off. I do not intend to power it back on in its current state on our network. All services have been relocated to other servers and working fine but obviously Active Directory and Exchange still think the server is there as it has not been properly removed. I still occasionally get the odd client that tries to bind to this DC on logon – so need it removed.

I *think* i have worked out what I need to do, but need a second opinion and fresh pair of eyes on my plan before proceeding.

1. Remove Dead Server from Exchange System Manager – Delete the Server Object from Administrative Group

2. Remove any Recipient Update Services from Exchange Organisation

(that should take care of My Exchange Organisation seeing this server?)

3. Check to ensure this DC does not hold any AD FSMO roles – done, none on it

4. Manually remove server from Active Directory using ntdsutil using my good DC and the details steps provided by d3ath5tar above

5. Manually check DNS and remove any reference to dead DC

6. Ensure DHCP isn’t referring to dead DC

7. Delete Server Object from AD if still present

That should be it?

*fingers crossed*



0
d3ath5tarCommented:
Pretty much, although if you would prefer a cleaner removal rather than picking it all apart, (you could even use a PC as temp measure...)
Assuking the dead DC is not going to be brough back online with the same name....

Do the DC removal first with NTDSUtil.
Then either on the old tin or a temp PC, do a clean build of the old exchange server with the disaster recovery switch using the same server name, then uninstall it.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
colin-andersonAuthor Commented:
OK. Good advice as per usual!

I would need to flatten a PC and install Server 2003 then Server Exchange in DR mode as i don't have a spare box. Bringing up the old tin would be problematic as that itself still thinks it's a DC.

I'm comfortable with Exchange DR mode - used it before and can understand why you suggest it.

Hold on - I could set up a Temp Server with the same name on VMWare, install Exchange in DR mode then un-install Exchange, then remove this 'Temp Server' from domain and delete my VMWare File :-)

Ok, I'll run the NTDSutil just now to eliminate the dead DC, then build a VMWare Image on Server 2003, join domain, install Exchange in DR mode then un-install it....

*fingers crossed*
0
d3ath5tarCommented:
VMWare.... winner every time :)

0
chakkoCommented:
You may need to use adsiedit and look in the Exchange and check for remnants in there.

There are articles about how to manually delete the public information store when trying to demote Exchange 2003 servers.  I think I remember you have to dig around using adsiedit.

Just something to keep in mind in case the issue arises.

Oh yeah, have good full backups of everything before doing that.
0
colin-andersonAuthor Commented:
Ok.....

Ran NTDSutil and manually removed my dead dc. So far so good. OK.

- Setup a Virtual Server with Server 2003 - Patched it to SP2, using same name as dead server
- Reset AD Computer account and joined Domain using same name.
- Installed Exchange in DR mode. It went mental at the end with services not starting due to system permissions and the event log filled up with permissions errors (Presume this may be something to do with the fact that Exchange on this server was previously on a DC and i'm trying to run a DR setup on a member server...?)
- Rebooted Virtual Server
- Installed Exchange SP2 in DR mode - Same outcome as above - but although concerned i wasn't worried i only wanted to do was get Exchange onto it in DR mode to immediately un-install it again.

From my other Exchange Servers i could now browse onto 'my dead server' - although the stores were down because the services wouldn't start.

- Un-Installed Exchange on my temporary Virtual Server...... Rebooted Virtual Server
- Once Rebooted, i browsed about on my Exchange Organisation on my other working Exchange Servers and all references to the dead server were gone. :-)

Did i smile? Oh yes :-)

Removed my temporary server from the domain, shut it down and deleted the virtual server.

I *think* it has worked. I have no references to the dead server both in my AD or my Exchange Organisation any more. I have tested mail flow between servers, both internal and external and all seems well.

Tonight is my server maintenance night so i am in the process of rebooting all the servers across the company for their monthly reboot / patch updates etc.

Let's see how it looks in the morning, but so far so good.

Big THANK YOU for the hand holding - Always good to get a second opinion before proceeding....!

I'll et you know in the morning how it went and we'll get this closed once and for all.

0
d3ath5tarCommented:
HTH :)

Cheers
Ad
0
colin-andersonAuthor Commented:
Everything seems to be okay this morning!

THANK YOU once again.
0
colin-andersonAuthor Commented:
Outstanding Help
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.