Admins might be handling Exchange 2010 database availability group (DAG) Environment every day and if Exchange is configured with proper hardware and software, there is nothing much left for admins to do. However there are few tasks where real expertise required. One of them is recovering a failed server in a DAG. Several of you might have performed this in your lab, but when it comes to recovering server in actual production environment, we all know how much we struggle. So, I am writing this article to provide you real time steps.
Scenario 1. The Exchange server is up, but Exchange services are not working. This could happen due to operating system patching, a third party tool or even a virus. You have decided to reinstall Exchange with a fresh operating system. Here I am using the name Server2 that is part of a DAG named as NADAG1.
Scenario 2. The Exchange server is crashed due to hardware failure.
Step 1. Documenting server configuration.
> First document all OS/NIC/Drives related configurations. Take screenshots, especially of the NIC configuration so you can compare later.
> Launch the Exchange Management Consolve (EMC) from another server and capture the database screenshot with your activation preference.
Step 2. Clean up failed server/db from AD,Exchange and Cluster configuration.
> Remove database copies. Launch EMS and run following command:
Note: You will get a warning, but it will disappear. If you have more databases you will need to run the command for each of them. Best copy paste in a text file, save it as remove.ps1 and run it from EMS from another DAG server. Like this:
> After the databases are cleaned up, it is time to remove Server2 configuration from the DAG. For this run below command from EMS:
Remove-DatabaseAvailabilityGroupServer -Identity NADAG1 -MailboxServer Server2 -ConfigurationOnly
If you don't know what the DAG name is, run
or check using the EMC.
> Now remove or evict Server2 from the Cluster. Launch Failover Cluster Management from another DAG server, expand Nodes>Right Click the Failed Server>Select Evict the server.
Step 3. Install the server
> Launch Active Directory snap-in. Find the server>right click>select reset. This is a very important step.
> Now Start installing the OS again. Make sure to use same operating system with same service pack and patches. For DAG you need Enterprise Edition Windows OS. Use the same Disk/NIC configuration, with the same computer name and join it to the domain. If you miss any step, your recovery will fail.
Step 4. Let's start Exchange recovery process.
> What do you need? First, the same service pack. If you have SP3 on the failed server, get SP3 and extract it.
> Now download the Rollup that was present on failed server. Never install the higher version of a service pack or Rollup.
> Verify your NIC configuration. Especially, on the Replication NIC, make sure:
Client for Microsoft Networks-Disabled
File and Printer Sharing for Microsoft Networks-Disabled
Uncheck Registration this connection address in DNS check box. For Replication NIC Only.
> Now download and install Office 2010 Filter Packs 64-bit.
> Next install the Exchange 2010 Pre-requisites. Launch Powershell in Admin mode and run below commands.
Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server -Restart
> Once Server is up. Now run below three commands:
sc config NetTcpPortSharing start= auto
sc config spooler start= disabled
netsh advfirewall set allprofiles state off
> Finally, we need to run recoverserver command and install Exchange again on failed server. For this, launch cmd in admin mode and run below command:
Note: Make sure to browse to setup path folder. Then run the above command.
> Once Server recover is completed. Install the Rollup. This was present earlier. Don't install higher or lower version.
> Restart the Server Now. Check the Exchange services, logs, EMC and EMS. All should be fine now.
Step 5. Adding Server and DB back to recover Server in DAG.
> Launch EMS and run below command to add server back to DAG.
Add-DatabaseAvailabilityGroupServer -Identity NADAG1 -MailboxServer Server2
> Add Database copies. Remember, I told you above to note down activation preference. You need that here:
Add-MailboxDatabaseCopy -Identity DB1 -MailboxServer Server2 -ActivationPreference 1
Add-MailboxDatabaseCopy -Identity DB2 -MailboxServer Server2 -ActivationPreference 2
> Update the database copies by running below command and wait for seeding to complete. You can use EMC also.
Update-MailboxDatabaseCopy -Identity "DB1\Server2" -DeleteExistingFiles
Update-MailboxDatabaseCopy -Identity "DB2\Server2" -DeleteExistingFiles
Now your failed server is up and back into production. Make a back-up, if needed. I hope this will help lot of Exchange admins. If I missed anything let me know.