How To Recover Failed Mailbox Sever In Exchange 2010 DAG Environment

Admins might be handling Exchange 2010 database availability group (DAG) Environment every day and if Exchange is configured with proper hardware and software, there is nothing much left for admins to do. However there are few tasks where real expertise required. One of them is recovering a failed server in a DAG. Several of you might have performed this in your lab, but when it comes to recovering server in actual production environment, we all know how much we struggle. So, I am writing this article to provide you real time steps.

Scenario 1. The Exchange server is up, but Exchange services are not working. This could happen due to operating system patching, a third party tool or even a virus. You have decided to reinstall Exchange with a fresh operating system. Here I am using the name Server2 that is part of a DAG named as NADAG1.

Scenario 2. The Exchange server is crashed due to hardware failure.

Step 1. Documenting  server configuration.
> First document all OS/NIC/Drives related configurations. Take screenshots, especially of the NIC configuration so you can compare later.
> Launch the Exchange Management Consolve (EMC) from another server and capture the database screenshot with your activation preference.

Step 2. Clean up failed server/db from AD,Exchange and Cluster configuration.
> Remove database copies. Launch EMS and run following command:

Remove-MailboxDatabaseCopy "DB1\Server2"

Select all Open in new window

Note: You will get a warning, but it will disappear. If you have more databases you will need to run the command for each of them. Best copy paste in a text file, save it as remove.ps1 and run it from EMS from another DAG server. Like this:

Remove-MailboxDatabaseCopy "DB1\Server2"
                      Remove-MailboxDatabaseCopy "DB2\Server2"
                      Remove-MailboxDatabaseCopy "DB3\Server2"

Select all Open in new window

> After the databases are cleaned up, it is time to remove Server2 configuration from the DAG. For this run below command from EMS:

Remove-DatabaseAvailabilityGroupServer -Identity NADAG1 -MailboxServer Server2 -ConfigurationOnly

Select all Open in new window

If you don't know what the DAG name is, run

​Get-DatabaseAvailabilityGroup

Select all Open in new window

or check using  the EMC.

> Now remove or evict Server2 from the Cluster. Launch Failover Cluster Management from another DAG server, expand Nodes>Right Click the Failed Server>Select Evict the server.

Step 3. Install the server

> Launch Active Directory snap-in. Find the server>right click>select reset. This is a very important step.
> Now Start installing the OS again. Make sure to use same operating system with same service pack and patches. For DAG you need Enterprise Edition Windows OS. Use the same Disk/NIC configuration, with the same computer name and join it to the domain. If you miss any step, your recovery will fail.

Step 4. Let's start Exchange recovery process.

> What do you need? First, the same service pack. If you have SP3 on the failed server, get SP3 and extract it.
> Now download the Rollup that was present on failed server. Never install the higher version of a service pack or Rollup.
> Verify your NIC configuration. Especially, on the Replication NIC, make sure:
> Now download and install Office 2010 Filter Packs 64-bit.
> Next install the Exchange 2010 Pre-requisites. Launch Powershell in Admin mode and run below commands.
 

Import-Module ServerManager
                      
                      Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server -Restart

Select all Open in new window

> Once Server is up. Now run below three commands:

sc config NetTcpPortSharing start= auto
                      sc config spooler start= disabled
                      netsh advfirewall set allprofiles state off

Select all Open in new window

> Finally, we need to run recoverserver command and install Exchange again on failed server. For this, launch cmd in admin mode and run below command:

setup /m:recoverserver

Select all Open in new window

Note: Make sure to browse to setup path folder. Then run the above command.

> Once Server recover is completed. Install the Rollup. This was present earlier. Don't install higher or lower version.
> Restart the Server Now. Check the Exchange services, logs, EMC and EMS. All should be fine now.

Step 5. Adding Server and DB back to recover Server in DAG.
> Launch EMS and run below command to add server back to DAG.

Add-DatabaseAvailabilityGroupServer -Identity NADAG1 -MailboxServer Server2

Select all Open in new window

> Add Database copies. Remember, I told you above to note down activation preference. You need that here:

Add-MailboxDatabaseCopy -Identity DB1 -MailboxServer Server2 -ActivationPreference 1
                      Add-MailboxDatabaseCopy -Identity DB2 -MailboxServer Server2 -ActivationPreference 2

Select all Open in new window

> Update the database copies by running below command and wait for seeding to complete. You can use EMC also.

Update-MailboxDatabaseCopy -Identity "DB1\Server2" -DeleteExistingFiles
                      Update-MailboxDatabaseCopy -Identity "DB2\Server2" -DeleteExistingFiles

Select all Open in new window

Now your failed server is up and back into production. Make a back-up, if needed. I hope this will help lot of Exchange admins. If I missed anything let me know.