Exchange 2016 Recovery Issue

doraemonz
doraemonz used Ask the Experts™
on
We have an Exchange 2016 server running on Server 2012. Last night our Exchange server stopped running and we attempted to run a Setup.exe /mode:recoverserver and originally we were receiving a buildtobuildupgrade error that would not allow us to proceed. After removing the BuildtoBuild entries in regedit we are now getting a new error when we try to run the installer. This is the error. I tried to find information on it but no one has seen this error before.

Error:
The following error was generated when "$error.Clear();
      buildToBuildUpgrade-ExsetdataAtom -AtomName MDB -DomainController $RoleDomainController

" was run: "Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685277' and message 'The service did not respond to the start or control request in a timely fashion.'.
   at Microsoft.Exchange.Configuration.Tasks.Task.ThrowTerminatingError(Exception exception, ErrorCategory category, Object target)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.HandleExsetdataReturnCode(UInt32 scErr)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.BuildToBuildUpgradeAtom(AtomID atomID)
   at Microsoft.Exchange.Management.Deployment.BuildToBuildUpgradeExsetdataAtom.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
   at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
AmitIT Architect
Distinguished Expert 2017

Commented:
Did you reset computer object in AD, before starting the recovery process.

Author

Commented:
No. We did not.
AmitIT Architect
Distinguished Expert 2017

Commented:
You need to reset the Computer account first and then start the recovery process. Check this KB:
https://docs.microsoft.com/en-us/exchange/high-availability/disaster-recovery/recover-exchange-servers?view=exchserver-2019

or check my KB: https://www.experts-exchange.com/articles/18110/How-To-Recover-Failed-Mailbox-Sever-In-Exchange-2010-DAG-Environment.html?headerLink=workspace_article

Though i wrote this to Exchange 2010, however steps are almost same to any Exchange version. You can skip DAG steps.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Trying it now.

Author

Commented:
I received the same error.

    Mailbox role: Mailbox service                             FAILED

The following error was generated when "$error.Clear();

buildToBuildUpgrade-ExsetdataAtom -AtomName MDB -DomainController
$RoleDomainController

" was run:
"Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred
with error code '3221685277' and message 'The service did not respond to the
start or control request in a timely fashion.'.
 at
Microsoft.Exchange.Configuration.Tasks.Task.ThrowTerminatingError(Exception
exception, ErrorCategory category, Object target)
 at
Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.HandleExsetdataRet
rnCode(UInt32
scErr)
 at
Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.BuildToBuildUpgrad
Atom(AtomID
atomID)
 at
Microsoft.Exchange.Management.Deployment.BuildToBuildUpgradeExsetdataAtom.Inter
alProcessRecord()

 at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
 at
Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String
funcName, Action func, Boolean terminatePipelineIfFailed)".


The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

Author

Commented:
The server is still in production. Does recovery only work on servers that have been removed from production?
AmitIT Architect
Distinguished Expert 2017

Commented:
Can you please answer below questions:

What do you mean by production?
Are you able to access Exchange and all Exchange related services are running.
What was the reason for starting recovery process.

Author

Commented:
After the initial update to CU10 all services were loading except for Microsoft Exchange Information Store, Microsoft Exchange Diagnostics, Microsoft Exchange DAG Management, and Microsoft Exchange Mailbox Assistants. When we tried to rerun the CU10 update we received an error saying the Submission Service could not be restarted. We looked and saw that was because the Microsoft Exchange Mailbox Transport Submission service was missing after the update as in gone from the service list. We received a recommendation to run the recovery to restore the missing service. Now the service is on the list and is running but the other original services still will not start and if we try to run the CU update we get the error that I submitted above.
AmitIT Architect
Distinguished Expert 2017

Commented:
I advise you to open case with Microsoft. As this might need in-depth analysis.

Author

Commented:
Thank you for your suggestions.
AmitIT Architect
Distinguished Expert 2017

Commented:
Keep me posted, how things goes with MS. If nothing works, extreme step, would be to re-install OS and then install Exchange server with recovery switch and restoring the db from backup. However, i want to avoid this last option.

Author

Commented:
We have a VM we are trying to restore as a last resort. Will the installation changes that have been made effect that machine with AD?
AmitIT Architect
Distinguished Expert 2017

Commented:
Don't restore VM Snapshot, as that would cause more issues. Refer:
https://thoughtsofanidlemind.com/2011/03/18/be-careful-with-vm-snapshots-of-exchange-2010-servers/

Also, note, if your schema already extended, then you cannot revert back.

Author

Commented:
That was why I was asking. Thank you again.
Issue was multifold. Bad update wiped out most of the DLLs that were on the server leaving several services unable to work. The Exchange database would not mount due to missing software and grabs from the antivirus software. Resolved by manually copying the DLLS to the correct folders and the Bin location where items were missing by comparing to an Exchange that was up and working. Made sure the database was not in an unusable state and mounted it once the services started consistently running again. Removed antivirus software temporarily. Reran the installer for CU10 in upgrade mode. This time all items passed. We're not going to CU11 after this debacle. We'll ultimately switch to Office 365.

Author

Commented:
Thank you very much for your assistance with this. Even after all of this we still had issues connecting due to missing files so we were forced to update to CU11 afterward and even with direct support from Microsoft it took 6 attempts to get the CU update to install  and completely fix the server.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial