Exchange 2016 Recovery Issue

We have an Exchange 2016 server running on Server 2012. Last night our Exchange server stopped running and we attempted to run a Setup.exe /mode:recoverserver and originally we were receiving a buildtobuildupgrade error that would not allow us to proceed. After removing the BuildtoBuild entries in regedit we are now getting a new error when we try to run the installer. This is the error. I tried to find information on it but no one has seen this error before.

Error:
The following error was generated when "$error.Clear();
      buildToBuildUpgrade-ExsetdataAtom -AtomName MDB -DomainController $RoleDomainController

" was run: "Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685277' and message 'The service did not respond to the start or control request in a timely fashion.'.
   at Microsoft.Exchange.Configuration.Tasks.Task.ThrowTerminatingError(Exception exception, ErrorCategory category, Object target)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.HandleExsetdataReturnCode(UInt32 scErr)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.BuildToBuildUpgradeAtom(AtomID atomID)
   at Microsoft.Exchange.Management.Deployment.BuildToBuildUpgradeExsetdataAtom.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
   at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
LVL 1
doraemonzAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AmitIT ArchitectCommented:
Did you reset computer object in AD, before starting the recovery process.
doraemonzAuthor Commented:
No. We did not.
AmitIT ArchitectCommented:
You need to reset the Computer account first and then start the recovery process. Check this KB:
https://docs.microsoft.com/en-us/exchange/high-availability/disaster-recovery/recover-exchange-servers?view=exchserver-2019

or check my KB: https://www.experts-exchange.com/articles/18110/How-To-Recover-Failed-Mailbox-Sever-In-Exchange-2010-DAG-Environment.html?headerLink=workspace_article

Though i wrote this to Exchange 2010, however steps are almost same to any Exchange version. You can skip DAG steps.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

doraemonzAuthor Commented:
Trying it now.
doraemonzAuthor Commented:
I received the same error.

    Mailbox role: Mailbox service                             FAILED

The following error was generated when "$error.Clear();

buildToBuildUpgrade-ExsetdataAtom -AtomName MDB -DomainController
$RoleDomainController

" was run:
"Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred
with error code '3221685277' and message 'The service did not respond to the
start or control request in a timely fashion.'.
 at
Microsoft.Exchange.Configuration.Tasks.Task.ThrowTerminatingError(Exception
exception, ErrorCategory category, Object target)
 at
Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.HandleExsetdataRet
rnCode(UInt32
scErr)
 at
Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.BuildToBuildUpgrad
Atom(AtomID
atomID)
 at
Microsoft.Exchange.Management.Deployment.BuildToBuildUpgradeExsetdataAtom.Inter
alProcessRecord()

 at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
 at
Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String
funcName, Action func, Boolean terminatePipelineIfFailed)".


The Exchange Server setup operation didn't complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
doraemonzAuthor Commented:
The server is still in production. Does recovery only work on servers that have been removed from production?
AmitIT ArchitectCommented:
Can you please answer below questions:

What do you mean by production?
Are you able to access Exchange and all Exchange related services are running.
What was the reason for starting recovery process.
doraemonzAuthor Commented:
After the initial update to CU10 all services were loading except for Microsoft Exchange Information Store, Microsoft Exchange Diagnostics, Microsoft Exchange DAG Management, and Microsoft Exchange Mailbox Assistants. When we tried to rerun the CU10 update we received an error saying the Submission Service could not be restarted. We looked and saw that was because the Microsoft Exchange Mailbox Transport Submission service was missing after the update as in gone from the service list. We received a recommendation to run the recovery to restore the missing service. Now the service is on the list and is running but the other original services still will not start and if we try to run the CU update we get the error that I submitted above.
AmitIT ArchitectCommented:
I advise you to open case with Microsoft. As this might need in-depth analysis.
doraemonzAuthor Commented:
Thank you for your suggestions.
AmitIT ArchitectCommented:
Keep me posted, how things goes with MS. If nothing works, extreme step, would be to re-install OS and then install Exchange server with recovery switch and restoring the db from backup. However, i want to avoid this last option.
doraemonzAuthor Commented:
We have a VM we are trying to restore as a last resort. Will the installation changes that have been made effect that machine with AD?
AmitIT ArchitectCommented:
Don't restore VM Snapshot, as that would cause more issues. Refer:
https://thoughtsofanidlemind.com/2011/03/18/be-careful-with-vm-snapshots-of-exchange-2010-servers/

Also, note, if your schema already extended, then you cannot revert back.
doraemonzAuthor Commented:
That was why I was asking. Thank you again.
doraemonzAuthor Commented:
Issue was multifold. Bad update wiped out most of the DLLs that were on the server leaving several services unable to work. The Exchange database would not mount due to missing software and grabs from the antivirus software. Resolved by manually copying the DLLS to the correct folders and the Bin location where items were missing by comparing to an Exchange that was up and working. Made sure the database was not in an unusable state and mounted it once the services started consistently running again. Removed antivirus software temporarily. Reran the installer for CU10 in upgrade mode. This time all items passed. We're not going to CU11 after this debacle. We'll ultimately switch to Office 365.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
doraemonzAuthor Commented:
Thank you very much for your assistance with this. Even after all of this we still had issues connecting due to missing files so we were forced to update to CU11 afterward and even with direct support from Microsoft it took 6 attempts to get the CU update to install  and completely fix the server.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.