Link to home
Start Free TrialLog in
Avatar of RyanIrish
RyanIrish

asked on

Help restoring exchange database following full VM restore

Hello,


Following a ransomware attack, I was forced to restore our virtual exchange server.  The backup was clean and there are no signs of the attack on the exchange server at this time.  Following the restore, all of our emails were stuck in the queue with the error message shown below.  The database was showing as mounted when I left for the night.


User generated image


This morning, I checked and the database was no longer mounted.  I attempted to mount, and this error was thrown.


Failed to mount database "Mailbox Database 1354787502". Error: An Active Manager operation failed with a transient error. Please retry the operation. Error: Database action failed with transient error. Error: A transient error occurred during a database operation. Error: MapiExceptionNetworkError: Unable to mount database. (hr=0x80040115, ec=-2147221227) Diagnostic context:    Lid: 65256      Lid: 12514   Win32Error: 0x6BE    Lid: 62184      Lid: 16280   dwParam: 0x0 Msg: EEInfo: ComputerName: n/a    Lid: 8600    dwParam: 0x0 Msg: EEInfo: ProcessID: 15660    Lid: 12696   dwParam: 0x0 Msg: EEInfo: Generation Time: 0423-04-27T13:41:38.8320000Z    Lid: 10648   dwParam: 0x0 Msg: EEInfo: Generating component: 2    Lid: 14744   dwParam: 0x0 Msg: EEInfo: Status: 1726    Lid: 9624    dwParam: 0x0 Msg: EEInfo: Detection location: 974    Lid: 13720   dwParam: 0x0 Msg: EEInfo: Flags: 0    Lid: 11672   dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 0    Lid: 49064   dwParam: 0x1    Lid: 12514   Win32Error: 0x6BE    Lid: 62184      Lid: 16280   dwParam: 0x0 Msg: EEInfo: ComputerName: n/a    Lid: 8600    dwParam: 0x0 Msg: EEInfo: ProcessID: 15660    Lid: 12696   dwParam: 0x0 Msg: EEInfo: Generation Time: 0423-04-27T13:42:01.4000000Z    Lid: 10648   dwParam: 0x0 Msg: EEInfo: Generating component: 2    Lid: 14744   dwParam: 0x0 Msg: EEInfo: Status: 1726    Lid: 9624    dwParam: 0x0 Msg: EEInfo: Detection location: 974    Lid: 13720   dwParam: 0x0 Msg: EEInfo: Flags: 0    Lid: 11672   dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 0    Lid: 1047    StoreEc: 0x80040115 [Database: Mailbox Database 1354787502, Server: TG3Mail.tg3.local]


Any ideas?


Edit:  Thank you for reading...sorry, I'm still in panic mode and falling short on courtesy.  

Avatar of Rodney Barnhardt
Rodney Barnhardt
Flag of United States of America image

Is it possible the server performed some updates and rebooted overnight causing a dirty shutdown? You can verify it with this command


eseutil /mh "folder_path\ExchangeDatabase.edb”

Open in new window

If the results show a dirty shutdown, run this to bring it to a clean state:

eseutil /p "folder_path\ExchangeDatabase.edb”

Open in new window

If it then shows clean, try to mount the database. 

Avatar of RyanIrish
RyanIrish

ASKER

Hi Rodney, thank you for the prompt response, I really appreciate it.  


Here is the result of the first command:


Microsoft Windows [Version 10.0.17763.4252]
(c) 2018 Microsoft Corporation. All rights reserved.


C:\Windows\System32>eseutil /mh "folder_path\ExchangeDatabase.edb"


Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.02
Copyright (C) Microsoft Corporation. All Rights Reserved.


Initiating FILE DUMP mode...
Error: Access to source database 'folder_path\ExchangeDatabase.edb' failed with Jet error -1811.


Operation terminated with error -1811 (JET_errFileNotFound, File not found) after 0.15 seconds.

Open in new window


It appears you just pasted in what I sent. You should have modified the "folder path" to the path on your server where where the database is located and ensure the name of the Exchange database matches the name of your Exchange database. That was just an example I provided. 

Got it...you can see my exchange knowledge on full display here :/

The database was showing as mounted when I left for the night.


if the database was mounted before and now isn't, i highly doubt an update would prevent the database from mounting

check uptime; if it didn't restart last night then that isn't the issue


my first thought is to check disk space; if it gets low then databases will automatically dismount

Hi Seth, thanks.  The drive has 500+ gigs of space, so I don't think that is the issue.


I still don't know what the heck I'm doing with the eseutil unfortunately...I'm a lost cause over here.


User generated image


path has spaces; need to wrap it in quotes

You need to put the path in quotes like in the example. Otherwise, when it hits a space it will drop "c:\Program Files\Microsoft\Exchange Server\v15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb"

Looks like you nailed it Rodney, I see a dirty shutdown in there.


C:\Users\tg3admin>eseutil /mh "c:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb"


Extensible Storage Engine Utilities for Microsoft(R) Exchange Server

Version 15.02

Copyright (C) Microsoft Corporation. All Rights Reserved.


Initiating FILE DUMP mode...

         Database: c:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb



DATABASE HEADER:

Checksum Information:

Expected Checksum: 0x99ffa9b1

  Actual Checksum: 0x99ffa9b1


Fields:

        File Type: Database

         Checksum: 0x99ffa9b1

   Format ulMagic: 0x89abcdef

   Engine ulMagic: 0x89abcdef

 Format ulVersion: 0x620,20,0  (attached by 0)

 Engine ulVersion: 0x620,60,140  (efvCurrent = 9060)

Created ulVersion: 0x620,20

     DB Signature: Create time:05/12/2020 15:42:40.944 Rand:2069087980 Computer:

         cbDbPage: 32768

           dbtime: 1226453687 (0x491a32b7)

            State: Dirty Shutdown

     Log Required: 1979220-1979326 (0x1e3354-0x1e33be)

    Log Committed: 0-1979327 (0x0-0x1e33bf)

   Log Recovering: 0 (0x0)

   Log Consistent: 1979220 (0x1e3354)

  GenMax Creation: 04/27/2023 01:35:27.162 UTC

         Shadowed: Yes

       Last Objid: 36408

     Scrub Dbtime: 0 (0x0)

       Scrub Date: 00/00/1900 00:00:00.000 LOC

     Repair Count: 0

      Repair Date: 00/00/1900 00:00:00.000 LOC

 Old Repair Count: 0

  Last Consistent: (0x1151D3,53,979)  03/09/2022 16:32:40.913 UTC

      Last Attach: (0x1151D4,2,268)  03/09/2022 17:03:25.440 UTC

      Last Detach: (0x0,0,0)  00/00/1900 00:00:00.000 LOC

    Last ReAttach: (0x1E33BF,2,268)  04/27/2023 14:54:18.682 UTC

             Dbid: 1

    Log Signature: Create time:05/12/2020 15:42:40.662 Rand:1590027183 Computer:

       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)


Previous Full Backup:

        Log Gen: 1976849-1976951 (0x1e2a11-0x1e2a77) - OSSnapshot

           Mark: (0x1E2A78,1,0)

           Mark: 04/25/2023 04:03:48.534 UTC


Previous Incremental Backup:

        Log Gen: 0-0 (0x0-0x0)

           Mark: (0x0,0,0)

           Mark: 00/00/1900 00:00:00.000 LOC


Previous Copy Backup:

        Log Gen: 597685-597805 (0x91eb5-0x91f2d) - OSSnapshot

           Mark: (0x91F2E,1,0)

           Mark: 05/29/2021 22:27:44.801 UTC


Previous Differential Backup:

        Log Gen: 0-0 (0x0-0x0)

           Mark: (0x0,0,0)

           Mark: 00/00/1900 00:00:00.000 LOC


Current Full Backup:

        Log Gen: 0-0 (0x0-0x0)

           Mark: (0x0,0,0)

           Mark: 00/00/1900 00:00:00.000 LOC


Current Shadow copy backup:

        Log Gen: 0-0 (0x0-0x0)

           Mark: (0x0,0,0)

           Mark: 00/00/1900 00:00:00.000 LOC


     cpgUpgrade55Format: 0

    cpgUpgradeFreePages: 0

cpgUpgradeSpaceMapPages: 0


       ECC Fix Success Count: none

   Old ECC Fix Success Count: none

         ECC Fix Error Count: none

     Old ECC Fix Error Count: none

    Bad Checksum Error Count: found (1590)

Last Bad Checksum Error Date: 04/27/2023 14:54:16.525 UTC

Old bad Checksum Error Count: none


  Last Database Maintenance Finish Date: 00/00/1900 00:00:00.000 LOC

Current Database Maintenance Start Date: 05/01/2022 03:24:28.534 UTC

      Highest Continuous Database Maintenance Page: 0

      Highest Database Maintenance Page: 0


  Database Header Flush Signature: Create time:04/27/2023 14:54:18.682 Rand:4214609386 Computer:

 Flush Map Header Flush Signature: Create time:00/00/1900 00:00:00.000 Rand:0 Computer:



Operation completed successfully in 0.110 seconds.


Run the second command, changing the path, to bring it to a clean state. 

My guess was that if you recovered the server from a snapshot, it may have went out last night and performed updates that it felt was needed and did a reboot of the server. 

Rgr that, it's running now.  


I'm amazed that you called it, some of you folks are just too good.  

Your welcome. Glad you got it running. I would check and make sure you updates are not set to install and reboot automatically if that is what the Event viewer shows happened. It could have rebooted for another reason though, so you need to check the logs. 

Well, the repair is running now, not sure I'm out of the woods yet, but hopefully it will mount after this finishes.


Thank you for your patience though, I know all too well what it's like to help someone like me...

Oh. I thought you meant the server was running LOL

It's still doing it's thing...


Do I need to worry about the section outlined in red?


User generated image


Crap...


User generated image


That is a problem. You can try a defrag. There is a previously resolved issue with that error. Just look at the solution and change your paths to match the defrag command. 


https://www.experts-exchange.com/dashboard/#/questions/28984953?

Seems like it needed to finish that previous recovery in order to defrag.

User generated image


Just to ask, since it mentions the logs. When you did the recovery, were the logs also recovered?

I believe so...the logs are all of these text docs in the mailbox db folder, correct?

User generated image


Unless someone else has a better idea, it seems the database has been corrupted. You could attempt another restore and use the commands to check the database. Since you did not have the server fully running yesterday, It may be best to attempt another restore and verify the database is clean before mounting it. If that does not work, you could either open and pay for a MS engineer who could assist or there is a company that makes a Exchange database recovery tool to fix corrupted databases https://www.systoolsgroup.com/exchange-bkf-recovery.html

Thanks, Rodney.  I'm going to try to restore the mailbox folder again and see if I can't get to a clean state.


Otherwise I'll  contact M$ and see if they can help.

Though I would check in and see if you were able to get this recovered. 

Thanks, for checking in.  Unfortunately, not yet...I have some help from a local consultant and he's running into some of the same errors I saw, but he's still plugging away.

I'll update with any news.  I appreciate you, Rodney!

So the solution was listed here Email Stuck in Exchange On-premises Transport Queues - Microsoft Community Hub 


I'm still seeing some errors I don't like, so I'm in the process of migrating all of the mailboxes to a new database to see if that does anything for me.


Information Store - Mailbox Database 1354787502 (12808,R,98,15.02.0986.015) Mailbox Database 1354787502: The database page read from the file "C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb" at offset 286329274368 (0x00000042aa8e0000) (database page 8738075 (0x85551B)) for 32768 (0x00008000) bytes failed verification due to a page checksum mismatch.  The stored checksum was [0000000000000000:0000000000000000:0000000000000000:0000000000000000] and the computed checksum was [0085551b1b755346:0000000000000000:0000000000000000:0000000000000000].  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

Open in new window


ASKER CERTIFIED SOLUTION
Avatar of Rodney Barnhardt
Rodney Barnhardt
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial

Yes, we started with a fresh restore, then applied the solution above.  This is why it was always starting out as mounted, giving me false hope, and then dismounting when this check failed.


My database migration is almost complete, so I'll know more about resolving that error soon.


I do have a user with a mailbox that is now quarantined, but not the user who was the source of the ransomware.  I'm trying to figure out why and how to fix, otherwise I think I'm almost back to normal.