asked on
Help restoring exchange database following full VM restore
Hello,
Following a ransomware attack, I was forced to restore our virtual exchange server. The backup was clean and there are no signs of the attack on the exchange server at this time. Following the restore, all of our emails were stuck in the queue with the error message shown below. The database was showing as mounted when I left for the night.
This morning, I checked and the database was no longer mounted. I attempted to mount, and this error was thrown.
|
Any ideas?
Edit: Thank you for reading...sorry, I'm still in panic mode and falling short on courtesy.
ASKER
Hi Rodney, thank you for the prompt response, I really appreciate it.
Here is the result of the first command:
Microsoft Windows [Version 10.0.17763.4252]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\Windows\System32>eseutil /mh "folder_path\ExchangeDatabase.edb"
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Error: Access to source database 'folder_path\ExchangeDatabase.edb' failed with Jet error -1811.
Operation terminated with error -1811 (JET_errFileNotFound, File not found) after 0.15 seconds.
It appears you just pasted in what I sent. You should have modified the "folder path" to the path on your server where where the database is located and ensure the name of the Exchange database matches the name of your Exchange database. That was just an example I provided.
ASKER
Got it...you can see my exchange knowledge on full display here :/
The database was showing as mounted when I left for the night.
if the database was mounted before and now isn't, i highly doubt an update would prevent the database from mounting
check uptime; if it didn't restart last night then that isn't the issue
my first thought is to check disk space; if it gets low then databases will automatically dismount
ASKER
path has spaces; need to wrap it in quotes
You need to put the path in quotes like in the example. Otherwise, when it hits a space it will drop "c:\Program Files\Microsoft\Exchange Server\v15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb"
ASKER
Looks like you nailed it Rodney, I see a dirty shutdown in there.
C:\Users\tg3admin>eseutil /mh "c:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb"
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Database: c:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x99ffa9b1
Actual Checksum: 0x99ffa9b1
Fields:
File Type: Database
Checksum: 0x99ffa9b1
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,20,0 (attached by 0)
Engine ulVersion: 0x620,60,140 (efvCurrent = 9060)
Created ulVersion: 0x620,20
DB Signature: Create time:05/12/2020 15:42:40.944 Rand:2069087980 Computer:
cbDbPage: 32768
dbtime: 1226453687 (0x491a32b7)
State: Dirty Shutdown
Log Required: 1979220-1979326 (0x1e3354-0x1e33be)
Log Committed: 0-1979327 (0x0-0x1e33bf)
Log Recovering: 0 (0x0)
Log Consistent: 1979220 (0x1e3354)
GenMax Creation: 04/27/2023 01:35:27.162 UTC
Shadowed: Yes
Last Objid: 36408
Scrub Dbtime: 0 (0x0)
Scrub Date: 00/00/1900 00:00:00.000 LOC
Repair Count: 0
Repair Date: 00/00/1900 00:00:00.000 LOC
Old Repair Count: 0
Last Consistent: (0x1151D3,53,979) 03/09/2022 16:32:40.913 UTC
Last Attach: (0x1151D4,2,268) 03/09/2022 17:03:25.440 UTC
Last Detach: (0x0,0,0) 00/00/1900 00:00:00.000 LOC
Last ReAttach: (0x1E33BF,2,268) 04/27/2023 14:54:18.682 UTC
Dbid: 1
Log Signature: Create time:05/12/2020 15:42:40.662 Rand:1590027183 Computer:
OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)
Previous Full Backup:
Log Gen: 1976849-1976951 (0x1e2a11-0x1e2a77) - OSSnapshot
Mark: (0x1E2A78,1,0)
Mark: 04/25/2023 04:03:48.534 UTC
Previous Incremental Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Previous Copy Backup:
Log Gen: 597685-597805 (0x91eb5-0x91f2d) - OSSnapshot
Mark: (0x91F2E,1,0)
Mark: 05/29/2021 22:27:44.801 UTC
Previous Differential Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Current Full Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Current Shadow copy backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
cpgUpgrade55Format: 0
cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0
ECC Fix Success Count: none
Old ECC Fix Success Count: none
ECC Fix Error Count: none
Old ECC Fix Error Count: none
Bad Checksum Error Count: found (1590)
Last Bad Checksum Error Date: 04/27/2023 14:54:16.525 UTC
Old bad Checksum Error Count: none
Last Database Maintenance Finish Date: 00/00/1900 00:00:00.000 LOC
Current Database Maintenance Start Date: 05/01/2022 03:24:28.534 UTC
Highest Continuous Database Maintenance Page: 0
Highest Database Maintenance Page: 0
Database Header Flush Signature: Create time:04/27/2023 14:54:18.682 Rand:4214609386 Computer:
Flush Map Header Flush Signature: Create time:00/00/1900 00:00:00.000 Rand:0 Computer:
Operation completed successfully in 0.110 seconds.
Run the second command, changing the path, to bring it to a clean state.
My guess was that if you recovered the server from a snapshot, it may have went out last night and performed updates that it felt was needed and did a reboot of the server.
ASKER
Rgr that, it's running now.
I'm amazed that you called it, some of you folks are just too good.
Your welcome. Glad you got it running. I would check and make sure you updates are not set to install and reboot automatically if that is what the Event viewer shows happened. It could have rebooted for another reason though, so you need to check the logs.
ASKER
Well, the repair is running now, not sure I'm out of the woods yet, but hopefully it will mount after this finishes.
Thank you for your patience though, I know all too well what it's like to help someone like me...
Oh. I thought you meant the server was running LOL
That is a problem. You can try a defrag. There is a previously resolved issue with that error. Just look at the solution and change your paths to match the defrag command.
https://www.experts-exchange.com/dashboard/#/questions/28984953?
Just to ask, since it mentions the logs. When you did the recovery, were the logs also recovered?
Unless someone else has a better idea, it seems the database has been corrupted. You could attempt another restore and use the commands to check the database. Since you did not have the server fully running yesterday, It may be best to attempt another restore and verify the database is clean before mounting it. If that does not work, you could either open and pay for a MS engineer who could assist or there is a company that makes a Exchange database recovery tool to fix corrupted databases https://www.systoolsgroup.com/exchange-bkf-recovery.html
ASKER
Thanks, Rodney. I'm going to try to restore the mailbox folder again and see if I can't get to a clean state.
Otherwise I'll contact M$ and see if they can help.
Though I would check in and see if you were able to get this recovered.
ASKER
I'll update with any news. I appreciate you, Rodney!
ASKER
So the solution was listed here Email Stuck in Exchange On-premises Transport Queues - Microsoft Community Hub
I'm still seeing some errors I don't like, so I'm in the process of migrating all of the mailboxes to a new database to see if that does anything for me.
Information Store - Mailbox Database 1354787502 (12808,R,98,15.02.0986.015) Mailbox Database 1354787502: The database page read from the file "C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1354787502\Mailbox Database 1354787502.edb" at offset 286329274368 (0x00000042aa8e0000) (database page 8738075 (0x85551B)) for 32768 (0x00008000) bytes failed verification due to a page checksum mismatch. The stored checksum was [0000000000000000:0000000000000000:0000000000000000:0000000000000000] and the computed checksum was [0085551b1b755346:0000000000000000:0000000000000000:0000000000000000]. The read operation will fail with error -1018 (0xfffffc06). If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
ASKER
Yes, we started with a fresh restore, then applied the solution above. This is why it was always starting out as mounted, giving me false hope, and then dismounting when this check failed.
My database migration is almost complete, so I'll know more about resolving that error soon.
I do have a user with a mailbox that is now quarantined, but not the user who was the source of the ransomware. I'm trying to figure out why and how to fix, otherwise I think I'm almost back to normal.
Is it possible the server performed some updates and rebooted overnight causing a dirty shutdown? You can verify it with this command
Open in new window
If the results show a dirty shutdown, run this to bring it to a clean state:
Open in new window
If it then shows clean, try to mount the database.