Exchange 2013 partial database corruption

We had a dirty shutdown on our main edb. we ran ESEUTIL /p to repair and were able to mount the database.

However a handful of users are unable to connect through outlook or OWA and we are getting NDR's from these accounts as follows:

EXCHSVR.local #554 5.2.0 STOREDRV.Deliver.Exception:StoragePermanentException.MapiExceptionDatabaseError; Failed to process message due to a permanent exception with message Cannot complete delivery-time processing. 16.55847:4B220000, 17.43559:540400001A010000000000000600000000000000, 4.32881:54040000, 0.50035:00000000, 4.64625:54040000, 20.52176:000F04821E00F01F000F0482, 20.50032:000F04828E17F01F0E000000, 0.50128:06000000, 255.1494:20030000, 6.45653:0500078040000830DD040000, 4.41064:05000780, 6.45653:050007801F00FA3F05000780, 4.41064:05000780, 6.45653:050007804000073005000780, 4.41064:05000780, 6.64024:1A0104800201133005000780, 4.41064:1A010480, 6.45653:050007801F00F83F1A010480, 4.41064:05000780, 4.65304:0F010480, 4.65480:BBF9FFFF, 4.64456:54040000, 255.1750:00000000, 255.1494:0E000000, 6.45653:0500078040000830DD040000, 4.41064:05000780, 6.45653:050007801F00FA3F05000780, 4.41064:05000780, 6.45653:050007804000073005000780, 4.41064:05000780, 6.64024:1A0104800201133005000780, 4.41064:1A010480, 6.45653:050007801F00F83F1A010480, 4.41064:05000780, 4.65304:0F010480, 4.65480:BBF9FFFF, 4.64456:54040000, 255.1750:9E000000, 0.50288:00000000, 4.23354:54040000, 0.25913:0E000000, 255.21817:54040000 [Stage: Delivery]

We also cannot use the migration tool in the ECP to move the user mailbox to another database as we get the following error:

The operation couldn't be performed because object 'USERMBX' couldn't be found on 'Microsoft.Exchange.Data.Storage.MailboxSession '.

We need to find a solution to either get this database working or to migrate the user mailboxes, unfortunately we have so many users we cannot bring down the partially broken database. We have a Clean DB but this is a number of hours old and some users have already been sending and receiving mail in this DB today.
LVL 2
purpleoakAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

HariomExchange ExpertsCommented:
Please check workaround section on following article also Which CU are you running , make sure you are running Cumulative Update 6 for Exchange Server 2013

https://support.microsoft.com/en-us/kb/2958434

What happened to broken database , have you checked the state using eseutil/mh , if it was dirty shutdown then have you tried to replay the log files to bring the database in clean state ?
purpleoakAuthor Commented:
we checked it using eseutil /mh and it was stating clean shutdown, we also have a copy of it which was showing dirty shutdown which we are running an offline defrag on now.
HariomExchange ExpertsCommented:
No need for offline defrage

If your database is in cleanstate then log files is no longer required move the transaction logs to some other directory and then try to mount the store.

Move all the log files and other files except .EDB file (if you have it on the same location as log file location) and mount the database.

“CLEAN SHUTDOWN” means almost for sure there is no corruption

Please move all the log files ,.chk files to another folder and then try to mount the store.

My blog :
http://viralr.wordpress.com/exchange-database-recovery/
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

purpleoakAuthor Commented:
if there is no DB corruption then how come we cant set up mailbox move request?
purpleoakAuthor Commented:
Mail Server Version is Version 15.0 (Build 516.32)
purpleoakAuthor Commented:
The Database is mounted, we have a few mailboxes that we cant move or access, if we try to run a mailbox repair we get mailbox not found errors, if we try to move the mailbox through ECP we get job cannot be found errors and if I try a new-MoveRequest CMDLet i get StalledDueToWriteThrottle and it stops moving at 25%.
Miguel Angel Perez MuñozCommented:
You can try create new database, then export all damaged mailboxes as pst, remove maliboxes, create on new database and finally reimport from exported pst: http://www.msexchange.org/articles-tutorials/exchange-server-2013/management-administration/managing-pst-import-export-process-exchange-server-2013-part1.html

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
purpleoakAuthor Commented:
Thanks for the idea Miguel, Unfortunately we cannot use this solution on our Database.
Will SzymkowskiSenior Solution ArchitectCommented:
I would recommend that you try using Kernel Data Recovery for Exchange Server. This works much better then ESEUtil and does a low level integrity check and scan to re-pair Exchange Databases.

Kernel Data Recoevery Exchange Server
http://www.kerneldatarecovery.com/exchange-server-recovery.html


Will.
Kim HartCommented:
Use the New-MailboxRepairRequest cmdlet to detect and fix mailbox corruptions. You can run this command against a specific mailbox or against a database: https://technet.microsoft.com/en-us/library/ff625226%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396

Recovery databases: https://technet.microsoft.com/en-us/library/dd876954%28v=exchg.150%29.aspx

Exchange Database Recovery – Using eseutil commands: http://msexchangeguru.com/2009/07/12/exchange-database-recovery-using-eseutil-commands/
Marshal HubsEmail ConsultantCommented:
If you are still unable to resolve your problem, then you can try Stellar Phoenix Mailbox Exchange Recovery software, which will surely help you in resolving your problem as I'm personally using it and every time it resolved my problem and restored inaccessible mailboxes from corrupt EDB. you can download the software from here -   http://download.cnet.com/Stellar-Phoenix-Mailbox-Exchange-Recovery/3000-2369_4-10745362.html
Brando PeterCommented:
Running eseutil /p or a hard repair does not ensure that the database is 100% healthy. There can still be lingering problems, such as broken index entries and mis-allocated pages. It used to be advised for you to run eseutil /d - defrag and then run an isinteg -fix (new-mailboxrepairrequest in 2010/2013).

These days though, Microsoft itself advises for you to move mailboxes out of the db after doing an eseutil /p, as the integrity of the database is still suspect and can continue to cause dismounts and other db issues.

If you can't move the mailboxes out of the DB permanently (due to the db name and capacity), then just do a temporary move of the db mailboxes to a temp db, remove the old db, recreate it and restore the mailboxes from backup. The process of recreating the databases ensures that the system tables and indexes are good and restoring mailboxes ensures that the mailbox tables are recreated and good as well.
purpleoakAuthor Commented:
Currently we are moving mailboxes using the new-moverequest cmdlet, a few are getting stuck with a status of relinquished with an error of Resource reservation failed with status 'CapacityExceeded' for reservation.... Any ideas?

Thanks
Brando PeterCommented:
Have you checked the target database mailbox quota?  It could less than the size of the mailboxes you're moving. Typically happens with those VIP mailboxes with unlimited or huge quotas.
purpleoakAuthor Commented:
The limits shouldn't be a problem here, they were set to 10.5gb per mailbox and the user that is relinquished is a 7.172gb mailbox
jrhelgesonCommented:
This has all the earmarks of an exchange problem combined with an Active Directory problem.  Is this exchange server a DC? Tell me more about your DC setup?  I'm wondering if the message store has the mailbox, but the reference within AD is what was broken.
purpleoakAuthor Commented:
The DC is on a separate server.
Mohammad Mahboob RahmatiSr . IT System Administration EngineerCommented:
ESEUTIL isn’t a new command-line tool, it’s been around since version 5.5 of Exchange Server and was used to defragment, repair and verify the integrity of Exchange databases. Depending on your experience with Exchange (and hopefully it has always been good) you may never need to use ESEUTIL.

However, if you ever find yourself restoring an Exchange database from a backup, you may find that the mounting aspect of that database won’t go so well. Your database is dirty and needs a little ESEUTIL scrub down.

How to Tell if Your Exchange Database is Dirty
What we mean by dirty is that the database is in a dirty shutdown state and typically means a transaction log(s) is missing (most likely your current log, ie. E00 or E01 or whatever your transaction log prefix).

So, first off, how can you tell if your database is in a dirty shutdown state? After you recover the database (but before trying to mount it) you can open a command prompt at the location of the database on the Exchange Server and type in: eseutil /mh “database name” as shown in Figure 1 below.

This screenshot, taken from my brand new Exchange 2010 Backup and Recovery Training course shows a database that has been recovered named stgirosgolfcourse.edb. We are using the /mh switch to help us read the header information from the database.

Using ESEUTIL
Figure 1: eseutil /mh “database name”

When searching the results of the eseutil command we can see in Figure 2 that it was found to be in a Dirty Shutdown state. If you want to check the header of the log file you can use eseutil /mh but this time you would indicate the log, for example E00.log. You can even review the header of the checkpoint file in the same way.

Using ESEUTIL
Figure 2: Identifying a database in dirty shutdown state

How to Repair a Dirty Database with ESEUTIL
You want to repair your dirty database and bring it to a clean shutdown state, but you have to make sure the database is consistent with the logs. So, you start off with a soft recovery using the eseutil /r switch. If your logs and database are in different locations you can use the /l “insert location” and /d “insert location” parameters as well.

You can see in Figure 3 that we used the command eseutil /r e02 to perform the soft recovery and you can see that it completed successfully.

Using ESEUTIL
Figure 3: How to perform a soft recovery

You hope this works when restoring from a backup because once the database is in a clean state (which you can now confirm by typing in eseutil /mh again) you can mount the database or switch it with a recovery database and begin recovering mailboxes or individual email items.

If it doesn’t work? Well, you might be looking at a hard repair, which is SURE to lose data along the way and it isn’t predictable in terms of how much. To perform a hard repair you type eseutil /p and indicate the database.

Playing with ESEUTIL
Most folks hope never to have to use ESEUTIL, but when they do, they aren’t ready. But you can play with ESEUTIL safely by backing up Exchange with Windows Server Backup, restoring to a folder on your system and using ESEUTIL on that restored database. You don’t have to mount it (although if you do, you should mount it to a Recovery Database, which is not so dangerous and good practice for you).

Once you are done playing you can just dismount and delete the Recovery Database and your production data will remain safe. However, if you aren’t too comfortable just yet with Windows Server Backup, creating and working with Recovery Databases and so forth, you might do well to check out my new course first.

if its not working then you have the only below choice .

Kernel for Exchange Server Recovery..

http://www.nucleustechnologies.com/Exchange-Server-Data-Recovery.html?gclid=Cj0KEQjwtO2wBRCu0d2dkvjVi5cBEiQAMEIVGZDIQBjce96vtMvZ5CY_5Lk09F5EBNh-9sfmpzNXzEMaAmVO8P8HAQ
purpleoakAuthor Commented:
To Fix this I have moved all users off of the corrupt DB onto a New DB using
new-moverequest cmdlet and deleted the corrupted DB.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.