Corrupt User Mailbox Exchange 2010

bhieb
bhieb used Ask the Experts™
on
VERY long story short, we had a failure. Had to restore exchange to VM. That all went fine.

The event log was throwing off I/O or read errors for the database. The DB is up, and will dismount clean and remount fine, but the event errors worried me. Since I wanted to free up whitespace anyway, and it is a good practice after something goes wrong. I created a new DB and began moving mailboxes to it. I'd say 80% moved fine, but I have a handful that have failed some at 0% all the way to 75%.

My first step was to run this to try and repair the user, and get him moved.
New-MailboxRepairRequest -Mailbox User -CorruptionType ProvisionedFolder,SearchFolder,AggregateCounts,Folderview

Open in new window


This created 2 events, one a 10047 that it started. Then a few mins later it fails with the below.
Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          8/29/2014 8:39:17 AM
Event ID:      10049
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      myserver.abc.loc
Description:
Online integrity check for request da339ff0-1356-4a49-a2c6-e3b47284aefa failed with error -1018.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeIS Mailbox Store" />
    <EventID Qualifiers="49158">10049</EventID>
    <Level>2</Level>
    <Task>6</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-08-29T13:39:17.000000000Z" />
    <EventRecordID>2151157</EventRecordID>
    <Channel>Application</Channel>
    <Computer>myserver.abc.loc</Computer>
    <Security />
  </System>
  <EventData>
    <Data>da339ff0-1356-4a49-a2c6-e3b47284aefa</Data>
    <Data>-1018</Data>
    <Binary>5B444941475F4354585D000006020000FF281500000000000003F801000029E340100F010480D25560200F0104800201FE670201FE6729E340100F010480D25560200F0104800201FE670F01048029E340100F010480D25560200F0104800201FE670F01048029E340100F010480D25560200F0104800201FE670201FE6729E340100F010480D25560200F0104800201FE670F01048029E340100F010480D25560200F0104800201FE671701048029E340100F010480D25560200F0104800201FE670201FE6729E340100F010480D25560200F0104800201FE671701048029E340100F010480D25560200F0104800201FE670F01048029E340100F010480D25560200F0104800201FE670201FE6729E340100F010480D25560200F0104800B00F46717010480D25560200F01048003006E670F010480D25560200F0104804000F6670F010480D25560200F0104800B00F4670201FE67D25560200F01048003006E670F010480D25560200F0104804000F6670F010480B434401006FCFFFF7FF200100F0104807FBA401006FCFFFF7FF5401006FCFFFF9FCA401006FCFFFF9FB210105FB4CA001FBD00100F0104801FFD401006FCFFFF9FFE101050B4CA009F9E401006FCFFFF9FE0101050B4CA009FA0401006FCFFFF1FBD00100F0104801FFD401006FCFFFF97B500100F01048097F5401006FCFFFF17C600100F01048037B9401006FCFFFF37F900100F01048017E6401006FCFFFF</Binary>
  </EventData>
</Event>

Open in new window


What are my options for getting this user moved?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Author

Commented:
If it helps here is the event log that caused me to move the users to another DB and let this one die.

Log Name:      Application
Source:        ESE
Date:          8/29/2014 8:53:13 AM
Event ID:      474
Task Category: Database Page Cache
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      myserver.abc.loc
Description:
Information Store (2060) Mailbox_2: The database page read from the file "F:\ExchangeMailbox\Mailbox_2\Mailbox_2.edb" at offset 596017152 (0x0000000023868000) (database page 18188 (0x470C)) for 32768 (0x00008000) bytes failed verification due to a page checksum mismatch.  The expected checksum was [f8ba07458bdd6052:2239ddc6c056fa0e:801c7fe3ebddfab3:f1c0f1c0165e8920] and the actual checksum was [f8bc07432cea7335:2239ddc6c056fa0e:801c7fe3ebddfab3:e136e136a19f8ab1].  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

For more information, click http://www.microsoft.com/contentredirect.asp.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ESE" />
    <EventID Qualifiers="0">474</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-08-29T13:53:13.000000000Z" />
    <EventRecordID>2151359</EventRecordID>
    <Channel>Application</Channel>
    <Computer>ARMAIL.alanritchey.loc</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Information Store</Data>
    <Data>2060</Data>
    <Data>Mailbox_2: </Data>
    <Data>F:\ExchangeMailbox\Mailbox_2\Mailbox_2.edb</Data>
    <Data>596017152 (0x0000000023868000)</Data>
    <Data>32768 (0x00008000)</Data>
    <Data>-1018 (0xfffffc06)</Data>
    <Data>[f8ba07458bdd6052:2239ddc6c056fa0e:801c7fe3ebddfab3:f1c0f1c0165e8920]</Data>
    <Data>[f8bc07432cea7335:2239ddc6c056fa0e:801c7fe3ebddfab3:e136e136a19f8ab1]</Data>
    <Data>18188 (0x470C)</Data>
  </EventData>
</Event>

Open in new window

Author

Commented:
I have a feeling my only option here is to export the users stuff to a PST kill the old mailbox, and setup a new one then import. Luckily I guess, mine is one of them so I have someone to test on.
Commented:
You could restore and mount a copy of the mailbox db and make an attempt to pull his mailbox from there:
http://exchangeserverpro.com/restore-individual-mailbox-exchange-2010/

This, of course, is a lengthy and tedious task, so I will have to agree that if you can in fact export his PST successfully with minimal corruption/errors, that creating a new mailbox maybe the most effective way to do this,

If the user still has a cached copy of his mailbox in Outlook, while offline, you could export everything from there, NK2, tasks, contacts and calendar included to a PST file. Create a new mailbox and import it.
Should you be charging more for IT Services?

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Yah the problem with a restore is that since this disaster happened last week and it took several days to restore all of the data, I'd actually lose almost a week of emails. PST sounds like the option.

One other thought I had was to try a eseutil /p then /d on the database since there are so few users still in it then move them afterward. That entails a good deal of downtime. So far mine is the only one that is completely inaccessible.

Commented:
ESEUTIL is great if you have the open window but its not always effective. I would recommend exporting any cached copies you may have in Outlook and use that PST to import to your new mailbox.

Author

Commented:
All great ideas. It is 80% through a backup so once that is done I'll give some of it a try. I'll hold this open for a day in case someone else comes up with a magic bullet I'm just not seeing.
Can you open the mailbox at this point to export it to PST?  You might want to look at the move mailbox log first. If the error is caused by a specific item or items that are corrupt, the log will tell you what they are and you may be able to open the mailbox, delete those items from the mailbox and then move it successfully.

Author

Commented:
Good catch. Here is the log. Doesn't look like it is any corrupt items, and is probably deeper seems to be a read failure right from the get go.

8/29/2014 12:26:58 AM [myserver] 'abc.loc/Users/Administrator' created move request.
8/29/2014 12:42:22 AM [myserver] The Microsoft Exchange Mailbox Replication service 'myserver.abc.loc' (14.3.178.0 caps:07) is examining the request.
8/29/2014 12:42:22 AM [myserver] Connected to target mailbox 'Primary (8f697241-dc48-4298-ac96-01357a40ab71)', database 'Mbx1', Mailbox server 'myserver.abc.loc' Version 14.3 (Build 123.0).
8/29/2014 12:42:22 AM [myserver] Connected to source mailbox 'Primary (8f697241-dc48-4298-ac96-01357a40ab71)', database 'Mailbox_1', Mailbox server 'myserver.abc.loc' Version 14.3 (Build 195.0).
8/29/2014 12:42:22 AM [myserver] Request processing started.
8/29/2014 12:42:22 AM [myserver] Fatal error MapiExceptionJetErrorReadVerifyFailure has occurred.
Error details: MapiExceptionJetErrorReadVerifyFailure: Unable to read mailbox signature basic info. (hr=0x80004005, ec=-1018)
Diagnostic context:
    Lid: 46143  
    Lid: 62527   StoreEc: 0xFFFFFC06
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 1238    Remote Context Overflow
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 13236   StoreEc: 0xFFFFFC06
    Lid: 43096  
    Lid: 61528   StoreEc: 0xFFFFFC06
    Lid: 60504  
    Lid: 35928   StoreEc: 0xFFFFFC06
    Lid: 36952  
    Lid: 56383   StoreEc: 0xFFFFFC06
    Lid: 33855   StoreEc: 0xFFFFFC06
    Lid: 37119   StoreEc: 0xFFFFFC06
    Lid: 1750    ---- Remote Context End ----
    Lid: 42047  
    Lid: 58431   StoreEc: 0xFFFFFC06
    Lid: 45887  
    Lid: 62271   StoreEc: 0xFFFFFC06
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcAdmin.GetMailboxSignatureBasicInfo(Guid guidMdb, Guid guidMailbox, MailboxSignatureFlags flags)
   at Microsoft.Mapi.ExRpcAdmin.GetMailboxBasicInfo(Guid guidMdb, Guid guidMailbox, MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.LocalSourceMailbox.Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo(MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.SourceMailboxWrapper.<>c__DisplayClass1.<Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo>b__0()
   at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(GenericCallDelegate operation)
   at Microsoft.Exchange.MailboxReplicationService.SourceMailboxWrapper.Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo(MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.<>c__DisplayClass2b.<CreateDestinationMailbox>b__25()
   at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(GenericCallDelegate operation)
   at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.CreateDestinationMailbox(Object[] wiParams)
   at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CatchKnownExceptions(GenericCallDelegate del, FailureDelegate failureDelegate)
Error context: --------
Operation: ISourceMailbox.GetMailboxBasicInfo
OperationSide: Source
Primary (8f697241-dc48-4298-ac96-01357a40ab71)
Flags: GetMailboxSignature
--------
Mailbox: Primary (8f697241-dc48-4298-ac96-01357a40ab71)
8/29/2014 12:42:22 AM [myserver] Relinquishing job.
8/29/2014 7:51:32 AM [myserver] 'abc.loc/Users/Administrator' resumed move request.
8/29/2014 7:51:38 AM [myserver] Job resumed with status 'Queued'.
8/29/2014 7:51:38 AM [myserver] Relinquishing job.
8/29/2014 7:51:44 AM [myserver] The Microsoft Exchange Mailbox Replication service 'myserver.abc.loc' (14.3.178.0 caps:07) is examining the request.
8/29/2014 7:51:45 AM [myserver] Connected to target mailbox 'Primary (8f697241-dc48-4298-ac96-01357a40ab71)', database 'Mbx1', Mailbox server 'myserver.abc.loc' Version 14.3 (Build 123.0).
8/29/2014 7:51:45 AM [myserver] Connected to source mailbox 'Primary (8f697241-dc48-4298-ac96-01357a40ab71)', database 'Mailbox_1', Mailbox server 'myserver.abc.loc' Version 14.3 (Build 195.0).
8/29/2014 7:51:45 AM [myserver] Request processing started.
8/29/2014 7:51:45 AM [myserver] Fatal error MapiExceptionJetErrorReadVerifyFailure has occurred.
Error details: MapiExceptionJetErrorReadVerifyFailure: Unable to read mailbox signature basic info. (hr=0x80004005, ec=-1018)
Diagnostic context:
    Lid: 46143  
    Lid: 62527   StoreEc: 0xFFFFFC06
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 1238    Remote Context Overflow
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67EF0003
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 23921   StoreEc: 0x3EC     
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x67910014
    Lid: 13236   StoreEc: 0xFFFFFC06
    Lid: 43096  
    Lid: 61528   StoreEc: 0xFFFFFC06
    Lid: 60504  
    Lid: 35928   StoreEc: 0xFFFFFC06
    Lid: 36952  
    Lid: 56383   StoreEc: 0xFFFFFC06
    Lid: 33855   StoreEc: 0xFFFFFC06
    Lid: 37119   StoreEc: 0xFFFFFC06
    Lid: 1750    ---- Remote Context End ----
    Lid: 42047  
    Lid: 58431   StoreEc: 0xFFFFFC06
    Lid: 45887  
    Lid: 62271   StoreEc: 0xFFFFFC06
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcAdmin.GetMailboxSignatureBasicInfo(Guid guidMdb, Guid guidMailbox, MailboxSignatureFlags flags)
   at Microsoft.Mapi.ExRpcAdmin.GetMailboxBasicInfo(Guid guidMdb, Guid guidMailbox, MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.LocalSourceMailbox.Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo(MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.SourceMailboxWrapper.<>c__DisplayClass1.<Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo>b__0()
   at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(GenericCallDelegate operation)
   at Microsoft.Exchange.MailboxReplicationService.SourceMailboxWrapper.Microsoft.Exchange.MailboxReplicationService.ISourceMailbox.GetMailboxBasicInfo(MailboxSignatureFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.<>c__DisplayClass2b.<CreateDestinationMailbox>b__25()
   at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(GenericCallDelegate operation)
   at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.CreateDestinationMailbox(Object[] wiParams)
   at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CatchKnownExceptions(GenericCallDelegate del, FailureDelegate failureDelegate)
Error context: --------
Operation: ISourceMailbox.GetMailboxBasicInfo
OperationSide: Source
Primary (8f697241-dc48-4298-ac96-01357a40ab71)
Flags: GetMailboxSignature
--------
Mailbox: Primary (8f697241-dc48-4298-ac96-01357a40ab71)
8/29/2014 7:51:45 AM [myserver] Relinquishing job.

Open in new window

This appears to be a pretty catastrophic failure.  I would be surprised if you can open the mailbox at all.  Have you tried?  If you can open it, you just might be able to export it to a PST, and/or use the Inbox Repair tool to repair it.

A Microsoft tech article I found about mailbox repair request errors indicated that if you keep getting this error you should run eseutil to repair the database.  That 474 error also points in the same direction.

If you can't open these mailboxes, and really need to try to recover them, I would go with dexIT's suggestion to try to restore a copy of the DB and repair it.  Or if you have a pretty recent database backup before the distaster occurred, you might be able to restore and mount that and at least recover most of the data from those mailboxes.

Author

Commented:
I do have a fairly recent backup, and mine is really the only one fully inaccessible. I'm in the process of moving every stable mailbox to a new store, after that is complete I'll come back and see what can be done about the few that won't move. But even then most of them are usable just too corrupt to move. I'm hopeful between a eseutil /p to repair it and possible a restore from recovery just to be sure some old ones don't go missing, it will work out.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial