Avatar of richardsit
richardsit
 asked on

Exchange 2010 (SBS 2011) Power failure, corrupted log files, eseutil /r doesn't work. Backups are no good.

Have an sbs 2011 server running exchange 2010. Server was crashed this morning. Only recourse was to have onsite customer power cycle. After the initial power cycle I know for sure Exchange worked because emails cmae to me from them that had been queued in their outlook. However, I guess someone got impatient with the long boot times of the server and power cycled the server two more times each time before it finished.

So now we have corrupted log files. Backups are no good for a variety of reasons and all of them boil down to the customer ignoring repeated warnings and not holding up their end. Which is nice for the blame game, but doesn't help fix the problem which is all I care about at this point.

I am not an expert on fixing this specific problem. I have tried following several online instructions, but things may have gotten worse.

I've tried eseutil /r, but it fails with a -501 log file is corrupt after 0.172 seconds.

Tried it on older set of log files from before the crash (shadow copy), but this gets half way then fails out with -1216 an outstanding database attachment has been detected at the start or end of recovery, but database is mising or does not match attchment info after 1.263 seconds

so before I make it any worse, I'm hoping someone can help me. I'd just love to get the thing back in the state it was at 9 am before the crash. I have shadow copies and that's it.

If we lose some recent data, fine. I just need this sucker mounted and running.
ExchangeSBS

Avatar of undefined
Last Comment
richardsit

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
I Qasmi

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
richardsit

ASKER
OK! Got a successful completion of recovery using a restored copy of both the logs and db file form the same cycle.

Is it safe to just mount or do I need to do any other steps first? restoring those shadow copies takes a long time.

/mh does report clean shutdown now.
richardsit

ASKER
OK, I went ahead and mounted it. It mounts but there are problems.

Some mailboxes are inaccessible via webmail, probably outlook as well. I've got a few dozen on there and of the 5 I've logged into, 2 are broken.

I'm getting errors in the event logs.

Will try a reboot I guess.
richardsit

ASKER
Didn't help. Many boxes are broken. Bummer. I need those working. Would going back to a shadow copy from eariler and doing /r /a on that one help?

Or am I going to have to recreate those mailboxes and import from .ost files?
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
I Qasmi

Reboot the server and check the event viewer and report your findings
If the stores are still dismounted
I Qasmi

Before performing the eseutil/r/a just move all the logs except the database files
should be there in the original location

Move all the other files from the location where the logs are located
and then perform the recovery After successfull recovery , Mount the database and u can mount them easily
richardsit

ASKER
after tha /r /a repair it mounts, but many mailboxes are broken. Reboot did not fix that, but it's still mounted. I will look through the logs.

Lots of ExchangeStoreDB error 203: database copy 'Mailbox Database 2011072611' on this server appears to have an I/O error that it may be able to repair.  To help identify the failure, consult the Event log on the server for other storage and "ExchangeStoreDb" events. Service recovery was attempted by failover to another copy. The failover was unsuccessful in restoring the service because of the following error: 'There is only one copy of this mailbox database (Mailbox Database 2011072611). Automatic recovery is not available..

followed by ExchangeStoreDB error 223: database copy 'Mailbox Database 2011072611' on this server encountered an error. For more information, consult the Event log for "ExchangeStoreDb" or "MSExchangeRepl" events

And a few ExchangeStoreDB error 209: At '3/25/2013 8:42:11 PM' the Exchange store database 'Mailbox Database 2011072611' copy on this server detected corruption on the active copy of the database. To help identify the specific failure, consult the Event log on the server for other storage and "ExchangeStoreDb" events.  Service recovery was attempted by failover to another copy, which was unsuccessful in restoring the service. Error: There is only one copy of this mailbox database (Mailbox Database 2011072611). Automatic recovery is not available.



A variety of warnings from activesync (1008)  and serch indexer (107) Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database Mailbox Database 2011072611 (GUID = 4c431b42-0a16-45d1-a343-332528a60766) due to an error (Microsoft.Mapi.MapiExceptionJetErrorReadVerifyFailure: MapiExceptionJetErrorReadVerifyFailure: Unable to update CI Retry table  (hr=0x80004005, ec=-1018)
Diagnostic context:

[There's more to that one if it's relevant]

A few 474 errors from ESE:
Information Store (16472) Mailbox Database 2011072611: The database page read from the file "D:\Program Files\Microsoft\Exchange Server\V14\Mailbox\Mailbox Database 2011072611\Mailbox Database 2011072611.edb" at offset 127433015296 (0x0000001dab998000) (database page 3888946 (0x3B5732)) for 32768 (0x00008000) bytes failed verification due to a page checksum mismatch.  The expected checksum was [3fb23fb2167feeb5:1550eaafcfaed7cd:3432cbcdbf2289f3:00a300a37ef802ba] and the actual checksum was [3fb93fb968321944:1550eaafcfaed7cd:e6db19241384ac20:00a100a17ef80273].  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.

that one says hardware, but that seems unlikely in this case.


I think that's all of it.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
richardsit

ASKER
by: iQasmiPosted on 2013-03-25 at 20:45:55ID: 39020255

Before performing the eseutil/r/a just move all the logs except the database files
should be there in the original location

Move all the other files from the location where the logs are located
and then perform the recovery After successfull recovery , Mount the database and u can mount them easily

do you mean that I should redo the repair on a restored copy of the db and remove all the log files? Or remove all the logfiles and run the repair again on the db that mounts but doesn't work right?

My log files are on a separate drive by the way. This is how SBS sets it up when you move the database files in the sbs management console. It also defaults to circular logging, just in case that matters. I meant to mention that in the first post.

I'm kind of getting lost here, so if you can tell me sepcifically what to try, I would really appreciate it. Thanks.
I Qasmi

Yes move all the files (including log files) to another folder or create a new folder there only and move all the log files and other files to this new folder.

Except the database files.Remove Circular logging ,
Then try mounting the database.

Also ESE 474 Event depicts corruption in your database.
Also - 1018 indicates that some antivirus is interfering in between
Add the exchange folders to the Antivirus exclusion list.
and then while maintainence disable the antivirus.

After performing eseutil/r/a
Repair the mailbx corruption using the link below
http://blogs.technet.com/b/exchange/archive/2010/08/23/3410708.aspx

or

Export it to OST files and then import back it to database format for those mailboxes those who are corrupted
richardsit

ASKER
I think I'm going to go with the import option. It's more time consuming, but I'm nervous taking the db offline and messing with it again. May make it even worse.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
richardsit

ASKER
Also, AV is off during all the repair and testing portions of this.
I Qasmi

check the drives for corruption as well with
CHKDSK and check whether the drive is corrupted and needs to be fixed

Importing option requires the database to be mounted
Also leave it if its time consuming and perform the mailbox cirruption fixation
once you have time

If Antivirus is off then it must be the backup which is interfering in it
richardsit

ASKER
I am confused at this point. I thik I have too many broken boxes to try and import and export.

I'm not sure I understand the procedure you recommended.

do you propose that I remove everything from the log file directory and then run the repair as /r /a? Will that break the boxes I have already salvaged?
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
richardsit

ASKER
I have tried the export, replace mailbox, import on one small box and it does not seem to be working well. the user is constantly having to log back in to exchange from outlook. I am attempting to restore that box from an old backup, then import. If that doesn't work I am back to needing a procedure I can understand to repair the database.