Restoring Corrupt Private Store with Backup Exec 9.1

I have a situation with a server that has a corruption in the private store, it is throwing ESE 474 events in the event log:

Information ation Store (132) First Storage Group: The database page read from the file "D:\Program Files\Exchsrvr\mdbdata\priv1.edb" at offset 7616733184 (0x00000001c5fe2000) (database page 1859553 (0x1C5FE1)) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch.  The expected checksum was 6829463886874313980 (0x5ec72138978084fc) and the actual checksum was 5783554595929540653 (0x504350432d75482d).  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
For more information, click

Server is SBS 2003 with SBS SP1, Exchange SP2.

Due to this error, backups of the store have been failing for almost a month.

Exchange is still currently operating OK, apart from the inability to be backed up.

From what I have read, the most likely situation to give full recovery is a restore from the last good backup of the database.

I have this backup, and the subsequent 11gb of log files that have been written since the last successful backup.

The information store is approximately 22gb.

I am considering taking the server offline over the weekend, restoring from tape and then replaying the log files to bring the server back up to its current state.

We are backing up using Veritas 9.1 for SBS.

I have been looking at the Veritas restore and want to ask a few questions about the restore options:

1 - I assume I can just leave the "Exchange v5.5" alone?

2 - "No Loss Restore (do not delete existing transaction logs)" - I assume I leave this on so that if something goes horribly wrong, I can try to restore and the log files will still be there?

3 - "Temporary location for log and patch files" - does it only read from this location, or do I need to make sure there is free space on this drive for it to write something to? If so, how much?

4 - "Commit after restore completes" - as far as I can tell this is the one that tells exchange to replay the existing log files once the restore is completed, to bring me up to the current state?

5 - "Mount database afer restore" - that one seems kind of obvious.

As far as the steps go, I plan to dismount the public store, move the existing priv.edb and priv.stm to another location (to give me a recovery option if it all goes terribly wrong), tick the box to allow the store to be overwritten with a backup and then fire off the job.

Is there anything else to consider? If all goes well, should the database come up with no data loss (ie the replayed logs will replace the data from the last month)?

Is there any way I can estimate how long it will take to replay the months worth of logs, so I can give a rough idea of downtime?

I know there are a lot of questions in there, I just want to cover my bases before I try anything. Appreciate your help in advance.


Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

the steps would be

1) rename the folder containing your logs and databases

2) create new folders by the same name in the same location

3) put a check " this database can be overwritten by a restore"

4) start the restore

"Temporary location for log and patch files"  - this is location where the restore.env and other logs are kept after restore

"Commit after restore completes" - this is same as running eseutil /cc (this commits all the logs in the temp location into the exchange database after the restore completes)

"Mount database afer restore" - this option mounts the database after the restore has completed (after playing all the logs)

don't select the "commit after restore completes" and "commit after restore"

what do need to do it complete the restore and check the "temp" location for the restore.env file and the logs present there.
run eseutil /cm against restore.env file and check the log files present there
after that check whether the 11Gb log files that you have, are in sequence with the logs present in the "temp" folder
also run eseutil /mh against the restored database to check the log required value
if all the logs (logs in temp folder and the 11 Gb logs) are in sequence then, once the restore completes, copy the 11Gb logs in the original location where you had the logs before the restore.

run the command as eseutil /cc <folder name of the restore.env file>
this command will first play the logs that you have in the temp folder and after that it would play the logs those 11Gb logs

take a backup of the folder containing restore.env and logs present in that folder, before running the eseutil /cc command

let me know if you need any more details

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
arrowtechAuthor Commented:
I am kind of with you here.

I think I have a hole in my information that is stopping me understanding part of this process.

When I do the restore, and specify a temp location, the priv.edb and priv.stm are restored from the backup, and some log files are restored to the temp directory.

What are these log files? From my understanding, day to day while exchange runs, changes to the database are written to the log files and held in memory, and then comitted to the database when convienient. When the backup is taken, the databases are backed up and the logs flushed. So what are the logs? The last few that hadn't been written to the store yet?

Assuming that, lets say that that there at 5 log files restored along with the database.

My most recent logfile currently is E000542D.log, my oldest from just after the last good backup is E0004F61.log. Would I want the restore.env file to say:
Log files range: E0004F5C.log - E0004F60.log

Also, what do I take away from running eseutil /mh? When I run it on my own server just to find out the output, it tells me:

Log Required: 0-0 (0x0-0x0)

What should I expect to see after a restore?

Finally, what do I do about the public folders? They are not busted, but since they have the same log files do I have to restore them the same way?

Due to hard drive constraints on the server, files have been put in different places.

System Path, Transaction Logs, and priv1.stm are in
E:\Program Files\Exchsrvr\MDBDATA

priv1.edb, pub1.stm and pub1.edb are in
D:\Program Files\Exchsrvr\mdbdata\

So obviously if I rename those two folders, that will upset my public folders slightly.

My log files are 11gb on a 20gb partition. Can I just move the existing log files back and forth rather than copying them - obviously that will be a slow process.

Finally, what do I do with E00tmp.log, E00.chk, E00.log, res1.log and res2.log during this process?

Sorry for all the followup questions, just want to make sure I don't really screw the pooch on this one.

when an online backup is done the databases are inconsistent by default.. when the backup takes place the databases are still running and users are using it. so any transaction commited at the time of backup goes in those logs that are restored in the temp folder.
the logs are required to bring the database into a consistent state.

when you run eseutil /mh on the database it will show you the log required value, which will be 0-0 incase database is consistent.
when you have restored from an online backup the database is not consistent and the log required value would be the logs present in the temp folder

now, the logs that you have in that temp folder should be in sequence with the logs that are in the mdbdata folder, so when you run eseutil /cc on the restore.env file it plays the logs that are present in the temp folder and then moves on the play the logs that are in the mdbdata folder

you need to rename all the mdbdata folders and create blank mdbdata folders in their place

after the restore completes, you need to copy the public store.edb and public folder.stm into the same folder where it existed earlier

after that you copy the 11GB logs into the mdbdata folder or rename the earlier renamed folder of logs back to it original folder (before that check whether you have any logs in the new logs folder)

once done, run the eseutil /cc on the restore.env file.. this would play the logs that are in the temp folder first and then will play the logs that are in the original logs folder (logs should be in sequence otherwise you will get an error)

to check the log files sequence you can run the command as eseutil /r E00 on the logs folder

Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

arrowtechAuthor Commented:
OK, that is pretty clear now.

Two more questions:

As above - do I keep the  E00tmp.log, E00.chk, E00.log, res1.log and res2.log files in place when copying my log files around?

Also, from reading Veritas/Symantec KB articles, this seems to be the same thing as if I do click the Commit after Restore Completes and the Mount after restore completes checkboxes, just with a lot more manual stops in between to check everything is going as it should be. I am all for that, but is that the case?

when you move out the logs, move out all of them. and when you copy them back then copy all of them except E00.chk

and you are right that "commit after restore" and "mount after restore" are the same as in running eseutil /cc, but if you put a check on these two checkboxes then you don't get the chance to copy back the logs and the public store into the original location, after the restore completes
arrowtechAuthor Commented:
Wow - that is some lightning fast response action.

Thanks so much for all of your help, I think I will take the server down Friday night, do an Exmerge of all users of to external disks to be safe, and then have at it.

I will let you know how I travel - your help has been invaluable.

One more thing - I certainly hope that your on-disk transaction log are not corrupted, I'd hate this to be the cause of the problem.

Also, make sure to check the underlying subsystems - memory, disk drives, raid controller firmware and drivers...

Rakeshmiglani gave you some excellent advice, good luck there.
arrowtechAuthor Commented:
If they are they are, there is not much I can do about it.

The eventlog shows no sign of a disk problem, no dropped disks in the raid controller or reported problems, so I just have to work under the assumption that the log files are not corrupted.

I have done individual brick level backups of all mailboxes today, and will exmerge all mailboxes before I start, just in case I do have a disc problem and can't recover.

arrowtechAuthor Commented:
For those of you playing at home, here is how it all shook down.

I spent all of Thursday doing brick level backups individually to make sure I got each mailbox.

I spent the better part of Friday night Exmerging the mailboxes out to PST files - the reason it took so long was 5 users with +2gb mailboxes that i had to break down by folders/dates/all sorts of crap.

Once I got all the data thoroughly backed up 6 ways, I dismounted the stores, selected for them not to be mounted at startup, stopped the IS service, renamed the folders and started the IS service again.

Kicked off the restore, and waited nervously, chewing my nails.

The restore succeeded, there was a restore.env and a single log file in the temp folder, this log file was the same name as the first log file in the log files folder, so everything looked good. I ran eseutil /cm, it called for the correct log file, so all was happy.

I moved the log files back to the correct location, ran the eseutil /cc and waited nervously, chewing my nails.

The event log gave me details of each log file that was being replayed (I set recovery logging to full in ESM). Replaying of the 11gb of log files (about a months worth) took 2 hours 15 minutes.

I moved the public folders back to the correct location, clicked to mount the stores, and waited nervously, chewing my nails.

Finally, the test was to backup the IS, as this was the whole problem to start with. Ran the backup and waited nervously chewing my nails.

Backup successful, logs deleted, everyone is happy again.

My eternal gratitude to rakeshmiglani - concise and accurate help in response to my uneducated questions.

Excellent job Arrowtech and thanks for the details.

ESE rules.
Thanks.. Happy to know that the issue is resolved.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.