Problems restoring Exchange 2003 SBS Mailbox stores

Hello.  I hope someone here can help me as I am in a bit over my head.  I have a client with a stand alone SBS 2003 Server.  A couple of days ago I started getting some reports of errors from the exchange mailboxes.  I ran a ESEutil /g which finished with error "1206 JET_errDatabseCorrupt"  during troubleshooting I at some point ran eseutil /P which claimed it ran find and solved the problem.  I then also ran another eseutil /g which came up clean so I thought I was done and mounted the database.

The next morning when everyone came back into the office (There are about 18 computers running off this server) Some of the people were missing a lot of e-mail from 2010.  We had a backup that was done on Friday the 16th using "Backup Assist" version 5..2.5 and I believe it is just an interface that runs NTBackup.  The backup log showed that it had completed successfully and verify is turned on.  so I proceeded to perform a restore of the information store only to be met after 3 hours with the following errors after which the store is no longer mountable now.
Here are the errors from the backup software and the event viewer:

Restore Status
Operation: Restore

Backup of "KPH01\Microsoft Information Store\First Storage Group", Restored to"KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"

Restore started on 20/04/2010 at 7:23 PM.
Unable to restore Exchange data to KPH01\Microsoft Information Store\First Storage Group,
check the application event log for more information
Restore completed on 20/04/2010 at 10:14 PM.
Directories: 2
Files: 2
Bytes: 21,319,733,610
Time:  2 hours,  50 minutes, and  41 seconds

Event Type:      Error
Event Source:      NTBackup
Event Category:      None
Event ID:      8019
Date:            20/04/2010
Time:            10:14:23 PM
User:            N/A
Computer:      KPH01
Description:
End Operation: Warnings or errors were encountered.
Consult the backup report for more details.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:      Error
Event Source:      NTBackup
Event Category:      None
Event ID:      8003
Date:            20/04/2010
Time:            10:14:21 PM
User:            N/A
Computer:      KPH01
Description:
End Restore to 'KPH01\Microsoft Information Store\First Storage Group' 'Failed'

       Verify:  Off
 
Consult the backup report for more detail.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type:      Error
Event Source:      ESE BACKUP
Event Category:      Callback
Event ID:      904
Date:            20/04/2010
Time:            10:14:21 PM
User:            N/A
Computer:      KPH01
Description:
Information Store (5220) Callback function call ErrESECBRestoreGetDestination ended with error 0xC7FE1F41 Database is in use.
.

For more information, click http://www.microsoft.com/contentredirect.asp.


also throughout the course of the day and while running the backup the following error kept being logged about once a minute.  I thought it was because the information store was offline but mabye there is something else to it.

Event Type:      Error
Event Source:      MSExchangeSA
Event Category:      MAPI Session
Event ID:      9175
Date:            20/04/2010
Time:            10:13:18 PM
User:            N/A
Computer:      KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000

For more information, click http://www.microsoft.com/contentredirect.asp.

Hopefully someone here can help and is either up late or in a time zone that is earlier in the day than mine  (Canada Pacific Time) as I need to get this back up and running by morning when everyone comes back to work.

also please note that I am working on the system remotely via LogMeIn (I also have remote desktop available to me if it makes a difference).

Thank you in advance.

Carl.
LVL 1
jctcomAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

oztrodamusCommented:
Hi jctcom,

After you ran the ESEUtil /P command to repair the database did you run the ISInteg to repair the database at the application level? You might have to rerun the ESEUtil /P command, because it's been a little while since you first ran it, but try running ISINTEG –FIX.

Make sure you have a proper back up of the databases before running any of these commands.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jctcomAuthor Commented:
If I run another eseutil /p at this point am I going to be running it on the attempted restored database or does the original database remain intact (Such as it is) because the restore failed?

also I have not used the isinteg command.  Could you provide me with the syntax?

Thank you.
0
oztrodamusCommented:
It depends on what you have to work with, but if you could go back to a copy of the database that you know works well enough and still has current enough email all the better. Otherwise you need to use the current database, and yes make backup copy of it.

ISINTEG command line syntax
http://support.microsoft.com/kb/301460
0
Acronis Data Cloud 7.8 Enhances Cyber Protection

A closer look at five essential enhancements that benefit end-users and help MSPs take their cloud data protection business further.

oztrodamusCommented:
I'm not sure if eseutil /p makes a backup copy of the database prior to the repair. I'm only aware of copy being made durring a defrag.
0
jctcomAuthor Commented:
I guess my real issue at this point is why can't I restore the backup I did from the 16th?  By all accounts it was listed as a good and verified backup. Why all those errors?  

I also  tried copying the whole MDBDATA Folder to a temporary folder before I started the restore but it would not let me copy some of the files.
These are the files I did manage to copy:

E00.chk
E00010B1.log
E00010B2.log
priv1.edb
priv1.stm
priv1.INTEG.RAW
pub1.INTEG.RAW
res1.log
res2.log

I also have files from the following folders:

From the MDBDATA Logs Temp Folder I have a lot of .log files plus one "E00.chk" file

From the MDBDATA.temp Folder I have a number of files including a priv1.edb file but I don't think this can be complete as it is far too small only 5GB whereas the original is about 21GB

Am I better to try working with the current data which won't mount at all or can I somehow use these files that I copied before attempting the restore and get them to the point where they are mountable again?
0
moregadCommented:
Did  you do the following before runnng restore with NTBackup:

   1. Start Exchange System Manager, right-click the database that you want to restore, and then click Dismount Store.
   2. When you are asked if you want to dismount the store and disconnect any users, click Yes.
   3. Right-click the store again, and then click Properties.
   4. Click the Database tab, and then click to select the This database can be overwritten check box.

cheers,
0
hutnorCommented:
Have you ticked the box on the exchange server that says "Allow this database to be overwritten by a restore" ??
0
hutnorCommented:
Make sure you have the box ticked on all the stores you are going to restore.

Sorry moregad for the repeat. your comment was not there when i was posting.
0
jctcomAuthor Commented:
Ok.  I did dismount the store (Actually I did not have to dismount it the second time as it would not mount in the first place),  but I did not tick an option to allow the database to be overwritten.  Please let this be the solution.  I am going to try another restore right now with that option.  Unfortunately it takes about 3 hours (Or it did the last times).  Should I restart the server before attempting the restore again?  (I did restart before the last attempt and no one has been in the office since).

Carl.
0
jctcomAuthor Commented:
Is there a way for me to stop these

Event Type:      Error
Event Source:      MSExchangeSA
Event Category:      MAPI Session
Event ID:      9175
Date:            20/04/2010
Time:            10:13:18 PM
User:            N/A
Computer:      KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000

During the restore?  They are flooding the event viewer and I have a system set up to send me e-mails for errors in the event viewer.  But more than that is it going to affect the restore operation at all?

Carl.
0
hutnorCommented:
stop the information store service
0
hutnorCommented:
When your restore from backup is complete you will need to let exchange play out the transaction log files so the restored database will be up to date. Otherwise you will have missing emails.
0
jctcomAuthor Commented:
Ok the information Store service ("Microsoft Exchange Information Store" in services.msc) is stopped.  So just waiting to see if you guys think a reboot is necessary or not before I start the restore operation again?

Carl.
0
jctcomAuthor Commented:
@hutnor.  Is there something I need to do to make "Exchange play out the transaction log files" or should it do that automatically after the restore?  also how long does that take?   for a priv1.edb that is about 21GB.
0
hutnorCommented:
If you do want to restart have a look at the store & see if it was mount when the server started back up.
I do not see any problem restarting.
0
jctcomAuthor Commented:
Also In the restore dialogue box after selecting the source file there are two optional check boxes.

1.  "Last Restore Set (Log File Replay with restart after this restore completes)"  I am guessing this is the answer to my question above to hutnor. and it should be checked.

2.  "Mount Database after restore."         Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?

Carl.
0
jctcomAuthor Commented:
The database definitely did not mount after a restart.  But I think I will restart it now just to make sure it is a clean as possible for the restore.

Carl.
0
jctcomAuthor Commented:
Hmmm  Did I stop the wrong service or something.   I just checked before restarting and I am still getting those Event ID 9175 errors every minute or so.

Carl.
0
jctcomAuthor Commented:
I also shut down the "Microsoft Exchange IMAP4" service.  I am thinking this might more likely be the cause of the Event ID 9175 errors no?  There is no problem leaving this service stopped as well during the restore?

Carl.
0
jctcomAuthor Commented:
Damn.  I am still getting these Event ID 9175 errors every minute.  any ideas on this.  I would like to get rid of this before I restart  (I know I will need to shut down the services again after restarting) and start the restore.

Carl.
0
hutnorCommented:
i will have a look at that service
0
hutnorCommented:
It will pick it up automatic & replay them itself. Yes that is what you want that first tick box.

As for time I have no idea. Maybe someone else has a better idea.

Leave all your services running after the restart & do the restore. I would not worry about having extra errors in the event viewer.
http://support.microsoft.com/kb/258243
read over that before you do anything & follow it step by step.
0
jctcomAuthor Commented:
Ok I found this link to info about that error:

http://support.microsoft.com/kb/262456

But since we don't want the Information Store running or database mounted how do i stop the "System Attendant" in the meantime?

also Still would like to get the answers to these two questions I posted above before starting the restore.

>>>>
Also In the restore dialogue box after selecting the source file there are two optional check boxes.

1.  "Last Restore Set (Log File Replay with restart after this restore completes)"  I am guessing this is the answer to my question above to hutnor. and it should be checked.

2.  "Mount Database after restore."         Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?
<<<<

0
jctcomAuthor Commented:
Ok have just restarted the server Checked that the database did not automount and the "allow restore to overwrite database" is still checked and stopped the Information store & IMAP4 services and I was going to stop the "System Attendant" service but it sounds like you are saying I should restart the services before I start the restore?

Carl.
0
moregadCommented:
try run: net stop iisadmin /y /y

you don't want smpt running while you are restoring...

cheers,
0
jctcomAuthor Commented:
and that last check box?  "Mount Database after restore"?  Checked or not checked?

Carl.
0
hutnorCommented:
Yes make sure all the services are running.
Information store & IMAP4 services & System Attendant
0
hutnorCommented:
check it.
also after restore is finish have a look yourself!
0
jctcomAuthor Commented:
@moreqad

I ran that command and all those services stopped successfully.  I think I am ready to start the restore process now.

I'll wait 5 minutes for any last comments and then I am going to start the restore.

Carl.
0
jctcomAuthor Commented:
Ok wait a minute.  We have conflicting info.  hutnor says leave the services running and moreqad just had me turn them all off?

Which should it be?

Carl.
0
hutnorCommented:
Good Luck - I will not be here to help in 3 hours time.

Just make sure you folow this http://support.microsoft.com/kb/258243 & have not stop any exchange services.
0
jctcomAuthor Commented:
Ok since that MS Article does not mention turning off any exchange services I think I will go with that.

Any quick commands to turn all those services back on?

I am going to guess at net start iisadmin /y /y

But I don't know if I need the "/y /y" at the end?

Carl.
0
hutnorCommented:
The KB from MS does not say to stop any services so If it was me doing it I would folow that.

I just tested on a exchange 2007 with info store service stop you can not mount mailbox. I assume it will be the same for 2003.

Provide that cmd did not change the services to manual start you can restart server & everything is back up. I do not know what the cmd did or how to use it.
0
jctcomAuthor Commented:
No.  It did not Change the services to manual startup.  In fact it looks like the services restarted on their own.
But I am going to restart again just in case.  Don't want to come back 3 hours from now to find out the restore did not work again.  Better to take the 8 minutes now to restart the server.

Carl.
0
jctcomAuthor Commented:
Thank you so much for your help.  I will start the restore as soon as the server finished restarting and then I am going to try to get a couple of hours of sleep while waiting for the restore to finish and come back and check it then.

Carl.
0
moregadCommented:
Good luck Carl,

last time i restored a single exchange with single information storage i stopped all client side services (which depends on iisadmin service, such as smtp, owa, etc) so it wouldn't clutter my event log with event id 9175 and simlar - but i left smtp running for receing incoming email.

cheers,
0
jctcomAuthor Commented:
Ok Restore is running.  I know you won't be here but I will check back in about 3 hours and post the results.

Thank you.

Carl.
0
hutnorCommented:
http://technet.microsoft.com/en-us/library/aa996953(EXCHG.65).aspx
http://www.mike-tech.com/article.php?gif=exchange2k3&article=304

If the restore doesn't work here is something you can look at doing. This is your other option.
0
jctcomAuthor Commented:
Damn restore finnished with the same errors.

Ok There is something else I don't get.  When I look at the "BackupAssist" backup logs they claim that the backup was successful:  here is an excerpt from the exchange component of the backup:

Backup Selection - Success
================
Computer:
Selection: KPH01\Microsoft Information Store\First Storage Group
Data type: ExchangeServer
Bytes: 21367989430
Files: 9
Directories: 4
Skipped: 0
Different: 0
--- Backup Log ---
Backup of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Media name: "Friday.bkf created 16/04/2010 at 9:11 PM"

Backup Type: Normal

Backup started on 17/04/2010 at 4:28 AM.
Backup completed on 17/04/2010 at 5:15 AM.
Directories: 4
Files: 9
Bytes: 21,367,989,430
Time:  47 minutes and  3 seconds

----------------------
--- Verify Log ---

Verify of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Verify started on 17/04/2010 at 5:15 AM.
Verify completed on 17/04/2010 at 5:27 AM.
Directories: 4
Files: 0
Different: 0
Bytes: 21,367,989,430
Time:  11 minutes and  8 seconds


Yet when I am going to do the restore it shows the "First Storage Group" as being 0 kb when I know that it is 21GB. I have attached a picture of the selection window.

Now it can't be 0 kb because even though the restore fails it still goes through 21GB of data before it fails.  So I don't get what's going on there.

Carl.
Restore-selections.jpg
0
jctcomAuthor Commented:
Ok I have copied back the files I copied before the first restore attempt.  I have run an eseutil /p which finished quite quickly withing a couple of minutes in fact.  I then ran an eseutil /D which took several hours.

I am now trying to run ISINTEG -S RHINO -FIX -TEST ALLTESTS  (With my server name in place of RHINO) but I am getting the following error message back:

Error: unable to get databases status from server. The reason could be either wr
ong server name or networking problems. Isinteg quits now.

I have not tried yet to mount the store.

Carl.
0
jctcomAuthor Commented:
Ok.  Managed to run the Isinteg on the original database without errors (But some warnings) and then afterwards Mount the store and verified with one of the workstations that had reported missing e-mail that they are now back  (Have not heard from anyone else in the office yet so I am hoping that it fixed the issue for everyone).

The problem I was having with running the Isinteg seemed to have to do with extra log files.  I found this reference to the error I was getting:
http://www.petri.co.il/mount_exchange_database_when_e00log_is_missing.htm and moved two of the log files out of the folder and the Isinteg ran successfully.

I am wondering if those extra log files could have prevented the restore from running successfully?
In any case I think this is better as I think they now have all their e-mail back instead of missing a day or two.

Thanks for everyone's help.

Carl.
0
jctcomAuthor Commented:
Anyone who reads this should read my additional comments near the end as to the final solution to this particular problem.

But I think everyone for pointing me in the right direction and for their accurate suggestions.

Carl
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.