jctcom
asked on
Problems restoring Exchange 2003 SBS Mailbox stores
Hello. I hope someone here can help me as I am in a bit over my head. I have a client with a stand alone SBS 2003 Server. A couple of days ago I started getting some reports of errors from the exchange mailboxes. I ran a ESEutil /g which finished with error "1206 JET_errDatabseCorrupt" during troubleshooting I at some point ran eseutil /P which claimed it ran find and solved the problem. I then also ran another eseutil /g which came up clean so I thought I was done and mounted the database.
The next morning when everyone came back into the office (There are about 18 computers running off this server) Some of the people were missing a lot of e-mail from 2010. We had a backup that was done on Friday the 16th using "Backup Assist" version 5..2.5 and I believe it is just an interface that runs NTBackup. The backup log showed that it had completed successfully and verify is turned on. so I proceeded to perform a restore of the information store only to be met after 3 hours with the following errors after which the store is no longer mountable now.
Here are the errors from the backup software and the event viewer:
Restore Status
Operation: Restore
Backup of "KPH01\Microsoft Information Store\First Storage Group", Restored to"KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Restore started on 20/04/2010 at 7:23 PM.
Unable to restore Exchange data to KPH01\Microsoft Information Store\First Storage Group,
check the application event log for more information
Restore completed on 20/04/2010 at 10:14 PM.
Directories: 2
Files: 2
Bytes: 21,319,733,610
Time: 2 hours, 50 minutes, and 41 seconds
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8019
Date: 20/04/2010
Time: 10:14:23 PM
User: N/A
Computer: KPH01
Description:
End Operation: Warnings or errors were encountered.
Consult the backup report for more details.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8003
Date: 20/04/2010
Time: 10:14:21 PM
User: N/A
Computer: KPH01
Description:
End Restore to 'KPH01\Microsoft Information Store\First Storage Group' 'Failed'
Verify: Off
Consult the backup report for more detail.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: ESE BACKUP
Event Category: Callback
Event ID: 904
Date: 20/04/2010
Time: 10:14:21 PM
User: N/A
Computer: KPH01
Description:
Information Store (5220) Callback function call ErrESECBRestoreGetDestinat
.
For more information, click http://www.microsoft.com/contentredirect.asp.
also throughout the course of the day and while running the backup the following error kept being logged about once a minute. I thought it was because the information store was offline but mabye there is something else to it.
Event Type: Error
Event Source: MSExchangeSA
Event Category: MAPI Session
Event ID: 9175
Date: 20/04/2010
Time: 10:13:18 PM
User: N/A
Computer: KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000
For more information, click http://www.microsoft.com/contentredirect.asp.
Hopefully someone here can help and is either up late or in a time zone that is earlier in the day than mine (Canada Pacific Time) as I need to get this back up and running by morning when everyone comes back to work.
also please note that I am working on the system remotely via LogMeIn (I also have remote desktop available to me if it makes a difference).
Thank you in advance.
Carl.
The next morning when everyone came back into the office (There are about 18 computers running off this server) Some of the people were missing a lot of e-mail from 2010. We had a backup that was done on Friday the 16th using "Backup Assist" version 5..2.5 and I believe it is just an interface that runs NTBackup. The backup log showed that it had completed successfully and verify is turned on. so I proceeded to perform a restore of the information store only to be met after 3 hours with the following errors after which the store is no longer mountable now.
Here are the errors from the backup software and the event viewer:
Restore Status
Operation: Restore
Backup of "KPH01\Microsoft Information Store\First Storage Group", Restored to"KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Restore started on 20/04/2010 at 7:23 PM.
Unable to restore Exchange data to KPH01\Microsoft Information Store\First Storage Group,
check the application event log for more information
Restore completed on 20/04/2010 at 10:14 PM.
Directories: 2
Files: 2
Bytes: 21,319,733,610
Time: 2 hours, 50 minutes, and 41 seconds
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8019
Date: 20/04/2010
Time: 10:14:23 PM
User: N/A
Computer: KPH01
Description:
End Operation: Warnings or errors were encountered.
Consult the backup report for more details.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8003
Date: 20/04/2010
Time: 10:14:21 PM
User: N/A
Computer: KPH01
Description:
End Restore to 'KPH01\Microsoft Information Store\First Storage Group' 'Failed'
Verify: Off
Consult the backup report for more detail.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: ESE BACKUP
Event Category: Callback
Event ID: 904
Date: 20/04/2010
Time: 10:14:21 PM
User: N/A
Computer: KPH01
Description:
Information Store (5220) Callback function call ErrESECBRestoreGetDestinat
.
For more information, click http://www.microsoft.com/contentredirect.asp.
also throughout the course of the day and while running the backup the following error kept being logged about once a minute. I thought it was because the information store was offline but mabye there is something else to it.
Event Type: Error
Event Source: MSExchangeSA
Event Category: MAPI Session
Event ID: 9175
Date: 20/04/2010
Time: 10:13:18 PM
User: N/A
Computer: KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000
For more information, click http://www.microsoft.com/contentredirect.asp.
Hopefully someone here can help and is either up late or in a time zone that is earlier in the day than mine (Canada Pacific Time) as I need to get this back up and running by morning when everyone comes back to work.
also please note that I am working on the system remotely via LogMeIn (I also have remote desktop available to me if it makes a difference).
Thank you in advance.
Carl.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
It depends on what you have to work with, but if you could go back to a copy of the database that you know works well enough and still has current enough email all the better. Otherwise you need to use the current database, and yes make backup copy of it.
ISINTEG command line syntax
http://support.microsoft.com/kb/301460
ISINTEG command line syntax
http://support.microsoft.com/kb/301460
I'm not sure if eseutil /p makes a backup copy of the database prior to the repair. I'm only aware of copy being made durring a defrag.
ASKER
I guess my real issue at this point is why can't I restore the backup I did from the 16th? By all accounts it was listed as a good and verified backup. Why all those errors?
I also tried copying the whole MDBDATA Folder to a temporary folder before I started the restore but it would not let me copy some of the files.
These are the files I did manage to copy:
E00.chk
E00010B1.log
E00010B2.log
priv1.edb
priv1.stm
priv1.INTEG.RAW
pub1.INTEG.RAW
res1.log
res2.log
I also have files from the following folders:
From the MDBDATA Logs Temp Folder I have a lot of .log files plus one "E00.chk" file
From the MDBDATA.temp Folder I have a number of files including a priv1.edb file but I don't think this can be complete as it is far too small only 5GB whereas the original is about 21GB
Am I better to try working with the current data which won't mount at all or can I somehow use these files that I copied before attempting the restore and get them to the point where they are mountable again?
I also tried copying the whole MDBDATA Folder to a temporary folder before I started the restore but it would not let me copy some of the files.
These are the files I did manage to copy:
E00.chk
E00010B1.log
E00010B2.log
priv1.edb
priv1.stm
priv1.INTEG.RAW
pub1.INTEG.RAW
res1.log
res2.log
I also have files from the following folders:
From the MDBDATA Logs Temp Folder I have a lot of .log files plus one "E00.chk" file
From the MDBDATA.temp Folder I have a number of files including a priv1.edb file but I don't think this can be complete as it is far too small only 5GB whereas the original is about 21GB
Am I better to try working with the current data which won't mount at all or can I somehow use these files that I copied before attempting the restore and get them to the point where they are mountable again?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Have you ticked the box on the exchange server that says "Allow this database to be overwritten by a restore" ??
Make sure you have the box ticked on all the stores you are going to restore.
Sorry moregad for the repeat. your comment was not there when i was posting.
Sorry moregad for the repeat. your comment was not there when i was posting.
ASKER
Ok. I did dismount the store (Actually I did not have to dismount it the second time as it would not mount in the first place), but I did not tick an option to allow the database to be overwritten. Please let this be the solution. I am going to try another restore right now with that option. Unfortunately it takes about 3 hours (Or it did the last times). Should I restart the server before attempting the restore again? (I did restart before the last attempt and no one has been in the office since).
Carl.
Carl.
ASKER
Is there a way for me to stop these
Event Type: Error
Event Source: MSExchangeSA
Event Category: MAPI Session
Event ID: 9175
Date: 20/04/2010
Time: 10:13:18 PM
User: N/A
Computer: KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000
During the restore? They are flooding the event viewer and I have a system set up to send me e-mails for errors in the event viewer. But more than that is it going to affect the restore operation at all?
Carl.
Event Type: Error
Event Source: MSExchangeSA
Event Category: MAPI Session
Event ID: 9175
Date: 20/04/2010
Time: 10:13:18 PM
User: N/A
Computer: KPH01
Description:
The MAPI call 'OpenMsgStore' failed with the following error:
The attempt to log on to the Microsoft Exchange Server computer has failed.
The MAPI provider failed.
Microsoft Exchange Server Information Store
ID no: 8004011d-0512-00000000
During the restore? They are flooding the event viewer and I have a system set up to send me e-mails for errors in the event viewer. But more than that is it going to affect the restore operation at all?
Carl.
stop the information store service
When your restore from backup is complete you will need to let exchange play out the transaction log files so the restored database will be up to date. Otherwise you will have missing emails.
ASKER
Ok the information Store service ("Microsoft Exchange Information Store" in services.msc) is stopped. So just waiting to see if you guys think a reboot is necessary or not before I start the restore operation again?
Carl.
Carl.
ASKER
@hutnor. Is there something I need to do to make "Exchange play out the transaction log files" or should it do that automatically after the restore? also how long does that take? for a priv1.edb that is about 21GB.
If you do want to restart have a look at the store & see if it was mount when the server started back up.
I do not see any problem restarting.
I do not see any problem restarting.
ASKER
Also In the restore dialogue box after selecting the source file there are two optional check boxes.
1. "Last Restore Set (Log File Replay with restart after this restore completes)" I am guessing this is the answer to my question above to hutnor. and it should be checked.
2. "Mount Database after restore." Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?
Carl.
1. "Last Restore Set (Log File Replay with restart after this restore completes)" I am guessing this is the answer to my question above to hutnor. and it should be checked.
2. "Mount Database after restore." Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?
Carl.
ASKER
The database definitely did not mount after a restart. But I think I will restart it now just to make sure it is a clean as possible for the restore.
Carl.
Carl.
ASKER
Hmmm Did I stop the wrong service or something. I just checked before restarting and I am still getting those Event ID 9175 errors every minute or so.
Carl.
Carl.
ASKER
I also shut down the "Microsoft Exchange IMAP4" service. I am thinking this might more likely be the cause of the Event ID 9175 errors no? There is no problem leaving this service stopped as well during the restore?
Carl.
Carl.
ASKER
Damn. I am still getting these Event ID 9175 errors every minute. any ideas on this. I would like to get rid of this before I restart (I know I will need to shut down the services again after restarting) and start the restore.
Carl.
Carl.
i will have a look at that service
It will pick it up automatic & replay them itself. Yes that is what you want that first tick box.
As for time I have no idea. Maybe someone else has a better idea.
Leave all your services running after the restart & do the restore. I would not worry about having extra errors in the event viewer.
http://support.microsoft.com/kb/258243
read over that before you do anything & follow it step by step.
As for time I have no idea. Maybe someone else has a better idea.
Leave all your services running after the restart & do the restore. I would not worry about having extra errors in the event viewer.
http://support.microsoft.com/kb/258243
read over that before you do anything & follow it step by step.
ASKER
Ok I found this link to info about that error:
http://support.microsoft.com/kb/262456
But since we don't want the Information Store running or database mounted how do i stop the "System Attendant" in the meantime?
also Still would like to get the answers to these two questions I posted above before starting the restore.
>>>>
Also In the restore dialogue box after selecting the source file there are two optional check boxes.
1. "Last Restore Set (Log File Replay with restart after this restore completes)" I am guessing this is the answer to my question above to hutnor. and it should be checked.
2. "Mount Database after restore." Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?
<<<<
http://support.microsoft.com/kb/262456
But since we don't want the Information Store running or database mounted how do i stop the "System Attendant" in the meantime?
also Still would like to get the answers to these two questions I posted above before starting the restore.
>>>>
Also In the restore dialogue box after selecting the source file there are two optional check boxes.
1. "Last Restore Set (Log File Replay with restart after this restore completes)" I am guessing this is the answer to my question above to hutnor. and it should be checked.
2. "Mount Database after restore." Will it even be able to mount the database without me turning back on the "Microsoft Exchange Information Store" ?
<<<<
ASKER
Ok have just restarted the server Checked that the database did not automount and the "allow restore to overwrite database" is still checked and stopped the Information store & IMAP4 services and I was going to stop the "System Attendant" service but it sounds like you are saying I should restart the services before I start the restore?
Carl.
Carl.
try run: net stop iisadmin /y /y
you don't want smpt running while you are restoring...
cheers,
you don't want smpt running while you are restoring...
cheers,
ASKER
and that last check box? "Mount Database after restore"? Checked or not checked?
Carl.
Carl.
Yes make sure all the services are running.
Information store & IMAP4 services & System Attendant
Information store & IMAP4 services & System Attendant
check it.
also after restore is finish have a look yourself!
also after restore is finish have a look yourself!
ASKER
@moreqad
I ran that command and all those services stopped successfully. I think I am ready to start the restore process now.
I'll wait 5 minutes for any last comments and then I am going to start the restore.
Carl.
I ran that command and all those services stopped successfully. I think I am ready to start the restore process now.
I'll wait 5 minutes for any last comments and then I am going to start the restore.
Carl.
ASKER
Ok wait a minute. We have conflicting info. hutnor says leave the services running and moreqad just had me turn them all off?
Which should it be?
Carl.
Which should it be?
Carl.
Good Luck - I will not be here to help in 3 hours time.
Just make sure you folow this http://support.microsoft.com/kb/258243 & have not stop any exchange services.
Just make sure you folow this http://support.microsoft.com/kb/258243 & have not stop any exchange services.
ASKER
Ok since that MS Article does not mention turning off any exchange services I think I will go with that.
Any quick commands to turn all those services back on?
I am going to guess at net start iisadmin /y /y
But I don't know if I need the "/y /y" at the end?
Carl.
Any quick commands to turn all those services back on?
I am going to guess at net start iisadmin /y /y
But I don't know if I need the "/y /y" at the end?
Carl.
The KB from MS does not say to stop any services so If it was me doing it I would folow that.
I just tested on a exchange 2007 with info store service stop you can not mount mailbox. I assume it will be the same for 2003.
Provide that cmd did not change the services to manual start you can restart server & everything is back up. I do not know what the cmd did or how to use it.
I just tested on a exchange 2007 with info store service stop you can not mount mailbox. I assume it will be the same for 2003.
Provide that cmd did not change the services to manual start you can restart server & everything is back up. I do not know what the cmd did or how to use it.
ASKER
No. It did not Change the services to manual startup. In fact it looks like the services restarted on their own.
But I am going to restart again just in case. Don't want to come back 3 hours from now to find out the restore did not work again. Better to take the 8 minutes now to restart the server.
Carl.
But I am going to restart again just in case. Don't want to come back 3 hours from now to find out the restore did not work again. Better to take the 8 minutes now to restart the server.
Carl.
ASKER
Thank you so much for your help. I will start the restore as soon as the server finished restarting and then I am going to try to get a couple of hours of sleep while waiting for the restore to finish and come back and check it then.
Carl.
Carl.
Good luck Carl,
last time i restored a single exchange with single information storage i stopped all client side services (which depends on iisadmin service, such as smtp, owa, etc) so it wouldn't clutter my event log with event id 9175 and simlar - but i left smtp running for receing incoming email.
cheers,
last time i restored a single exchange with single information storage i stopped all client side services (which depends on iisadmin service, such as smtp, owa, etc) so it wouldn't clutter my event log with event id 9175 and simlar - but i left smtp running for receing incoming email.
cheers,
ASKER
Ok Restore is running. I know you won't be here but I will check back in about 3 hours and post the results.
Thank you.
Carl.
Thank you.
Carl.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Damn restore finnished with the same errors.
Ok There is something else I don't get. When I look at the "BackupAssist" backup logs they claim that the backup was successful: here is an excerpt from the exchange component of the backup:
Backup Selection - Success
================
Computer:
Selection: KPH01\Microsoft Information Store\First Storage Group
Data type: ExchangeServer
Bytes: 21367989430
Files: 9
Directories: 4
Skipped: 0
Different: 0
--- Backup Log ---
Backup of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Media name: "Friday.bkf created 16/04/2010 at 9:11 PM"
Backup Type: Normal
Backup started on 17/04/2010 at 4:28 AM.
Backup completed on 17/04/2010 at 5:15 AM.
Directories: 4
Files: 9
Bytes: 21,367,989,430
Time: 47 minutes and 3 seconds
----------------------
--- Verify Log ---
Verify of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Verify started on 17/04/2010 at 5:15 AM.
Verify completed on 17/04/2010 at 5:27 AM.
Directories: 4
Files: 0
Different: 0
Bytes: 21,367,989,430
Time: 11 minutes and 8 seconds
Yet when I am going to do the restore it shows the "First Storage Group" as being 0 kb when I know that it is 21GB. I have attached a picture of the selection window.
Now it can't be 0 kb because even though the restore fails it still goes through 21GB of data before it fails. So I don't get what's going on there.
Carl.
Restore-selections.jpg
Ok There is something else I don't get. When I look at the "BackupAssist" backup logs they claim that the backup was successful: here is an excerpt from the exchange component of the backup:
Backup Selection - Success
================
Computer:
Selection: KPH01\Microsoft Information Store\First Storage Group
Data type: ExchangeServer
Bytes: 21367989430
Files: 9
Directories: 4
Skipped: 0
Different: 0
--- Backup Log ---
Backup of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Media name: "Friday.bkf created 16/04/2010 at 9:11 PM"
Backup Type: Normal
Backup started on 17/04/2010 at 4:28 AM.
Backup completed on 17/04/2010 at 5:15 AM.
Directories: 4
Files: 9
Bytes: 21,367,989,430
Time: 47 minutes and 3 seconds
----------------------
--- Verify Log ---
Verify of "KPH01\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 17/04/2010 at 4:28 AM"
Verify started on 17/04/2010 at 5:15 AM.
Verify completed on 17/04/2010 at 5:27 AM.
Directories: 4
Files: 0
Different: 0
Bytes: 21,367,989,430
Time: 11 minutes and 8 seconds
Yet when I am going to do the restore it shows the "First Storage Group" as being 0 kb when I know that it is 21GB. I have attached a picture of the selection window.
Now it can't be 0 kb because even though the restore fails it still goes through 21GB of data before it fails. So I don't get what's going on there.
Carl.
Restore-selections.jpg
ASKER
Ok I have copied back the files I copied before the first restore attempt. I have run an eseutil /p which finished quite quickly withing a couple of minutes in fact. I then ran an eseutil /D which took several hours.
I am now trying to run ISINTEG -S RHINO -FIX -TEST ALLTESTS (With my server name in place of RHINO) but I am getting the following error message back:
Error: unable to get databases status from server. The reason could be either wr
ong server name or networking problems. Isinteg quits now.
I have not tried yet to mount the store.
Carl.
I am now trying to run ISINTEG -S RHINO -FIX -TEST ALLTESTS (With my server name in place of RHINO) but I am getting the following error message back:
Error: unable to get databases status from server. The reason could be either wr
ong server name or networking problems. Isinteg quits now.
I have not tried yet to mount the store.
Carl.
ASKER
Ok. Managed to run the Isinteg on the original database without errors (But some warnings) and then afterwards Mount the store and verified with one of the workstations that had reported missing e-mail that they are now back (Have not heard from anyone else in the office yet so I am hoping that it fixed the issue for everyone).
The problem I was having with running the Isinteg seemed to have to do with extra log files. I found this reference to the error I was getting:
http://www.petri.co.il/mount_exchange_database_when_e00log_is_missing.htm and moved two of the log files out of the folder and the Isinteg ran successfully.
I am wondering if those extra log files could have prevented the restore from running successfully?
In any case I think this is better as I think they now have all their e-mail back instead of missing a day or two.
Thanks for everyone's help.
Carl.
The problem I was having with running the Isinteg seemed to have to do with extra log files. I found this reference to the error I was getting:
http://www.petri.co.il/mount_exchange_database_when_e00log_is_missing.htm and moved two of the log files out of the folder and the Isinteg ran successfully.
I am wondering if those extra log files could have prevented the restore from running successfully?
In any case I think this is better as I think they now have all their e-mail back instead of missing a day or two.
Thanks for everyone's help.
Carl.
ASKER
Anyone who reads this should read my additional comments near the end as to the final solution to this particular problem.
But I think everyone for pointing me in the right direction and for their accurate suggestions.
Carl
But I think everyone for pointing me in the right direction and for their accurate suggestions.
Carl
ASKER
also I have not used the isinteg command. Could you provide me with the syntax?
Thank you.