tvacc
asked on
Exchange 2003 - Corrupted Database
I was called in to take a look at a down exchange server (2003, part of a small sbs 2003 installation).
There is a small number of users (<10) with a small exchange database (<4GB).
Today, the server went offline, froze, and hard to be hard reset. It came up fine, except the database for exchange would not mount.
I looked at the logs and before the freeze I see this:
A few minutes later:
Then the store went down and this even occurred:
They have no decent backup and there are a LOT of log files that have not been rolled into the database.
I just want to verify the order of steps I should be taking here. I have made a manual backup of the edb files and log files. Could someone list the steps I should take that will lead to the least amount of data loss?
There is a small number of users (<10) with a small exchange database (<4GB).
Today, the server went offline, froze, and hard to be hard reset. It came up fine, except the database for exchange would not mount.
I looked at the logs and before the freeze I see this:
Information Store (2240) First Storage Group: A request to write to the file "D:\exchange\Logs\E00tmp.log" at offset 0 (0x0000000000000000) for 1048576 (0x00100000) bytes succeeded, but took an abnormally long time (95 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. A few minutes later:
Information Store (2872) First Storage Group: Corruption was detected during soft recovery in logfile D:\exchange\Logs\E00.log. The failing checksum record is located at position END. Data not matching the log-file fill pattern first appeared in sector 517 (0x00000205). This logfile has been damaged and is unusable. Then the store went down and this even occurred:
Information Store (2872) First Storage Group: Database recovery/restore failed with unexpected error -501. Error Log file is corrupt starting Storage Group /DC=local/DC=domain/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=First Organization/CN=Administrative Groups/CN=first administrative group/CN=Servers/CN=SERVER/CN=InformationStore/CN=First Storage Group on the Microsoft Exchange Information Store.
Storage Group - Initialization of Jet failed. They have no decent backup and there are a LOT of log files that have not been rolled into the database.
I just want to verify the order of steps I should be taking here. I have made a manual backup of the edb files and log files. Could someone list the steps I should take that will lead to the least amount of data loss?
ASKER
eseutil /mh d:\exchange\priv1.edb results:
Initiating FILE DUMP mode...
Database: d:\exchange\priv1.edb
File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,11
Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
DB Signature: Create time:12/14/2004 14:59:33 Rand:191535 Computer:
cbDbPage: 4096
dbtime: 86824565 (0x52cd675)
State: Dirty Shutdown
Log Required: 14347-14439 (0x380b-0x3867)
Streaming File: Yes
Shadowed: Yes
Last Objid: 6659
Scrub Dbtime: 0 (0x0)
Scrub Date: 00/00/1900 00:00:00
Repair Count: 0
Repair Date: 00/00/1900 00:00:00
Old Repair Count: 0
Last Consistent: (0x380B,4CD,124) 10/24/2011 10:36:39
Last Attach: (0x380B,4CE,195) 10/28/2011 01:19:04
Last Detach: (0x0,0,0) 00/00/1900 00:00:00
Dbid: 1
Log Signature: Create time:12/14/2004 14:59:30 Rand:205684 Computer:
OS Version: (5.2.3790 SP 2)
Check for free space where the logs write to.
If you can make some room and get the database up successfully, then close it down gracefully and do an off-line backup. (copy of database - don't need logs).
If this is done, you can then delete the log files and start exchange up for use.
If you can make some room and get the database up successfully, then close it down gracefully and do an off-line backup. (copy of database - don't need logs).
If this is done, you can then delete the log files and start exchange up for use.
ASKER
There is plenty of free space on the D: drive (where both the database files and the log files are stored). There is around 40GB free on this partition.
ASKER
I ran eseutil /r E00 /ld:\exchange\logs and then again with the /i switch. Both times returned:
D:\exchange>"c:\program files\Exchsrvr\bin\eseutil.exe" /r E00 /i /ld:\exchange\
logs
Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating RECOVERY mode...
Logfile base name: E00
Log files: d:\exchange\logs
System files: <current directory>
Performing soft recovery...
Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access fi
le, the file is locked or in use) after 1822.62 seconds.
is your information store service running? try stopping that before using eseutil
First of all make sure there are no physical problems surrounding the actual server hw - nothing is more disappointing than solving a database corruption problem only to fall right back into it. Check your Windows application and system logs for problems.
ASKER
No, the only services related to exchange that are running:
Exchange Management
Exchange imap4
Exchange routing engine
Exchange system attendant
Exchange Management
Exchange imap4
Exchange routing engine
Exchange system attendant
ASKER
From the event log, related to the access denied error after running /r E00 /i /ld:\exchange\logs
Then:
Then:
I should note that while the priv1 files are in d:\exchange, the pub1 files are still in the default path since they are never used/grow in size in any significant way.
eseutil (4912) An attempt to open the file "C:\Program Files\Exchsrvr\mdbdata\pub1.stm" for read / write access failed with system error 5 (0x00000005): "Access is denied. ". The open file operation will fail with error -1032 (0xfffffbf8). Then:
eseutil (4912) Unable to write a shadowed header for file C:\Program Files\Exchsrvr\mdbdata\pub1.stm. Error -1032. Then:
eseutil (4912) Database recovery/restore failed with unexpected error -1032. eseutil (4912) The database engine stopped. I should note that while the priv1 files are in d:\exchange, the pub1 files are still in the default path since they are never used/grow in size in any significant way.
If you have a lot and lot of log files, then I would move them (and the chk file) to a different folder. Replaying them could take a long time. Maybe some error will popup while copying a log file. Could be a corrupt log file is 'locked' or not accessible?
I would guess that most of the log files have already been committed to the database, so moving them doesn't really hurt any, maybe some recent emails will get lost depending on when the database stopped.
Then try the command again. You may need to do a repair with /p option, but I would try the /r first.
I would guess that most of the log files have already been committed to the database, so moving them doesn't really hurt any, maybe some recent emails will get lost depending on when the database stopped.
Then try the command again. You may need to do a repair with /p option, but I would try the /r first.
use the /d and specify the database folder location. also are you specifying the priv1.edb file in the command?
ASKER
If I move the log files, what attributes do I need to specify for /r?
just looked again. you don't need priv1.edb in the /r syntax
you have a full backup already, right?
so I would just delete them. Then try the /p to repair the database.
One error already indicates a corrupt log file, so that could cause a problem if running recovery mode. So I would just skip it and go straight to the repair.
Also, in your log files, is the filename sequence looking like it is from the very begining? Or the file dates start from when exchange was installed?
If no backup has ever been does since exchange was installed. Meaninng, you have a log file chain from the beginning of installation. Then, exchange can probably rebuild your entire database from the log files (except of course if some are corrupted).
So, I would keep the full copy of the DB and LOGS in another folder if you need it later.
so I would just delete them. Then try the /p to repair the database.
One error already indicates a corrupt log file, so that could cause a problem if running recovery mode. So I would just skip it and go straight to the repair.
Also, in your log files, is the filename sequence looking like it is from the very begining? Or the file dates start from when exchange was installed?
If no backup has ever been does since exchange was installed. Meaninng, you have a log file chain from the beginning of installation. Then, exchange can probably rebuild your entire database from the log files (except of course if some are corrupted).
So, I would keep the full copy of the DB and LOGS in another folder if you need it later.
Just to follow up. This is a small office, do they care much if some recent email is lost during the repair? if that is not a big issue, then I would skip to the /P repair process.
If it is really important to preserve all the email then you can try to work through the log files.
Or just go to the repair and if missing some email, you can use the backup copy and start again on the /R way.
If it is really important to preserve all the email then you can try to work through the log files.
Or just go to the repair and if missing some email, you can use the backup copy and start again on the /R way.
Lastly, when running the eseutil program. Make sure the window has Focus - is the active window
If you click away from it, then make sure to reactivate it as the active window and also hit the enter key 1 or 2 times (some input/activity).
If that window loses focus then that eseutil sometimes goes extra slow for some reason.
If you click away from it, then make sure to reactivate it as the active window and also hit the enter key 1 or 2 times (some input/activity).
If that window loses focus then that eseutil sometimes goes extra slow for some reason.
ASKER
I suppose it would depend on how much email went missing :)
I think giving /p a try makes sense, especially with a backup.
If I wanted to try /r again, would I just run eseutil /r without anything else from the working directory with priv1.edb? (no E00, /i, /l)?
The log files go back about a year. The exchange server has been in use for 4-5 years. The /mh tells me that no backup has even been done, but I know that to be incorrect, so I'm not sure why that is reporting as such. I am looking at old backups right now, but they are obviously too old to restore from being over a year old.
I think giving /p a try makes sense, especially with a backup.
If I wanted to try /r again, would I just run eseutil /r without anything else from the working directory with priv1.edb? (no E00, /i, /l)?
The log files go back about a year. The exchange server has been in use for 4-5 years. The /mh tells me that no backup has even been done, but I know that to be incorrect, so I'm not sure why that is reporting as such. I am looking at old backups right now, but they are obviously too old to restore from being over a year old.
ASKER
Also, thank you for the advice throughout this thread. It is very much appreciated. I just like to verify this kind of stuff with someone else in case I am overlooking something.
I would make a copy of the copy and use that to work with for further recovery attempts - if that is clear. Keep 1 orginal copy in case you need it later.
You would need to use /r E00 /l and /d switches. It can be in the new folder (working directory) location, just specify the required paths
I think email is normally committed to the database pretty fast - for such as small exchange as yours. So I would guess probably no email loss, except if any email was coming in at the time that the database crashed.
You would need to use /r E00 /l and /d switches. It can be in the new folder (working directory) location, just specify the required paths
I think email is normally committed to the database pretty fast - for such as small exchange as yours. So I would guess probably no email loss, except if any email was coming in at the time that the database crashed.
ASKER
I guess I'm a a little confused now. I thought you said that I should get rid of everything in the /log directory and then run /r
Is this not the case?
If that is the case, then what would be the point of the /l switch if there is nothing in the log directory?
Is this not the case?
If that is the case, then what would be the point of the /l switch if there is nothing in the log directory?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I went with the /p option. We'll see how it goes.
Should I run isinteg afterwards? What switches?
Should I run isinteg afterwards? What switches?
ASKER
eseutil /p d:\exchange\priv1.edb completed sucessfully
isinteg -fix next? Should I do another backup of the priv1 files before running this command?
isinteg -fix next? Should I do another backup of the priv1 files before running this command?
try the /mh again and check state.
isinteg -fix (I think that's the one)
isinteg -fix (I think that's the one)
no need to backup.
try to start information store and mount the database.
if any error, then maybe need to run isinteg
try to start information store and mount the database.
if any error, then maybe need to run isinteg
ASKER
It mounted! Thanks again. Everythingi is current as well, as far as I can tell. Email right up to the outage.
pub1.edb is still inconsistent, so I'll have to work on that, but I'm not too concerned.
Again, thanks for the advice.
pub1.edb is still inconsistent, so I'll have to work on that, but I'm not too concerned.
Again, thanks for the advice.
good news.
basically same procedure for public folders.
basically same procedure for public folders.
ASKER
I don't know why, but I'm back to having the same problem. It only stayed mounted for about 15 minutes before the server froze again. I had to have it restarted again this morning and it worked th entire day without the exchange store mounted. I copied my original files back, ran /p again and restarted.
I can't tell if it mounted or not. I right click the mailbox store to see if it is mounted and it freezes the entire computer.
Any ideas?
I can't tell if it mounted or not. I right click the mailbox store to see if it is mounted and it freezes the entire computer.
Any ideas?
ASKER
The private store (in d:\exchange) is in a clean shutdown state.
The public store (in c:\program files\exchsrvr\mdbdata) is in a dirty shutdown state.
When I try to run eseutil /p on the pub1.edb file I get an access denied error:
Is it possible that the private mailbox store is being corrupted somehow by the public store?
The public store (in c:\program files\exchsrvr\mdbdata) is in a dirty shutdown state.
When I try to run eseutil /p on the pub1.edb file I get an access denied error:
C:\Program Files\Exchsrvr\bin>eseutil.exe /p "c:\program files\exchsrvr\mdbdata\
pub1.edb"
Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating REPAIR mode...
Database: c:\program files\exchsrvr\mdbdata\pub1.edb
Streaming File: c:\program files\exchsrvr\mdbdata\pub1.STM
Temp. Database: TEMPREPAIR3912.EDB
Checking database integrity.
The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.
To ensure the database is up-to-date please use the 'Recovery' operation.
Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access fi
le, the file is locked or in use) after 16.78 seconds.Is it possible that the private mailbox store is being corrupted somehow by the public store?
ASKER
I ran eseutil /d on priv1.edb with no errors.
I can't run isinteg -fix -s server -test alltests on the priv1.edb file. It says it cannot be run.
I get this error when trying to mount the private store:
eseutil /g integrity check passes on priv1.edb
I can't run isinteg -fix -s server -test alltests on the priv1.edb file. It says it cannot be run.
I get this error when trying to mount the private store:
eseutil /g integrity check passes on priv1.edb
I would say try to reboot your exchange server and than try if you are not able to mount the database,
Please ran eseutil /d once defrag will be completed also ran isinteg..
please read the Kb before perform the activity..
http://support.microsoft.com/kb/259851
Please ran eseutil /d once defrag will be completed also ran isinteg..
please read the Kb before perform the activity..
http://support.microsoft.com/kb/259851
I would do a chkdsk on your server drives. Stop all exchange services first and also make a backup first.
Are your public folders used a lot (frequently)? Did you also remove log files related to the public folders database before doing the /P repair?
Are your public folders used a lot (frequently)? Did you also remove log files related to the public folders database before doing the /P repair?
ASKER
I started from the beginning with my original priv1.edb/stm files. I ran eseutil /p, eseutil /d, isinteg -fix -s server -test alltests (in that order) and the database mounted and has stayed mounted for about 15 hours now. I took a backup and I believe we are good.
I performed a chkdsk on both drives yesterday.
I still have no luck with the public store. We do not use public folders at all. Is there any way to go about deleting it and recreating it?
I performed a chkdsk on both drives yesterday.
I still have no luck with the public store. We do not use public folders at all. Is there any way to go about deleting it and recreating it?
Try moving the pub1.edb and stm files out of the folder (like delete them or move them).
Then restart the Information Store service.
Then restart the Information Store service.
ASKER
To anyone that is looking at this at a later point for their own exchange issue:
1. Back up your store before you do anything. It took me three times (starting from the backup each time) to finally get my store mounted.
2. I ran eseutil /p, /d, then isinteg -fix -s server -test alltests (in that order) on my private store and that finally got me up and running.
3. My disk seems to have some bad sectors, which caused the problem to begin with.
1. Back up your store before you do anything. It took me three times (starting from the backup each time) to finally get my store mounted.
2. I ran eseutil /p, /d, then isinteg -fix -s server -test alltests (in that order) on my private store and that finally got me up and running.
3. My disk seems to have some bad sectors, which caused the problem to begin with.
After all this hard work you may now realize the suggestion I made early on to check your hardware first was right on target. All aspects of the problem pointed towards hardware failure and you should never repair a data base (e.g. Exchange or SQL) before correcting the underlying hw problem - else you may be fixing corruption over and over again.
ASKER