[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 998
  • Last Modified:

Exchange 2003 - Corrupted Database

I was called in to take a look at a down exchange server (2003, part of a small sbs 2003 installation).

There is a small number of users (<10) with a small exchange database (<4GB).

Today, the server went offline, froze, and hard to be hard reset. It came up fine, except the database for exchange would not mount.

I looked at the logs and before the freeze I see this:
Information Store (2240) First Storage Group: A request to write to the file "D:\exchange\Logs\E00tmp.log" at offset 0 (0x0000000000000000) for 1048576 (0x00100000) bytes succeeded, but took an abnormally long time (95 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

Open in new window


A few minutes later:
Information Store (2872) First Storage Group: Corruption was detected during soft recovery in logfile D:\exchange\Logs\E00.log. The failing checksum record is located at position END. Data not matching the log-file fill pattern first appeared in sector 517 (0x00000205). This logfile has been damaged and is unusable. 

Open in new window


Then the store went down and this even occurred:
Information Store (2872) First Storage Group: Database recovery/restore failed with unexpected error -501. 

Open in new window


Error Log file is corrupt starting Storage Group /DC=local/DC=domain/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=First Organization/CN=Administrative Groups/CN=first administrative group/CN=Servers/CN=SERVER/CN=InformationStore/CN=First Storage Group on the Microsoft Exchange Information Store. 
Storage Group - Initialization of Jet failed. 

Open in new window


They have no decent backup and there are a LOT of log files that have not been rolled into the database.

I just want to verify the order of steps I should be taking here. I have made a manual backup of the edb files and log files. Could someone list the steps I should take that will lead to the least amount of data loss?
0
tvacc
Asked:
tvacc
  • 18
  • 14
  • 2
  • +2
1 Solution
 
tvaccAuthor Commented:
I should mention that the location of the database files is d:\exchange and the log files are located in d:\exchange\logs
0
 
tvaccAuthor Commented:
eseutil /mh d:\exchange\priv1.edb results:

Initiating FILE DUMP mode...
         Database: d:\exchange\priv1.edb

        File Type: Database
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
     DB Signature: Create time:12/14/2004 14:59:33 Rand:191535 Computer:
         cbDbPage: 4096
           dbtime: 86824565 (0x52cd675)
            State: Dirty Shutdown
     Log Required: 14347-14439 (0x380b-0x3867)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 6659
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x380B,4CD,124)  10/24/2011 10:36:39
      Last Attach: (0x380B,4CE,195)  10/28/2011 01:19:04
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:12/14/2004 14:59:30 Rand:205684 Computer:
       OS Version: (5.2.3790 SP 2)

Open in new window

0
 
PostmasterCommented:
Check for free space where the logs write to.
If you can make some room and get the database up successfully, then close it down gracefully and do an off-line backup. (copy of database - don't need logs).

If this is done, you can then delete the log files and start exchange up for use.
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

 
tvaccAuthor Commented:
There is plenty of free space on the D: drive (where both the database files and the log files are stored). There is around 40GB free on this partition.
0
 
tvaccAuthor Commented:
I ran eseutil /r E00 /ld:\exchange\logs and then again  with the /i switch. Both times returned:

D:\exchange>"c:\program files\Exchsrvr\bin\eseutil.exe" /r E00 /i /ld:\exchange\
logs

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: E00
            Log files: d:\exchange\logs
         System files: <current directory>

Performing soft recovery...



Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access fi
le, the file is locked or in use) after 1822.62 seconds.

Open in new window

0
 
chakkoCommented:
is your information store service running?  try stopping that before using eseutil
0
 
skaraiCommented:
First of all make sure there are no physical problems surrounding the actual server hw - nothing is more disappointing than solving a database corruption problem only to fall right back into it. Check your Windows application and system logs for problems.
0
 
tvaccAuthor Commented:
No, the only services related to exchange that are running:

Exchange Management
Exchange imap4
Exchange routing engine
Exchange system attendant
0
 
tvaccAuthor Commented:
From the event log, related to the access denied error after running /r E00 /i /ld:\exchange\logs

eseutil (4912) An attempt to open the file "C:\Program Files\Exchsrvr\mdbdata\pub1.stm" for read / write access failed with system error 5 (0x00000005): "Access is denied. ".  The open file operation will fail with error -1032 (0xfffffbf8). 

Open in new window


Then:

eseutil (4912) Unable to write a shadowed header for file C:\Program Files\Exchsrvr\mdbdata\pub1.stm. Error -1032. 

Open in new window


Then:

eseutil (4912) Database recovery/restore failed with unexpected error -1032. 

Open in new window


eseutil (4912) The database engine stopped. 

Open in new window


I should note that while the priv1 files are in d:\exchange, the pub1 files are still in the default path since they are never used/grow in size in any significant way.
0
 
chakkoCommented:
If you have a lot and lot of log files, then I would move them (and the chk file) to a different folder.  Replaying them could take a long time.  Maybe some error will popup while copying a log file.  Could be a corrupt log file is 'locked' or not accessible?

I would guess that most of the log files have already been committed to the database, so moving them doesn't really hurt any, maybe some recent emails will get lost depending on when the database stopped.

Then try the command again.  You may need to do a repair with /p option, but I would try the /r first.

0
 
chakkoCommented:
use the /d and specify the database folder location.  also are you specifying the priv1.edb file in the command?  
0
 
tvaccAuthor Commented:
If I move the log files, what attributes do I need to specify for /r?
0
 
chakkoCommented:
just looked again.  you don't need priv1.edb in the /r syntax
0
 
chakkoCommented:
you have a full backup already, right?

so I would just delete them.  Then try the /p to repair the database.

One error already indicates a corrupt log file, so that could cause a problem if running recovery mode.  So I would just skip it and go straight to the repair.

Also, in your log files, is the filename sequence looking like it is from the very begining?  Or the file dates start from when exchange was installed?

If no backup has ever been does since exchange was installed.  Meaninng, you have a log file chain from the beginning of installation.  Then, exchange can probably rebuild your entire database from the log files (except of course if some are corrupted).

So, I would keep the full copy of the DB and LOGS in another folder if you need it later.
0
 
chakkoCommented:
Just to follow up.  This is a small office, do they care much if some recent email is lost during the repair?  if that is not a big issue, then I would skip to the /P repair process.

If it is really important to preserve all the email then you can try to work through the log files.

Or just go to the repair and if missing some email, you can use the backup copy and start again on the /R way.
0
 
chakkoCommented:
Lastly, when running the eseutil program.  Make sure the window has Focus - is the active window

If you click away from it, then make sure to reactivate it as the active window and also hit the enter key 1 or 2 times (some input/activity).

If that window loses focus then that eseutil sometimes goes extra slow for some reason.
0
 
tvaccAuthor Commented:
I suppose it would depend on how much email went missing :)

I think giving /p a try makes sense, especially with a backup.

If I wanted to try /r again, would I just run eseutil /r without anything else from the working directory with priv1.edb? (no E00, /i, /l)?

The log files go back about a year. The exchange server has been in use for 4-5 years. The /mh tells me that no backup has even been done, but I know that to be incorrect, so I'm not sure why that is reporting as such. I am looking at old backups right now, but they are obviously too old to restore from being over a year old.
0
 
tvaccAuthor Commented:
Also, thank you for the advice throughout this thread. It is very much appreciated. I just like to verify this kind of stuff with someone else in case I am overlooking something.
0
 
chakkoCommented:
I would make a copy of the copy and use that to work with for further recovery attempts - if that is clear.  Keep 1 orginal copy in case you need it later.

You would need to use /r E00 /l and /d switches. It can be in the new folder (working directory) location, just specify the required paths

I think email is normally committed to the database pretty fast - for such as small exchange as yours.  So I would guess probably no email loss, except if any email was coming in at the time that the database crashed.
0
 
tvaccAuthor Commented:
I guess I'm a a little confused now. I thought you said that I should get rid of everything in the /log directory and then run /r

Is this not the case?

If that is the case, then what would be the point of the /l switch if there is nothing in the log directory?
0
 
chakkoCommented:
sorry,

I would either proceed with a repair /P or try the recovery /R

/P - get rid of log files and the chk file

if you want to try the recovery (to process the log files) then the /R way.  The way you tried first (but got the file access error).

if the results of /P repair are not good, then you can use the backup copy folder that you made to try the whole process again using Recovery first /R
0
 
tvaccAuthor Commented:
I went with the /p option. We'll see how it goes.

Should I run isinteg afterwards? What switches?
0
 
tvaccAuthor Commented:
eseutil /p d:\exchange\priv1.edb completed sucessfully

isinteg -fix next? Should I do another backup of the priv1 files before  running this command?
0
 
chakkoCommented:
try the /mh again and check state.

isinteg -fix    (I think that's the one)
0
 
chakkoCommented:
no need to backup.

try to start information store and mount the database.

if any error, then maybe need to run isinteg
0
 
tvaccAuthor Commented:
It mounted! Thanks again. Everythingi is current as well, as far as I can tell. Email right up to the outage.

pub1.edb is still inconsistent, so I'll have to work on that, but I'm not too concerned.

Again, thanks for the advice.
0
 
chakkoCommented:
good news.

basically same procedure for public folders.
0
 
tvaccAuthor Commented:
I don't know why, but I'm back to having the same problem. It only stayed mounted for about 15 minutes before the server froze again. I had to have it restarted again this morning and it worked th entire day without the exchange store mounted. I copied my original files back, ran /p again and restarted.

I can't tell if it mounted or not. I right click the mailbox store to see if it is mounted and it freezes the entire computer.

Any ideas?
0
 
tvaccAuthor Commented:
The private store (in d:\exchange) is in a clean shutdown state.
The public store (in c:\program files\exchsrvr\mdbdata) is in a dirty shutdown state.

When I try to run eseutil /p on the pub1.edb file I get an access denied error:
C:\Program Files\Exchsrvr\bin>eseutil.exe /p "c:\program files\exchsrvr\mdbdata\
pub1.edb"
Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating REPAIR mode...
        Database: c:\program files\exchsrvr\mdbdata\pub1.edb
  Streaming File: c:\program files\exchsrvr\mdbdata\pub1.STM
  Temp. Database: TEMPREPAIR3912.EDB
Checking database integrity.
The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.
To ensure the database is up-to-date please use the 'Recovery' operation.
Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access fi
le, the file is locked or in use) after 16.78 seconds.

Open in new window


Is it possible that the private mailbox store is being corrupted somehow by the public store?
0
 
tvaccAuthor Commented:
I ran eseutil /d on priv1.edb with no errors.
I can't run isinteg -fix -s server -test alltests on the priv1.edb file. It says it cannot be run.

I get this error when trying to mount the private store:
mount error
eseutil /g integrity check passes on priv1.edb
0
 
Satya PathakLead Technical ConsultantCommented:
I would say try to reboot your exchange server and than try if you are not able to mount the database,
Please ran eseutil /d once defrag will be completed also ran isinteg..

please read the Kb before perform the activity..

http://support.microsoft.com/kb/259851
0
 
chakkoCommented:
I would do a chkdsk on your server drives.  Stop all exchange services first and also make a backup first.

Are your public folders used a lot  (frequently)?  Did you also remove log files related to the public folders database before doing the /P repair?
0
 
tvaccAuthor Commented:
I started from the beginning with my original priv1.edb/stm files. I ran eseutil /p, eseutil /d, isinteg -fix -s server -test alltests (in that order) and the database mounted and has stayed mounted for about 15 hours now. I took a backup and I believe we are good.

I performed a chkdsk on both drives yesterday.

I still have no luck with the public store. We do not use public folders at all. Is there any way to go about deleting it and recreating it?
0
 
chakkoCommented:
Try moving the pub1.edb and stm files out of the folder (like delete them or move them).

Then restart the Information Store service.
0
 
tvaccAuthor Commented:
To anyone that is looking at this at a later point for their own exchange issue:

1. Back up your store before you do anything. It took me three times (starting from the backup each time) to finally get my store mounted.
2. I ran eseutil /p, /d, then isinteg -fix -s server -test alltests (in that order) on my private store and that finally got me up and running.
3. My disk seems to have some bad sectors, which caused the problem to begin with.
0
 
skaraiCommented:
After all this hard work you may now realize the suggestion I made early on to check your hardware first was right on target. All aspects of the problem pointed towards hardware failure and you should never repair a data base (e.g. Exchange or SQL) before correcting the underlying hw problem - else you may be fixing corruption over and over again.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

  • 18
  • 14
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now