Solved

Emergancy!  Cannot mount mailbox store!

Posted on 2004-08-26
35
7,698 Views
Last Modified: 2012-05-05
Running Exchange 2003 and Windows Server 2003 SBS,  with NO backup available.

Due to a power failure our Exchange mailbox store and public folder store cannot be mounted.  

We are receiving the error:

"An internal processing error has occurred.  Try restarting the Exchange System Manager or the Microsoft Exchange Information Store, or both.

ID no: c1041724
Exchange System Manager"

I have tried using eseutil but no luck.

We have been recommended to backup all the .edb, .pub and .stm files to a different location,  UNINSTALL exchange,  
then re-install exchange,   and copy back the files.   Is this going to work?   Is this going to kill Active Directory?

It would be a massive problem if something went wrong, unimaginable infact!

Could anyone suggest a list of things I can try instead of this?   And if it is the only way of getting Exchange running again,  what are the risks?   This question is worth way more than 500, if its possible to give more I will.
0
Comment
Question by:Powerhousecomputing
  • 15
  • 10
  • 9
  • +1
35 Comments
 

Author Comment

by:Powerhousecomputing
ID: 11903522
Update:  

eseutil /p  "C:\Program Files\Exchsrvr\MDBDATA\priv1.edb"

returns the following:

"The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database."

In the MDBDATA folder there are four 5MB log files called:

E00.log
E00000FA.log
RS1.log
RS2.log


Is this significant?
0
 
LVL 3

Expert Comment

by:Dejan Foro
ID: 11903590

You probably forgot to delete log files after eseutil /P

For detail see:

Microsoft Knowledge Base Article - 822449: Error c1041724 Occurs When You Try to Mount a Database on an Exchange 2003 Server
http://support.microsoft.com/default.aspx?scid=kb;en-us;822449

Dejan Foro
www.exchangemaster.net
www.eugeurope.org
0
 

Author Comment

by:Powerhousecomputing
ID: 11903696
eseutil /mh "C:\program files\exchsrvr\mdbdata\pub1.edb"

returns a repair count = 0

A Dirty Shutdown
0
 

Author Comment

by:Powerhousecomputing
ID: 11903727
eseutil /p  

returns

"The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database."

A dialog box saying "You should only run Repair on damaged or corrupted databases.  Repair will not  apply information in the log files to the database and may cause information to be lost.  Do you want to proceed?"

I have always cancelled so far.   If I say YES,  will I *only* lose the info in the uncommitted log files?   Will everything in the database already remain intact?
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11903770

- First of all - you have verified you have enough disk space correct?
- This issue also occurs if the E00.log file is missing or is no longer present in the Exchsrvr\Mdbdata folder or in the log file folder.
- Do you have a recent good online or offline backup of the server?

Read this article - http://support.microsoft.com/default.aspx?kbid=284205
**Read the repair count attribute.
If the repair count attribute is 0 (zero), the eseutil /p command-line command was not run.
If the repair count attribute is a number other than 0, the eseutil /p command-line command was run on the database.

Another article - http://support.microsoft.com/default.aspx?scid=kb;en-us;822449&Product=exch2003

0
 

Author Comment

by:Powerhousecomputing
ID: 11903880
>- First of all - you have verified you have enough disk space correct?

120GB free space


>- This issue also occurs if the E00.log file is missing or is no longer present in the >Exchsrvr\Mdbdata folder or in the log file folder.

This log file does exist in that location.


>-- Do you have a recent good online or offline backup of the server?

None whatsoever!  With the mailbox store un-mounted, Backup Exec wont touch it - is there another way?   I have however copied the whole exchsrvr folder to another drive.


I haven't gone ahead with eseutil /p  yet, see my post above for what it asks me.  The checks in those links you posted tell me that the repair count is zero and it was a dirty shutdown.
0
 
LVL 12

Assisted Solution

by:BNettles73
BNettles73 earned 200 total points
ID: 11904106
First thing you need to do is -

Copy your priv1 file, strm file and log files to a safe location prior to running any utilities. This way if you run the utility and notice you have lost a considerable amount of data or it doesn't work - you'll still have the original files.

Make sure you have the database files (.edb and .stm) and the logs!!! If you are unsure about which files you may want to copy the whole exchsrvr directory.

Read through this article ...

The two most powerful tools for disaster recovery with Exchange databases are ESEUTIL and ISINTEG, two command-line tools that come with Exchange. Any Exchange administrator worth his salt should know how they work. They can save your skin if you have no backups, or if your backups are not recent enough to replace everything with. However, they are also powerful enough that if you don't know what you're doing with them, you can only make the damage worse.

The Exchange database (the .EDB file) is stored in a series of 4K segments called "pages," each of which has a checksum to guard against data corruption. ESEUTIL examines the database page-by-page, looking for checksum problems, but doesn't actually inspect the underlying data (such as the mail stored in the database). ISINTEG attempts to recover the actual mail data from the database (what are called "application-level" errors) after you've used ESEUTIL to repair its underlying structure.

The first kind of checks that should be run with ESEUTIL are the /G, /M and /K checks, so named for the switches used to invoke them in ESEUTIL. /G checks the integrity of the database file to determine if there are physical problems. /M (usually invoked as /MH) prints detailed diagnostic information about a database, including the last time an incremental or full backup was made. /M can also be used to determine if the data in a damaged page isn't worth recovering—for instance, if that page holds mail for a user with a long-closed account, or someone's deleted items. (Microsoft Knowledge Base article 262196 talks about how to do this.) The /K switch verifies the checksum of each database page, and logs an event whenever a page with an invalid checksum turns up. These events can be used to determine which pages may be lost in a recovery operation.

Once you've run these non-destructive scans and determined what is or isn't damaged and what can or cannot be repaired, then you can consider using ESEUTIL to do actual work. Before you do this, make a copy of the database and work on the copy if you can—you should never do potentially destructive work on an original. If you suspect a hardware problem, move the database to a completely different computer with a copy of Exchange present and work on it there, or look at Microsoft Knowledge Base article 244525 for instructions on how to use ESEUTIL on a computer without Exchange installed.

The /R switch in ESEUTIL goes through the database's transaction logs and rolls forward any as-yet-uncommitted transactions, but does not repair or touch damaged pages. This is useful when a check of the database structure shows no problems, but there are still uncommitted transactions in the logs. Microsoft Knowledge Base article 259751 tells you how to determine if there are uncommitted transactions pending; it involves running ESEUTIL /M /K on the transaction log files. When the power goes out abruptly, this will be the most common problem found, and it's not hard to recover from.

ESEUTIL's /P switch repairs corrupt database pages, but doesn't commit unfinished transactions. You should find out if any transactions are pending (and commit them, if need be) before you run /P, and you should also find out if there are damaged pages which do not have anything crucial in them. If so, running /P will cause that data to be lost as a way to restoring the whole database's integrity.

After you have finished with ESEUTIL, put the database back into its original location (if you were working on a copy) and run ISINTEG with the switches –fix –test alltests. ISINTEG works through the Information Store process to rebuild the structures stored inside the database that Exchange sees -- messages, mailboxes and so on. ISINTEG generates very verbose warnings, but don't panic -- what matters is the error count at the end. If ISINTEG continually produces errors you can't resolve, it's best to create a new database and move what you can into there manually instead of relying on a possibly broken data structure.

0
 

Author Comment

by:Powerhousecomputing
ID: 11904193
Ok I have read that.

I have a copy of the whole Exchsrvr folder,  will all logs, .edb and .stm files on a different drive.  

Shall I say YES to repairing the database and then run isinteg?  As it says in the article you posted, I can always copy back my backup copies of the files.
0
 
LVL 8

Accepted Solution

by:
ITCEXCHANGE earned 300 total points
ID: 11904195
For this error there is no need to reinstall exchange or anything like that.

In the Mdbdata directory verify that e00.chk e00.log exist. If they do you should first attemp a soft recovery.

Go into the mdbdata directory at a command prompt and type Eseutil /r e00

If you receive a message saying "the command completed succesfully" check the database header of the priv1.edb using the eseutil /mh switch.

If your file is now in a "clean shutdown state" you can mount your databases.

If your file is in a "dirty shutdown state", and you have no backups then you will have to run a repair on the file to get it back to a clean shutdown state.

To run a repair  first remove all the log files in the mdbdata directory, including the e00.chk file.

Then run eseutil /p against the priv1.edb and then the pub1.edb file.

You must also in Exchange System manager, click onto the properties of the Mailbox store, and public store and then under database tab, check the box that says " allow overwrite by restore".

After the repairs complete go into Exchange System manager and mount your database. Also your Exchange Information Store service maybe stopped, this is caused from corrupt log files, so you may first have to start this service then mount the database if not already mounted.

PS::::::: If during the eseutil /r command you get an "invalid API error" message then you need to skip to removing the log files and then running the repair option.  

Also you should run eseutil /D after the repair to remove any unuseable Pages that may come up from running the repair.  

ITCEXCHANGE
0
 

Author Comment

by:Powerhousecomputing
ID: 11904231
Ok ITCEXCHANGE  I shall give what you have suggested in your post a go!

I shall post back with what happens! :)
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11904242
Ok, yes please let me know :)

0
 

Author Comment

by:Powerhousecomputing
ID: 11905069
It is a dirty shutdown so I did the following:

- Removed all log files from MDBDATA folder
- Ran eseutil /p for priv1.edb and pub1.edb  (Only pub1 completed successfully)
- Allow overwrite was already set for both
- The public folder store now mounts succesfully
- The mailbox store does not mount,  but gives a different error now


The mailbox store gives the error "The database files in this store are inconsistent.  ID no: c1041739"

Running eseutil /mh on priv1.edb returns:

---------
Initiating FILE DUMP mode...
Database: C:\Program Files\Exchsrvr\MDBDATA\priv1.edb

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
     DB Signature: Create time:06/23/2004 15:20:38 Rand:2879797 Computer:
         cbDbPage: 4096
           dbtime: 4001794 (0x3d1002)
            State: Dirty Shutdown
     Log Required: 251-251 (0xfb-0xfb)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 3660
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
  Last Consistent: (0xF7,B1C,1C9)  08/24/2004 08:49:39
      Last Attach: (0xF7,B24,E2)  08/24/2004 08:54:19
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:06/28/2004 16:06:01 Rand:884409 Computer:
       OS Version: (5.2.3790 SP 0)

Previous Full Backup:
        Log Gen: 247-247 (0xf7-0xf7)
           Mark: (0xF7,1A97,1FB)
           Mark: 08/24/2004 21:50:59

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

Operation completed successfully in 3.31 seconds.

--------



Have we made any progress?  I'm hoping so as the mailbox store has a new error!
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905104
What message did you get when you tried to run the repair on the priv1.edb file?
0
 

Author Comment

by:Powerhousecomputing
ID: 11905119
Repair produces the following message:

The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.

To ensure the database is up-to-date please use the 'Recovery' operation.

Operation terminated with error -2204 (JET_errSLVStreamingFileMissing, Cannot find
streaming file associated with this database) after 7.63 seconds.
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905178
EWWWWWWWW, well, if you are missing your stream file you will be missing all the information in HTML format. But..... anything in rich text format and all your attachments should be there. Hopefully you backed up your database already, but if not backup your priv1.edb file.

Then dismount the public information store, remove the log files and check files again. Run eseutil/p priv1.edb /createstm.

This will repair your database and create a blank stream file so that you can mount your database. After the repair runs you will be able to mount both stores.

ITCEXCHANGE
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11905183
before running eseutil /p you should have ran ESEUTIL with the /M /K switches to check for pages which were not committed to the database. Initially you needed to run /r against the corrupt log file ... if you ran the /p switch you probably will need to run isinteg –fix –test alltests afterwards to mount the database ...

I'd run isinteg –fix –test alltests if you've run the /p switch and try to mount the database ... you still have the copy of all your originals right?
0
 

Author Comment

by:Powerhousecomputing
ID: 11905348


It worked!!!   Mailbox Store and the Public Folder store are working!  Phew that was close. Thank you!

Is that i now?   If the server reboots, will everything be ok?  Is there anything I should still be concerned about?
0
Promote certifications in your email signature

Has your company recently won an award or achieved a certification? They'll no doubt want to show it off. Email signature images used to promote certifications & awards can instantly establish credibility with a recipient and provide you with numerous benefits.

 
LVL 12

Expert Comment

by:BNettles73
ID: 11905390
The isinteg -fix worked? I'd definitely reboot ... keep that offline copy somewhere safe ... probably on another server if you can or at least backed up to tape ... verify if all of your data is there.
0
 

Author Comment

by:Powerhousecomputing
ID: 11905405
I haven't done the isinteg - fix yet,  it was /createstm that seems to have done it.

Should I still run isinteg?
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905412
No  problem powerhouse. No you dont have to wory about anything but you should perform a backup using NTBACKUP as it uses shadow copy to backup files that are in use.

So you shouldnt have any problems

ITCEXCHANGE
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11905462
nope ... don't run isinteg if the database is mounted and working ... you might want to check with your users ...

some things to note about having recreated the .stm file -

The amount of the data that you may lose depends on your mail client software -  If everybody uses Outlook (MAPI protocol), then there will be very little user data in the .STM file. You may lose some in transit messages that haven't been delivered yet. If clients connect via POP3 or IMAP then most of their stuff will be in the .STM file, and its loss will be catastrophic to them. If clients use Outlook Web Access, messages will be in the .EDB file, but attachments sent will be in the .STM file.
0
 

Author Comment

by:Powerhousecomputing
ID: 11905508
I'm not sure about MAPI.  I know for sure that we are using SMTP and not POP3 or IMAP.
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905563
Well... useally only HTML and DHTML are stored in the .stm file. Your attachments are normally stored in the .edb file itself.
0
 

Author Comment

by:Powerhousecomputing
ID: 11905597
So emails sent in HTML format will be lost, but plain text emails will be fine?  And some attachments may have disappeared?
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905617
Correct, but normally your attachments should be fine, it is just that the MESSAGE BODY will be blank if it was in HTML.
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11905638


If you use Outlook clients connecting over MAPI, then the messages are sent to the .EDB - very little data is sent to the .STM db. Only email that was in transit when the database went down will be lost.

If you use Outlook Web Access, messages are sent to the .EDB file but attachments are sent to the .STM db.

If you connect with Outlook, Eudora etc using POP3 or IMAP then most of the data is sent to the .STM (This means you probably had significant data loss)

0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905697
So powerhousingcomputing let me know how it worked out for relating to data loss.

ITCEXCHANGE.COM
0
 

Author Comment

by:Powerhousecomputing
ID: 11905847
Will do!
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11905878
Thanks For the Points Powerhousecomputing.

0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11905908
Congrats on getting your server back up!
0
 

Author Comment

by:Powerhousecomputing
ID: 11911703
Update

Almost everything is fine, no emails have been reported missing but some are inaccessible.  A few users get an access is denied error message when trying to open some older emails.  They also cannot delete them.  The Exchange user permissions seem fine to me but probably would be worth checking again.  Should I open another thread for this or is it just a simple permission problem?  Or corrupted email?


Apart from that Exchange is fine.  :)
0
 
LVL 8

Expert Comment

by:ITCEXCHANGE
ID: 11912623
It is corrupted email. Most of the time offline defrag will do the trick. That is why I suggested to do eseutil /d after the repair. See, the repair removes the corrupt pages, but the reference is still in the database tables.

Isinteg would probably help, but..... I would not run that utility but do the following:

If you have already run the defrag then i would suggest to run exmerge and merge out all your users to PST files. Then once this is completed dismount the the Mailbox store and move the log files(.chk file) out of the mdbdata directory.

Then Go into Exchange System manager and mount the mailbox store, it will say it is going to create a blank database, choose yes, and BOOM you have a blank database.

Now simply run exmerge again and merge all your pst files into the blank database. You will have a good working database with no errors.

Keep in mind, under the properties of the mailbox store, on the security tab, that you must have full permission's for the DOMAIN admin, ENTERPRISE admin, and Administrator or whatever you are logged in as.  You can find exmerge at http://www.microsoft.com/exchange


Then go to most popular downloads and download exmerge.

Also the exmerge.exe and exmerge.ini  must be copied into the \program files\exchsrvr\bin   directory, and executed from there.

ITCEXCHANGE.COM
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11913243
This article applied to 5.5 but the process is pretty much the same for 2K3 http://support.microsoft.com/default.aspx?kbid=259688

Get a single item with ExMerge http://www.msexchange.org/tutorials/ExMerge-Recover-Mailbox.html ...

Direct download for E2K3 Exmerge http://www.petri.co.il/download_exmerge.htm ... a few articles on how to use it as well ...
0
 

Author Comment

by:Powerhousecomputing
ID: 11914464
Just a few questions before I attempt anything.

1. Are the corrupted emails repairable?
2. Would an offline defrag return them to their original accessible state?
3. Will using Exmerge to remove the mailboxes, starting with a blank database and bringing the mailboxes back in fix the problem , or just delete the corrupted emails?
0
 
LVL 12

Expert Comment

by:BNettles73
ID: 11914576
1. In most cases you cannot repair the corrupted emails ....

2. Offline defrag will not repair the files.

3. Exmerge will extract email, calendar, contact, and other objects from the database (including dumpster items) and put them into a PST file ... after you create the new database you can exmerge (import) the files back into the database .... essentially this process will clear out the corrupt items (I think there are a few options in the exmerge software for this ... should read the help documentation)

The items that are corrupt are pretty much gone unless you have a good backup with the items.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Resolve Outlook connectivity issues after moving mailbox to new Exchange 2016 server
Check out this infographic on what you need to make a good email signature that will work perfectly for your organization.
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager
The basic steps you have just learned will be implemented in this video. The basic steps are shown to configure an Exchange DAG in a live working Exchange Server Environment and manage the same (Exchange Server 2010 Software is used in a Windows Ser…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now