Emergancy! Cannot mount mailbox store!

Update:  

eseutil /p  "C:\Program Files\Exchsrvr\MDBDATA\priv1.edb"

returns the following:

"The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database."

In the MDBDATA folder there are four 5MB log files called:

E00.log
E00000FA.log
RS1.log
RS2.log


Is this significant?

You probably forgot to delete log files after eseutil /P

For detail see:

Microsoft Knowledge Base Article - 822449: Error c1041724 Occurs When You Try to Mount a Database on an Exchange 2003 Server
http://support.microsoft.com/default.aspx?scid=kb;en-us;822449 

Dejan Foro
www.exchangemaster.net
www.eugeurope.org 

eseutil /mh "C:\program files\exchsrvr\mdbdata\pub1.edb"

returns a repair count = 0

A Dirty Shutdown

eseutil /p  

returns

"The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database."

A dialog box saying "You should only run Repair on damaged or corrupted databases.  Repair will not  apply information in the log files to the database and may cause information to be lost.  Do you want to proceed?"

I have always cancelled so far.   If I say YES,  will I *only* lose the info in the uncommitted log files?   Will everything in the database already remain intact?

- First of all - you have verified you have enough disk space correct?
- This issue also occurs if the E00.log file is missing or is no longer present in the Exchsrvr\Mdbdata folder or in the log file folder.
- Do you have a recent good online or offline backup of the server?

Read this article - http://support.microsoft.com/default.aspx?kbid=284205
**Read the repair count attribute.
If the repair count attribute is 0 (zero), the eseutil /p command-line command was not run.
If the repair count attribute is a number other than 0, the eseutil /p command-line command was run on the database.

Another article - http://support.microsoft.com/default.aspx?scid=kb;en-us;822449&Product=exch2003

>- First of all - you have verified you have enough disk space correct?

120GB free space


>- This issue also occurs if the E00.log file is missing or is no longer present in the >Exchsrvr\Mdbdata folder or in the log file folder.

This log file does exist in that location.


>-- Do you have a recent good online or offline backup of the server?

None whatsoever!  With the mailbox store un-mounted, Backup Exec wont touch it - is there another way?   I have however copied the whole exchsrvr folder to another drive.


I haven't gone ahead with eseutil /p  yet, see my post above for what it asks me.  The checks in those links you posted tell me that the repair count is zero and it was a dirty shutdown.

Ok I have read that.

I have a copy of the whole Exchsrvr folder,  will all logs, .edb and .stm files on a different drive.  

Shall I say YES to repairing the database and then run isinteg?  As it says in the article you posted, I can always copy back my backup copies of the files.

Ok ITCEXCHANGE  I shall give what you have suggested in your post a go!

I shall post back with what happens! :)

Ok, yes please let me know :)

It is a dirty shutdown so I did the following:

- Removed all log files from MDBDATA folder
- Ran eseutil /p for priv1.edb and pub1.edb  (Only pub1 completed successfully)
- Allow overwrite was already set for both
- The public folder store now mounts succesfully
- The mailbox store does not mount,  but gives a different error now


The mailbox store gives the error "The database files in this store are inconsistent.  ID no: c1041739"

Running eseutil /mh on priv1.edb returns:

---------
Initiating FILE DUMP mode...
Database: C:\Program Files\Exchsrvr\MDBDATA\priv1.edb

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
     DB Signature: Create time:06/23/2004 15:20:38 Rand:2879797 Computer:
         cbDbPage: 4096
           dbtime: 4001794 (0x3d1002)
            State: Dirty Shutdown
     Log Required: 251-251 (0xfb-0xfb)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 3660
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
  Last Consistent: (0xF7,B1C,1C9)  08/24/2004 08:49:39
      Last Attach: (0xF7,B24,E2)  08/24/2004 08:54:19
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:06/28/2004 16:06:01 Rand:884409 Computer:
       OS Version: (5.2.3790 SP 0)

Previous Full Backup:
        Log Gen: 247-247 (0xf7-0xf7)
           Mark: (0xF7,1A97,1FB)
           Mark: 08/24/2004 21:50:59

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

Operation completed successfully in 3.31 seconds.

--------



Have we made any progress?  I'm hoping so as the mailbox store has a new error!

What message did you get when you tried to run the repair on the priv1.edb file?

Repair produces the following message:

The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.

To ensure the database is up-to-date please use the 'Recovery' operation.

Operation terminated with error -2204 (JET_errSLVStreamingFileMissing, Cannot find
streaming file associated with this database) after 7.63 seconds.

EWWWWWWWW, well, if you are missing your stream file you will be missing all the information in HTML format. But..... anything in rich text format and all your attachments should be there. Hopefully you backed up your database already, but if not backup your priv1.edb file.

Then dismount the public information store, remove the log files and check files again. Run eseutil/p priv1.edb /createstm.

This will repair your database and create a blank stream file so that you can mount your database. After the repair runs you will be able to mount both stores.

ITCEXCHANGE

before running eseutil /p you should have ran ESEUTIL with the /M /K switches to check for pages which were not committed to the database. Initially you needed to run /r against the corrupt log file ... if you ran the /p switch you probably will need to run isinteg –fix –test alltests afterwards to mount the database ...

I'd run isinteg –fix –test alltests if you've run the /p switch and try to mount the database ... you still have the copy of all your originals right?

It worked!!!   Mailbox Store and the Public Folder store are working!  Phew that was close. Thank you!

Is that i now?   If the server reboots, will everything be ok?  Is there anything I should still be concerned about?

The isinteg -fix worked? I'd definitely reboot ... keep that offline copy somewhere safe ... probably on another server if you can or at least backed up to tape ... verify if all of your data is there.

I haven't done the isinteg - fix yet,  it was /createstm that seems to have done it.

Should I still run isinteg?

No  problem powerhouse. No you dont have to wory about anything but you should perform a backup using NTBACKUP as it uses shadow copy to backup files that are in use.

So you shouldnt have any problems

ITCEXCHANGE

nope ... don't run isinteg if the database is mounted and working ... you might want to check with your users ...

some things to note about having recreated the .stm file -

The amount of the data that you may lose depends on your mail client software -  If everybody uses Outlook (MAPI protocol), then there will be very little user data in the .STM file. You may lose some in transit messages that haven't been delivered yet. If clients connect via POP3 or IMAP then most of their stuff will be in the .STM file, and its loss will be catastrophic to them. If clients use Outlook Web Access, messages will be in the .EDB file, but attachments sent will be in the .STM file.

I'm not sure about MAPI.  I know for sure that we are using SMTP and not POP3 or IMAP.

Well... useally only HTML and DHTML are stored in the .stm file. Your attachments are normally stored in the .edb file itself.

So emails sent in HTML format will be lost, but plain text emails will be fine?  And some attachments may have disappeared?

Correct, but normally your attachments should be fine, it is just that the MESSAGE BODY will be blank if it was in HTML.

If you use Outlook clients connecting over MAPI, then the messages are sent to the .EDB - very little data is sent to the .STM db. Only email that was in transit when the database went down will be lost.

If you use Outlook Web Access, messages are sent to the .EDB file but attachments are sent to the .STM db.

If you connect with Outlook, Eudora etc using POP3 or IMAP then most of the data is sent to the .STM (This means you probably had significant data loss)

So powerhousingcomputing let me know how it worked out for relating to data loss.

ITCEXCHANGE.COM

Will do!

Thanks For the Points Powerhousecomputing.

Congrats on getting your server back up!

Update

Almost everything is fine, no emails have been reported missing but some are inaccessible.  A few users get an access is denied error message when trying to open some older emails.  They also cannot delete them.  The Exchange user permissions seem fine to me but probably would be worth checking again.  Should I open another thread for this or is it just a simple permission problem?  Or corrupted email?


Apart from that Exchange is fine.  :)

It is corrupted email. Most of the time offline defrag will do the trick. That is why I suggested to do eseutil /d after the repair. See, the repair removes the corrupt pages, but the reference is still in the database tables.

Isinteg would probably help, but..... I would not run that utility but do the following:

If you have already run the defrag then i would suggest to run exmerge and merge out all your users to PST files. Then once this is completed dismount the the Mailbox store and move the log files(.chk file) out of the mdbdata directory.

Then Go into Exchange System manager and mount the mailbox store, it will say it is going to create a blank database, choose yes, and BOOM you have a blank database.

Now simply run exmerge again and merge all your pst files into the blank database. You will have a good working database with no errors.

Keep in mind, under the properties of the mailbox store, on the security tab, that you must have full permission's for the DOMAIN admin, ENTERPRISE admin, and Administrator or whatever you are logged in as.  You can find exmerge at http://www.microsoft.com/exchange 


Then go to most popular downloads and download exmerge.

Also the exmerge.exe and exmerge.ini  must be copied into the \program files\exchsrvr\bin   directory, and executed from there.

ITCEXCHANGE.COM

This article applied to 5.5 but the process is pretty much the same for 2K3 http://support.microsoft.com/default.aspx?kbid=259688

Get a single item with ExMerge http://www.msexchange.org/tutorials/ExMerge-Recover-Mailbox.html ...

Direct download for E2K3 Exmerge http://www.petri.co.il/download_exmerge.htm ... a few articles on how to use it as well ...

Just a few questions before I attempt anything.

1. Are the corrupted emails repairable?
2. Would an offline defrag return them to their original accessible state?
3. Will using Exmerge to remove the mailboxes, starting with a blank database and bringing the mailboxes back in fix the problem , or just delete the corrupted emails?

1. In most cases you cannot repair the corrupted emails ....

2. Offline defrag will not repair the files.

3. Exmerge will extract email, calendar, contact, and other objects from the database (including dumpster items) and put them into a PST file ... after you create the new database you can exmerge (import) the files back into the database .... essentially this process will clear out the corrupt items (I think there are a few options in the exmerge software for this ... should read the help documentation)

The items that are corrupt are pretty much gone unless you have a good backup with the items.