Solved

Information store is corrupt...now what?  Log files are piling up!

Posted on 2006-11-02
25
272 Views
Last Modified: 2012-06-21
Hello experts.

Here is my situation.  I am running E2K on W2K using BE 9.1.  During my nightly backup of the information store, I now receive a failed backup because the information store is corrupt.  

What I have done.
1.  I've tried to backup using windows backup on the store but it too reported a corrupt information store.
2.  I was able to backup each mailbox, public folder, and system state using BE last night.

What do I do to fix the information store?  I need step by step as I have never come across this before.  I will be working on the problem tomorrow night.  Let me know if you need more info.

Thanks in advance.
0
Comment
Question by:braman1
  • 11
  • 5
  • 5
  • +2
25 Comments
 
LVL 14

Expert Comment

by:inbarasan
ID: 17859367
Are you seeing any event log entry on your exchange server. If so pls post it.

Normally When it shows your IS corrupted, You may need to run Softrecovert using Eseutil tool and run Isinteg to repair the logical errors.

Check this article how to recover from IS corruption
http://support.microsoft.com/kb/272570

To know more about ESEutil and Isinteg check the below articles
http://www.msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html
http://www.computerperformance.co.uk/exchange2003/exchange2003_eseutil.htm

May be some more experts will help you on this issue.
0
 

Author Comment

by:braman1
ID: 17859567
This is the most recent event log entry:

Event Type:      Error
Event Source:      ESE
Event Category:      Logging/Recovery
Event ID:      478
Date:            11/1/2006
Time:            2:48:55 PM
User:            N/A
Computer:      US
Description:
Information Store (3840) The streaming page read from the file "H:\priv1.stm" at offset 1662038016 (0x000000006310b000) for 781 (0x0000030d) bytes failed verification due to a page checksum mismatch.  The expected checksum was 2818420213 (0xa7fdadf5) and the actual checksum was 2491458009 (0x9480a1d9).  The read operation will fail with error -613 (0xfffffd9b).  If this condition persists then please restore the database from a previous backup.

For more information, click http://www.microsoft.com/contentredirect.asp.
0
 
LVL 14

Expert Comment

by:inbarasan
ID: 17859624
Looks like a bug in Exchange.
Check the below article

http://support.microsoft.com/kb/814891
http://support.microsoft.com/kb/840013
0
 
LVL 16

Expert Comment

by:glenn_1984
ID: 17859712
For the moment...stop exchange services and backup priv1.stm and put1.stm, just in case.

Also, the failure message is on verification, not backup.
If you backup without verifying does the error go away?

Also, did you see http://support.microsoft.com/default.aspx?scid=kb;en-us;318429.
0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 17862433
To expand on what glenn_1984 wrote:
Check your system log for any errors related to your hard drive or controller card. Most of the time these errors are caused by hardware problems. A hard drive going bad, SCSI controller cards causing problems.
If indeed it is a hardware issue, then you need to fix that first. If you repair the database and do not fix the hardware problem, it will just more than likely happen again. And since the only way to repair a database (other than restore from a backup that was made prior to the problem) is to run ESEUTIL /P, you do not want to do this any more than you have to. Using ESEUTIL /P can cause a loss of data.
0
 
LVL 1

Accepted Solution

by:
paulwilliams99 earned 500 total points
ID: 17866475
i would be tempted to move all the mailboxes to another database. You can the delete the corrupted database and back up the new database. It sounds like a disk or other hardware problem thats causing this and so none of the Exchange tools may work.

If you can move the mailboxes quickly then you will minimise downtime and the potential for any data loss.

Of course you may need a new server and or disk array (if you have one) as the hardware issues may affect the new database.


0
 

Author Comment

by:braman1
ID: 17866732
Well...I reviewed my event logs and there is nothing about any issue with any hardware.  I am in the middle of downloading the server management utility from Dell.  I will run that utility and post what I find out.  

If it is not hardware related, what course of action should I take to fix my information store?  I don't have another server to put into production and move mailboxes.  Well that is not totally true...I could use a beefed up workstation I suppose.

Matt
0
 
LVL 1

Expert Comment

by:paulwilliams99
ID: 17866831
This looks similar to the dreaded 1018. See article 318429 at MS.

If you dont want to restore from previous back up and replay logs, then I suggest beg, stealing or borrowing another server to move the mailboxes to.

you may not see hardware errors yet, but by the time you do it may be too late and you will have run out disk space for the transaction logs.

0
 

Author Comment

by:braman1
ID: 17868062
Just ran the Dell utility and the only conflict is that my Raid controller firmware is outdated.  I will update the firmware tonight.  Everything else came back clean.

Any other suggestions?
0
 

Author Comment

by:braman1
ID: 17868170
Glen 1984:

I tried to backup the information store without verifying and the error still exists.  No luck.

I will try backup the store tonight with the services stopped.
0
 
LVL 16

Expert Comment

by:glenn_1984
ID: 17871403
Did you try to backup to disk file as well?  Not just to tape. (i.e. is it hardware or software)
With the services off, you should be able to just copy and paste the databases to another location.
There are some safer ESEUTIL command lines to try as well.  See http://support.microsoft.com/kb/317014


0
 

Author Comment

by:braman1
ID: 17890420
I was able to stop all the services and backup the databases to tape.  The backup and verification was successful.  Now what do I do?  Should I run the ESEUTIL /P utility?  Should I be worried about losing my database?  

Thanks.
0
Free book by J.Peter Bruzzese, Microsoft MVP

Are you using Office 365? Trying to set up email signatures but you’re struggling with transport rules and connectors? Let renowned Microsoft MVP J.Peter Bruzzese show you how in this exclusive e-book on Office 365 email signatures. Better yet, it’s free!

 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 17895370
Anytime ESEUTIL /P is run there is a possibility that you can lose data.

ESEUTIL /P examines each 4KB page of the database. If there are errors or CRC mismatches then ESEUTIL will try to correct them. If they cannot be corrected then that page is discarded.

In my years of working with MS PSS I have seen the ESEUTIL /P come back with various results:
One customer had a 13 GB database. After ESEUTIL /P there was no data in any mailbox. ESEUTIL /P must have deleted pages that contained data in the main database table.

Alot of other customers either lost nothing or it was minimal.

What I am trying to say is that you never know. That is why you ALWAYS copy the databases to a secure location prior to running ESEUTIL /P.
0
 

Author Comment

by:braman1
ID: 17898101
Okay...so would you say the following steps would be the right way to proceed?

1.  Stop exchange services.
2.  Backup databases
3.  Run ESEUTIL /P with exchange services stopped
4.  Start exchange services
5.  Check mailboxes

BTW...where would I find the ESEUTIL utility?  Can I assume on the Exchange 2000 CD?

Thanks.

0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 17903188
ESEUTIL is part of a normal Exchange install. It is in the EXCHSRVR\BIN folder.

It is always a good idea to run ISINTEG -FIX against the store after ESEUTIL /P

Both of these are in the EXCHSRVR\BIN folder. If you run ISINTEG -? you will get the help screen.
Same with ESEUTIL /?

Also look up the use of these commands in the MS KnowledgeBase.
0
 

Author Comment

by:braman1
ID: 17914150
Would you agee my steps are the correct to follow?
0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 17914775
yes except you should run ISINTEG -s <servername> -fix -test alltests
after the ESEUTIL /P
0
 

Author Comment

by:braman1
ID: 17915215
One more question...

What about this...what if I create another database and move only the users mailboxes.  Then delete the corrupt database, recreate a new database and move the mailboxes back?  I'm very worried about losing data running the utilities.

Will this scenerio accomplish anything or am I just delaying the enventable?

I'm running E2K Enterprise on W2K server.

Thanks.
0
 
LVL 1

Expert Comment

by:paulwilliams99
ID: 17915296
This was the resolution i described earlier

"""i would be tempted to move all the mailboxes to another database. You can the delete the corrupted database and back up the new database. It sounds like a disk or other hardware problem thats causing this and so none of the Exchange tools may work."""

And

"""If you dont want to restore from previous back up and replay logs, then I suggest beg, stealing or borrowing another server to move the mailboxes to.

you may not see hardware errors yet, but by the time you do it may be too late and you will have run out disk space for the transaction logs"""

Im surprised your log drives havent fallen over by now.

0
 

Author Comment

by:braman1
ID: 17916248
I have less than 25 mb of space left.  I have to remove the log files tonight.

Do I just move the active user mailboxes?  What about the other mailboxes...system attendent and others?

How do I delete the old database after I move the mailboxes?

I will award pts. after this is completed tonight.

Thanks.

 
0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 17916698
Unless you are running Exchange Enterprise version, you will not be able to create a new mailstore on this server to move the mailboxes to.
If you are running Exchange 2000 Standard then yo be able to move the mailboxs to another store you will have to install an additional Exchange server.

Normally if you stop the MS Information Store service, all the logfiles will commit to the database.
IDEALLY you can then *move all the logfiles to another machine, restart the Infromation Store service, and make sure the databases are mounted.

* Move all the logfiles, any files inthe EXCHSRVR tree withthe .chk extentsion.
This will free up alot of space on your server. If the databases fail to mount then copy the logs back into place.

You may also want to consider calling Microsoft PSS. They will be able to work with you incorrecting this problem. It would probably be well worth the money.
As a past employee of MS PSS, I can attest that they will do everything in their power to fix the issue without causing dataloss. BUT if an ESEUTIL ]P is required, then there is always that possibility.

By the way my comment above about dataloss is just a cautionary statement. If you copy the database to a secure location prior to running ESEUTIL /P then if you determine that you have lost data, you can always restore the copy.
0
 
LVL 1

Expert Comment

by:paulwilliams99
ID: 17928844
Just move the user mailboxes.

Once they have moved, and tested as working, you can dismount the old corrupt database and then delete it.

You can then recreate the database and move them back. Assuming any hardware issues have been fixed, you shouldnt see the corruption come back.

Whilst there are times when eseutil and isnteg are invaluable, the risk of data loss and the associated downtime means I rarely have chance in this situation to do anything else but utlilise a spare server. Which is why I always have one available.

0
 

Author Comment

by:braman1
ID: 17930612
Okay...everyone's mailbox is moved except for the administrator's.  Keeps giving me an error when I try and move it.  Can I just unmount the store and delete it with the mailbox still there?  Can I just re-create the mailbox in the new store?  I will try and backup the information store tonight.

Thanks.
0
 
LVL 1

Expert Comment

by:paulwilliams99
ID: 17930638
you can delete the store with the mailbox still there

just recreate it on the new store
0
 

Author Comment

by:braman1
ID: 17931223
Great!  That's what I thought.  Thanks paulwilliams99.  

Starting off this week on a good note.  I hope it continues.

Thanks again everyone for your assistance.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Resolve DNS query failed errors for Exchange
This article explains in simple steps how to renew expiring Exchange Server Internal Transport Certificate.
In this video we show how to create a Shared Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Sha…
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now