Solved

checksum errors in newly created exchange 2003 mailbox store

Posted on 2010-09-19
10
725 Views
Last Modified: 2012-05-10
Hi folks,

I'm in a bind here. Our mail server (exchange 2003) crashed last weekend due to our SAN going down. After we restarted, exchange would not mount our main mailbox store because it was corrupted. We were down for two days while I ran eseutil /p, then eseutil /d, then isinteg -fix -test alltests - until it came up as clean as possible. After running isinteg, I could not get any lower than 30 errors in the folder test. We mounted the mail store, gave a quick check of some mailboxes, then dumped the mail queue. Everything to the clients was working fine but backup exec 12.5 could not backup the mail store because it failed a consistency check. NTBACKUP gave us similar errors.

Since you don't want to trust a repaired database, we created two new mailbox stores (one for "large" users, and one for "regular" users) and I began to migrate users from the old mailbox store into the two new mailbox stores. The plan was to migrate everyone over the weekend, delete the old mail store and finally get a good information store backup. But nothing ever goes as planned, and while everything appears to be moving fine, I see errors in event viewer for BOTH of the new databases I created! It looks like there are two of these checksum errors in both of my newly created databases. Here is a sample of the event log:

Event Type:      Error
Event Source:      ESE
Event Category:      Database Page Cache
Event ID:      474
Date:            9/19/2010
Time:            2:31:48 AM
User:            N/A
Computer:      XXXX
Description:
Information Store (2164) First Storage Group: The database page read from the file "D:\Program Files\Exchsrvr\mdbdata\ProUsers.edb" at offset 13690195968 (0x000000032fffd000) (database page 3342332 (0x32FFFC)) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch.  The expected checksum was -3010982791536671029 (0xd636d71a9da486cb) and the actual checksum was 14355210566124788 (0x0032fffcea7844f4).  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.

____________________________________

I haven't had a good backup in over a week now, getting worried that bad things are going to happen. I did a flat file backup this morning by dismounting the mail stores then copying the .edb, .stm and transaction log files to another system.

Any suggestions on what to do?

Also- I am actually running out of disk space with this migration too. (ugh!) I am running eseutil /d on the old mail store to shrink it since I have a good number of space hogs moved to the new database. Is this wise, or should I temporarily put the mail store on a mapped drive to another system with more disk space?

Thanks for any help!

Trent

0
Comment
Question by:CTrenton
  • 4
  • 4
10 Comments
 
LVL 5

Expert Comment

by:sosinc3
ID: 33711398
Did you delete the old mailstore database files? There are probably lots of transaction log files that you can also trim to gain some disk space. As for you error, sounds like you brought over some error with you when you tried to move the data from the old store to the new store. Does backup exec give you any errors on a particular mailbox or are you just doing exchange database backups (not folder/leaf backups)? Have you checked your hard drives for errors via CHKDSK and/or your raid array utility assuming this is a raid disk subsystem?

0
 

Author Comment

by:CTrenton
ID: 33711470
Hi!

Yes, I deleted all of the old transaction log files after we did the repair. The new transaction logs are from when we got back online. When I migrated mailboxes, some didn't go the first time because of too many corrupted items, when I upped the setting from 3 to 20, the mailboxes migrated just fine.

Backup exec just bombs out with the consistency error, nothing too specific. I will run a chkdsk on the volume holding our mail store after a do a flat file copy. (just in case - chkdsk can scramble things pretty good!)

Thanks!
0
 
LVL 5

Expert Comment

by:sosinc3
ID: 33711488
Right, I didn't mention the flat copy because you said you just did that this morning but it never hurts to have multiple backups. If you have a list of the mailboxes that you had problems with you may want to export all their items using Outlook and then delete the mailboxes in question. Once you have deleted all the mailboxes in question, see if you have any of the same errors. If not, then create the mailboxes again and simply import all the data back in.
0
 

Author Comment

by:CTrenton
ID: 33711707
I went back into the system log and found a bunch of these:

Event Type:      Error
Event Source:      Ntfs
Event Category:      Disk
Event ID:      55
Date:            9/17/2010
Time:            9:53:25 PM
User:            N/A
Computer:      XXXX
Description:
The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume New Volume.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 02 00 52 00   ......R.
0008: 02 00 00 00 37 00 04 c0   ....7..À
0010: 00 00 00 00 02 01 00 c0   .......À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 81 01 22 00               .".    

__________________________________________________

So I'm guessing that the database errors probably have nothing to do with the mailbox moves, but these errors are a result of the SAN going down last week. (This is a VMWare virtual server, and the SAN that stores the VM images unexpectedly went out, causing all these issues.

I will run the chkdsk on the volume, but now the question is do I run isinteg -fix on the new databases? And if I do, can I trust them in a production environment or do I need to (yet again) create new mail stores and migrate again.

0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:CTrenton
ID: 33712169
OK, I dismounted the stores, did a flat file copy to another server (.edb, .stm, and all transaction logs) and did a chkdsk on the volume. chkdsk found and fixed errors, and of course, when the server restarted those two new mailstores would not mount! I guess I need to run the eseutil /p then eseutil /d and isinteg -fix all over again on these two mail stores and start all over again? I don't have the disk capacity on the server, would it be OK to mount a network drive to temporarily store the new databases?

0
 
LVL 5

Expert Comment

by:sosinc3
ID: 33735156
when you run the utilities, you can specify where to place the temp files and those can be located on a mapped drive. The problem is it will probably take forever if the databases are large. You may want to add s temporary hard drive to your system if it will accept any kind of internal drive even if it means unplugging the CD drive while you do this.
0
 

Accepted Solution

by:
CTrenton earned 0 total points
ID: 33735390
Hi,

What I ended up doing was to copy the .edb and .stm files to another server, copy the eseutil executable and .dlls to that server, run eseutil /p the eseutil /d - move the files back to the exchange server then run the isinteg -fix and then got the mailbox stores back online. I then did a check of some random mailboxes and everything looked OK. Then I unfroze the mail queue and was back in business 5:30 AM Monday morning.

Thanks!

0
 
LVL 5

Expert Comment

by:sosinc3
ID: 33736420
Ok, great. So you are going to close the question I assume. If you need anything else, let me know.
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 34740847
This question has been classified as abandoned and is being closed as part of the Cleanup Program. See my comment at the end of the question for more details.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

This process describes the steps required to Import and Export data from and to .pst files using Exchange 2010. We can use these steps to export data from a user to a .pst file, import data back to the same or a different user, or even import data t…
Scam emails are a huge burden for many businesses. Spotting one is not always easy. Follow our tips to identify if an email you receive is a scam.
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…
This video discusses moving either the default database or any database to a new volume.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now