Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 736
  • Last Modified:

checksum errors in newly created exchange 2003 mailbox store

Hi folks,

I'm in a bind here. Our mail server (exchange 2003) crashed last weekend due to our SAN going down. After we restarted, exchange would not mount our main mailbox store because it was corrupted. We were down for two days while I ran eseutil /p, then eseutil /d, then isinteg -fix -test alltests - until it came up as clean as possible. After running isinteg, I could not get any lower than 30 errors in the folder test. We mounted the mail store, gave a quick check of some mailboxes, then dumped the mail queue. Everything to the clients was working fine but backup exec 12.5 could not backup the mail store because it failed a consistency check. NTBACKUP gave us similar errors.

Since you don't want to trust a repaired database, we created two new mailbox stores (one for "large" users, and one for "regular" users) and I began to migrate users from the old mailbox store into the two new mailbox stores. The plan was to migrate everyone over the weekend, delete the old mail store and finally get a good information store backup. But nothing ever goes as planned, and while everything appears to be moving fine, I see errors in event viewer for BOTH of the new databases I created! It looks like there are two of these checksum errors in both of my newly created databases. Here is a sample of the event log:

Event Type:      Error
Event Source:      ESE
Event Category:      Database Page Cache
Event ID:      474
Date:            9/19/2010
Time:            2:31:48 AM
User:            N/A
Computer:      XXXX
Description:
Information Store (2164) First Storage Group: The database page read from the file "D:\Program Files\Exchsrvr\mdbdata\ProUsers.edb" at offset 13690195968 (0x000000032fffd000) (database page 3342332 (0x32FFFC)) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch.  The expected checksum was -3010982791536671029 (0xd636d71a9da486cb) and the actual checksum was 14355210566124788 (0x0032fffcea7844f4).  The read operation will fail with error -1018 (0xfffffc06).  If this condition persists then please restore the database from a previous backup.  This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

For more information, click http://www.microsoft.com/contentredirect.asp.

____________________________________

I haven't had a good backup in over a week now, getting worried that bad things are going to happen. I did a flat file backup this morning by dismounting the mail stores then copying the .edb, .stm and transaction log files to another system.

Any suggestions on what to do?

Also- I am actually running out of disk space with this migration too. (ugh!) I am running eseutil /d on the old mail store to shrink it since I have a good number of space hogs moved to the new database. Is this wise, or should I temporarily put the mail store on a mapped drive to another system with more disk space?

Thanks for any help!

Trent

0
CTrenton
Asked:
CTrenton
  • 4
  • 4
1 Solution
 
sosinc3Commented:
Did you delete the old mailstore database files? There are probably lots of transaction log files that you can also trim to gain some disk space. As for you error, sounds like you brought over some error with you when you tried to move the data from the old store to the new store. Does backup exec give you any errors on a particular mailbox or are you just doing exchange database backups (not folder/leaf backups)? Have you checked your hard drives for errors via CHKDSK and/or your raid array utility assuming this is a raid disk subsystem?

0
 
CTrentonAuthor Commented:
Hi!

Yes, I deleted all of the old transaction log files after we did the repair. The new transaction logs are from when we got back online. When I migrated mailboxes, some didn't go the first time because of too many corrupted items, when I upped the setting from 3 to 20, the mailboxes migrated just fine.

Backup exec just bombs out with the consistency error, nothing too specific. I will run a chkdsk on the volume holding our mail store after a do a flat file copy. (just in case - chkdsk can scramble things pretty good!)

Thanks!
0
 
sosinc3Commented:
Right, I didn't mention the flat copy because you said you just did that this morning but it never hurts to have multiple backups. If you have a list of the mailboxes that you had problems with you may want to export all their items using Outlook and then delete the mailboxes in question. Once you have deleted all the mailboxes in question, see if you have any of the same errors. If not, then create the mailboxes again and simply import all the data back in.
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

 
CTrentonAuthor Commented:
I went back into the system log and found a bunch of these:

Event Type:      Error
Event Source:      Ntfs
Event Category:      Disk
Event ID:      55
Date:            9/17/2010
Time:            9:53:25 PM
User:            N/A
Computer:      XXXX
Description:
The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume New Volume.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 02 00 52 00   ......R.
0008: 02 00 00 00 37 00 04 c0   ....7..À
0010: 00 00 00 00 02 01 00 c0   .......À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 81 01 22 00               .".    

__________________________________________________

So I'm guessing that the database errors probably have nothing to do with the mailbox moves, but these errors are a result of the SAN going down last week. (This is a VMWare virtual server, and the SAN that stores the VM images unexpectedly went out, causing all these issues.

I will run the chkdsk on the volume, but now the question is do I run isinteg -fix on the new databases? And if I do, can I trust them in a production environment or do I need to (yet again) create new mail stores and migrate again.

0
 
CTrentonAuthor Commented:
OK, I dismounted the stores, did a flat file copy to another server (.edb, .stm, and all transaction logs) and did a chkdsk on the volume. chkdsk found and fixed errors, and of course, when the server restarted those two new mailstores would not mount! I guess I need to run the eseutil /p then eseutil /d and isinteg -fix all over again on these two mail stores and start all over again? I don't have the disk capacity on the server, would it be OK to mount a network drive to temporarily store the new databases?

0
 
sosinc3Commented:
when you run the utilities, you can specify where to place the temp files and those can be located on a mapped drive. The problem is it will probably take forever if the databases are large. You may want to add s temporary hard drive to your system if it will accept any kind of internal drive even if it means unplugging the CD drive while you do this.
0
 
CTrentonAuthor Commented:
Hi,

What I ended up doing was to copy the .edb and .stm files to another server, copy the eseutil executable and .dlls to that server, run eseutil /p the eseutil /d - move the files back to the exchange server then run the isinteg -fix and then got the mailbox stores back online. I then did a check of some random mailboxes and everything looked OK. Then I unfroze the mail queue and was back in business 5:30 AM Monday morning.

Thanks!

0
 
sosinc3Commented:
Ok, great. So you are going to close the question I assume. If you need anything else, let me know.
0
 
Glen KnightCommented:
This question has been classified as abandoned and is being closed as part of the Cleanup Program. See my comment at the end of the question for more details.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 4
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now