exchange backup: database element is corrupt

Hey all,

Using Windows Small Business Server 2003 with Exchange Server 2003
Backing up using Veritas Backup Exec 9.1

Every day my backup fails, it started with 1 corrupt file in a mailbox, now it appears to have increased to more, which can be seen from the job history below...

The backup Job History has the following information:


Backup - \\CBS1\Microsoft Exchange Mailboxes
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Cassie [cassie]Top of Information StoreSync IssuesServer FailuresŒ〰〰〰〰㝦㑤愹愳ㄲ㘹㕣愴戹㔵㙤㜳㤹㉡㈳㝢㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㄰㠵㐷〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㔰〸㔳〰〰Letterhead & Bcard Proofs" is a corrupt file.
This file cannot verify.
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Cassie [cassie]Top of Information StoreSync IssuesServer FailuresŒ〰〰〰〰㝦㑤愹愳ㄲ㘹㕣愴戹㔵㙤㜳㤹㉡㈳㝢㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㄰㠵㐷〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㔰〸㘳〰〰Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Cassie [cassie]Top of Information StoreSync IssuesServer FailuresŒ〰〰〰〰㝦㑤愹愳ㄲ㘹㕣愴戹㔵㙤㜳㤹㉡㈳㝢㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㄰㠵㐷〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㔰〸㜳〰〰FW: Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Michael Shanahan [mas]Top of Information StoreInboxŒ〰〰〰〰㝤扢㐳扥㕢摢ㄲㄴぢ晥㐹㤴搶晣ち昸㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰〰㙡搳〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㐰㥤㜳〰〰Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Michael Shanahan [mas]Top of Information StoreInboxŒ〰〰〰〰㝤扢㐳扥㕢摢ㄲㄴぢ晥㐹㤴搶晣ち昸㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰〰㙡搳〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㐰㥤㠳〰〰FW: Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "\\CBS1\Microsoft Exchange Mailboxes\Terry Redman [txr]Top of Information StoreSync IssuesServer FailuresŒ〰〰〰〰㍥㌳〲㤳㈹㘳ㄹ昴㈹搶㘱ㄹ㥤㤷㤹㌹㜰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㄰㠵攰〰〰㕦ㄶㄸ㜶换㠲㜸㔴㠹昷㐶㥦摢扣攱ぢ〰〰〰㔰改昶〰〰FW: Business Card Proofs" is a corrupt file.
This file cannot verify.

Verify - \\CBS1\Microsoft Exchange Mailboxes
WARNING: "Letterhead & Bcard Proofs" is a corrupt file.
This file cannot verify.
WARNING: "Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "FW: Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "FW: Business Card Proofs" is a corrupt file.
This file cannot verify.
WARNING: "FW: Business Card Proofs" is a corrupt file.
This file cannot verify.


Please help me fix this nasty scenereo!
LVL 1
undysheltsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

NiclasHCommented:
Hi,

Does your full backup fail? This is brick level backup and normally isinteg solve this issue. Have a look at http://support.microsoft.com/default.aspx?scid=kb;en-us;301460.
Run isinteg -s servername -fix -test alltests several times until "fixes" 0 or the same amout twice.

But the point is to make sure you have a full backup that works, brick level backup is not a replacment for that.
undysheltsAuthor Commented:
Ok, i can do that! but.. how do i take the database offline, and more importantly, put it back online!

F:\Program Files\Exchsrvr\bin>isinteg -s cbs1 -fix -test alltests
Databases for server cbs1:
Only databases marked as Offline can be checked

Index  Status       Database-Name
Storage Group Name: First Storage Group
  1    Online       Mailbox Store (CBS1)
  2    Online       Public Folder Store (CBS1)
Enter a number to select a database or press Return to exit.
1
Invalid selection, this database cannot be checked (it is online).
Please make another selection or try this database again later.
Enter a number to select a database or press Return to exit.
NiclasHCommented:
Dismount/Mount store in Exchange System Manager
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

undysheltsAuthor Commented:
for those who don't know how to do that..

servers --> [server name] --> first storage Group --> Right click on 'Mailbox Store' and click dismount/mount volume

undysheltsAuthor Commented:
I did what NiclasH mentioned above, and it did not work, veritas backup is STILL showing in the backup log that mailbox's are corrypt... below is what i did...


F:\Program Files\Exchsrvr\bin>isinteg -s cbs1 -fix -test alltests
Databases for server cbs1:
Only databases marked as Offline can be checked

Index  Status       Database-Name
Storage Group Name: First Storage Group
  1    Offline      Mailbox Store (CBS1)
  2    Offline      Public Folder Store (CBS1)
Enter a number to select a database or press Return to exit.
1
You have selected First Storage Group / Mailbox Store (CBS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);time: 0h:0m:0s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time: 0h:0m:0s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 8 row(s); time: 0h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:9s
Test Folder result: 0 error(s); 0 warning(s); 0 fix(es); 510 row(s); time: 0h:0m:10s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 14460 row(s); time: 0h:0m:5s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 6430 row(s); time:0h:0m:0s
Test Mailbox result: 0 error(s); 0 warning(s); 0 fix(es); 18 row(s); time: 0h:0m:0s
Test Sites result: 0 error(s); 0 warning(s); 0 fix(es); 70 row(s); time: 0h:0m:0s
Test Categories result: 0 error(s); 0 warning(s); 0 fix(es); 6 row(s); time: 0h:0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s
Test special folders result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 23 row(s); time: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Now in test  20(reference count verification) of total  20 tests; 100% complete.


F:\Program Files\Exchsrvr\bin>
F:\Program Files\Exchsrvr\bin>isinteg -s cbs1 -fix -test alltests
Databases for server cbs1:
Only databases marked as Offline can be checked

Index  Status       Database-Name
Storage Group Name: First Storage Group
  1    Offline      Mailbox Store (CBS1)
  2    Offline      Public Folder Store (CBS1)
Enter a number to select a database or press Return to exit.
2
You have selected First Storage Group / Public Folder Store (CBS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);time: 0h:0m:0s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time: 0h:0m:0s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Folder result: 0 error(s); 0 warning(s); 0 fix(es); 40 row(s); time: 0h:0m:0s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 492 row(s); time: 0h:0m:0s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 1867 row(s); time:0h:0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s
Test Replication result: 0 error(s); 0 warning(s); 0 fix(es); 4 row(s); time: 0h:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 2 row(s); time: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s
Now in test  17(reference count verification) of total  17 tests; 100% complete.

F:\Program Files\Exchsrvr\bin>
NiclasHCommented:
You have an output file called isinteg.raw in the \mbdata folder. What does the summary part in the en say?
undysheltsAuthor Commented:
There's a isinteg.pri and a isinteg.pub but no isinteg.raw file.

I'll do the isinteg command again and see if that file appears.
undysheltsAuthor Commented:
No luck still.

Is there any way where I can search for an email stored on the server in a users mailbox by an email subject? If i can find the emails then hopefully delete them off the server this should fix the problem, for some reason the emails are not showing up under outlook.
undysheltsAuthor Commented:
Figured it out!

1.  Open Outlook Web Access from IE (usually http://www.SERVERNAME/exchange).
2.  Log in under each user that is causing the error in the backup log (in my case: Cassie, Michael, and Terry)
3.  Goto the folder Sync Issues, now the email will appear in one of the folders within. Either Conflicts, Local Failures, or Server Failures (forgot which one)
4. When you've found the corrupted email, delete PERMANENTLY!!! (if you just delete it into deleted items, the backup will still fail)

Could I please have my points refunded.

Cheers
CetusMODCommented:
PAQed with points refunded (200)

CetusMOD
Community Support Moderator

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.