Exchange 2007 Backup Failure (Event ID 565), When Running VSS Backup of Exchange Mailbox Database

Hello!

I am experiencing issues backing up an MS Exchange 2007 SP 3 instance running on Windows Server 2008 R2, using the Windows Server Backup solution.

The backup solution is experiencing consistency check issues when backing up the database, causing the backup to complete with the warning that the Exchange application will be non-recoverable.

The mailbox database on which the check is failing was previously the source database for an SCR seed.  I have read that this type of consistency check backup error is not uncommon when SCR is used.  However, SCR replication for that storage group has since been disabled, and even so, the Exchange portion of the backup still fails.

I have already tried creating a DWORD entry for HKLM\SOFTWARE\Microsoft\Exchange\Replay\Parameters\EnableDSSWriter = 0, followed by restarting the "Microsoft Exchange Replication Service", which I've read that other folks have found fruitful in other scenarios.  It does not appear to have eased my symptoms, however.

I'm pasting below a listing of all the messages logged to the Application logs while the backup was running, by order of appearance.  Any assistance is greatly needed and appreciated.

Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          9/21/2013 7:30:00 PM
Event ID:      753
Task Category: None
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      server.domain.local
Description:
The Block Level Backup Engine service has successfully started.
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:09 PM
Event ID:      9606
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 50ae3333-8bc9-41eb-a211-b461434272ed) has prepared for backup successfully. 
====================
Log Name:      Application
Source:        ESE
Date:          9/21/2013 7:30:11 PM
Event ID:      2005
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Information Store (4700) Shadow copy instance 3 starting. This will be a Full shadow copy. 

For more information, click http://www.microsoft.com/contentredirect.asp.
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:11 PM
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 3) has successfully prepared the database engine for a full or copy backup of storage group 'PHP Database'. The following 1 databases are mounted and will be backed up:
 
	Mailbox Database 1
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:11 PM
Event ID:      9811
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 3) has successfully prepared the database engine for a full or copy backup of storage group 'PHP Public Folders 1'. The following 1 databases are mounted and will be backed up:
 
	Public Folder Database 1
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:11 PM
Event ID:      9608
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 50ae3333-8bc9-41eb-a211-b461434272ed:3) has prepared for Snapshot successfully. 
====================
Log Name:      Application
Source:        ESENT
Date:          9/21/2013 7:30:11 PM
Event ID:      103
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
DFSRs (1240) \\.\H:\System Volume Information\DFSR\database_C4C2_1120_C211_17EE\dfsr.db: The database engine stopped the instance (0).
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3197
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3197 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

model

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        ESE
Date:          9/21/2013 7:30:12 PM
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Information Store (4700) Shadow copy instance 3 freeze started. 

For more information, click http://www.microsoft.com/contentredirect.asp.
====================
Log Name:      Application
Source:        ESE
Date:          9/21/2013 7:30:12 PM
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
MSExchangeIS (4700) PHP Database: Shadow copy instance 3 freeze started. 

For more information, click http://www.microsoft.com/contentredirect.asp.
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3197
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3197 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

msdb

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        ESE
Date:          9/21/2013 7:30:12 PM
Event ID:      2001
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
MSExchangeIS (4700) PHP Public Folders 1: Shadow copy instance 3 freeze started. 

For more information, click http://www.microsoft.com/contentredirect.asp.
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3197
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3197 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

SUSDB

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3197
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3197 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

master

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:12 PM
Event ID:      9610
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 50ae3333-8bc9-41eb-a211-b461434272ed:3) has frozen the storage group(s) successfully. 
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3198
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3198 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

SUSDB

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3198
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3198 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

model

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3198
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3198 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

master

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:12 PM
Event ID:      3198
Task Category: (2)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 3198 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

msdb

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        ESE
Date:          9/21/2013 7:30:12 PM
Event ID:      2003
Task Category: ShadowCopy
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Information Store (4700) Shadow copy instance 3 freeze ended. 

For more information, click http://www.microsoft.com/contentredirect.asp.
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:12 PM
Event ID:      9612
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 50ae3333-8bc9-41eb-a211-b461434272ed:3) has thawed the storage group(s) successfully. 
====================
Log Name:      Application
Source:        ESENT
Date:          9/21/2013 7:30:15 PM
Event ID:      102
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
DFSRs (1240) \\.\H:\System Volume Information\DFSR\database_C4C2_1120_C211_17EE\dfsr.db: The database engine (6.01.7601.0000) started a new instance (0).
====================
Log Name:      Application
Source:        MSExchangeIS
Date:          9/21/2013 7:30:16 PM
Event ID:      9622
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Exchange VSS Writer (instance 50ae3333-8bc9-41eb-a211-b461434272ed:3) has processed the post-snapshot event successfully. 
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:16 PM
Event ID:      18264
Task Category: (6)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 18264 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

master
2013/09/21
15:12:35
1
398:368:37
398:392:1
1
FILE=1, TYPE=VIRTUAL_DEVICE: {'{FF0B7DF7-6405-403D-8DD3-697BC5F41916}4'}

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:16 PM
Event ID:      18264
Task Category: (6)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 18264 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

model
2003/04/08
09:13:36
1
29:464:37
29:488:1
1
FILE=1, TYPE=VIRTUAL_DEVICE: {'{FF0B7DF7-6405-403D-8DD3-697BC5F41916}1'}

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:16 PM
Event ID:      18264
Task Category: (6)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 18264 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

SUSDB
2013/01/21
12:06:42
1
64165:93:51
64165:114:1
1
FILE=1, TYPE=VIRTUAL_DEVICE: {'{FF0B7DF7-6405-403D-8DD3-697BC5F41916}3'}

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        MSSQL$MICROSOFT##SSEE
Date:          9/21/2013 7:30:16 PM
Event ID:      18264
Task Category: (6)
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      server.domain.local
Description:
The description for Event ID 18264 from source MSSQL$MICROSOFT##SSEE cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

msdb
2005/10/14
01:54:05
1
118:576:172
118:656:1
1
FILE=1, TYPE=VIRTUAL_DEVICE: {'{FF0B7DF7-6405-403D-8DD3-697BC5F41916}2'}

The specified resource type cannot be found in the image file
====================
Log Name:      Application
Source:        VSS
Date:          9/21/2013 7:30:34 PM
Event ID:      8220
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.domain.local
Description:
Ran out of time while deleting files. 

Operation:
   OnPostSnapshot event
   PostSnapshot Event

Context:
   Execution Context: Shadow Copy Optimization Writer
   Execution Context: Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {79842fdc-b2d0-4ef1-8323-54894580dc16}
====================
Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          9/21/2013 7:30:46 PM
Event ID:      565
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      server.domain.local
Description:
The consistency check for the component '188a696e-0ee3-4637-9108-6f859dcab338'\'Microsoft Exchange Server\Microsoft Information Store\EXCHANGESERVER1' has failed. The application 'Exchange' will not be available in the backup created at '¿2013¿-¿09¿-¿21T19:30:06.145448400Z'. Review the event details for information about consistency check issues.

Open in new window

V16ProAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

V16ProAuthor Commented:
Additionally, I have just tried:

1) enabling circular logging on both the mailbox and public folder databases,
2) dismounting and remounting each database to truncate log files,
3) returning circular logging setting to 'disabled' for the mailbox and public folder databases,
4) dismounting and remounting each database to apply the setting.

Subsequently, upon attempting a backup, I again received the customary consistency check error message several seconds into the backup job.  The job proceeded to complete "Successfully" with the "Warning" that Exchange would not be recoverable.

I would appreciate any additional insight you might offer.
0
Peter HutchisonSenior Network Systems SpecialistCommented:
On the disk where the databases are stored, open properties of the disk and make sure Shadow Copies are enabled (keep the max space small e.g. 10GB) to allow VSS to work.
0
V16ProAuthor Commented:
Thanks, cmsxpjh.  I've just enabled Shadow Copies for that disk as per your suggestion, and re-ran the backup.  I again received the "consistency check" warning for the Exchange application several seconds after the backup began.
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

Peter HutchisonSenior Network Systems SpecialistCommented:
I would run ESEUTIL /K and maybe ESEUTIL /P to check and repair the databases before trying another backup.

http://support.microsoft.com/kb/2616952
0
Simon Butler (Sembee)ConsultantCommented:
I would just create a new database and see if that works correctly. If it does, then move all content to the new database and remove the original. Repairing the database uising eseutil isn't risk free and can result in data loss.

Simon.
0
V16ProAuthor Commented:
Folks --

Thanks for those suggestions!  I think I'll use the 'eseutil /k' command to discover which database(s) (mailbox or public folder) have integrity issues, and then, should I choose to move forward from there, investigate moving contents to the new database as per Simon's suggestion.

I am in the midst of upgrading our Exchange environment to Exchange 2010, so I don't foresee needing the 2007 databases much longer.  However, while I have the second node in my two-node Exchange environment running Exchange 2010 in coexistence (and thus no SCR configuration or public folder replication in place), I'd like to have some confidence that the bulk of my data, while it remains in the 2007 environment, has some change at being recoverable "if the worst should happen."

I'm familiar with the process for moving mailboxes to a new mailbox server.  However, if it should happen that the public folders database is not consistent, what would be the procedure for moving data to a different public folders database on the same node?

Thanks!
0
Peter HutchisonSenior Network Systems SpecialistCommented:
For moving public folders, you can use the supplied powershell script to move or copy to another server or public folder DB:
MoveAllReplicas.ps1
AddReplicaToPFRecursive.ps1

You can also use the Public Folder Management Console to configure replicas for individual folders.
0
V16ProAuthor Commented:
Folks --

An interesting discovery on the VSS Full Backup concern regarding the mailbox database: I noticed that the "Microsoft Exchange Server Extension for Windows Server Backup" service ('wsbexchange') is marked as "Disabled."  Based on this site, I see that the default Startup Type for this service should be "Manual."

Is it possible that, when enabling SCR for this database some months ago, SCR may have Disabled this service?

Is it safe to set it to "Manual," as long as SCR is disabled to the former target server, and retry the backup?

I simply am having a difficult time believing that both the Public Folder and Mailbox databases are corrupted...

Thanks!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Peter HutchisonSenior Network Systems SpecialistCommented:
I would consider it unlikely that SCR would disable the service.

Yes, if the service is not required, then setting it to manual is okay.
0
V16ProAuthor Commented:
The fix was to adjust the "Startup Type" for the "Microsoft Exchange Server Extension for Windows Server Backup" service from "Disabled" to "Manual."  Not sure how it became disabled if SCR didn't do it.  We have only two administrators supervising our Exchange environment, and neither of us changed it to "Disabled."

I'll accept solutions from Sembee2 and cmsxpjh as a token of my thanks for your assistance with this issue.  Had our databases turned out to have been corrupted, I have no doubt that the solutions you each offered would have proven invaluable.

Thanks!
0
V16ProAuthor Commented:
The fix was to adjust the "Startup Type" for the "Microsoft Exchange Server Extension for Windows Server Backup" service from "Disabled" to "Manual."  Not sure how it became disabled if SCR didn't do it.  We have only two administrators supervising our Exchange environment, and neither of us changed it to "Disabled."

I'll accept solutions from Sembee2 and cmsxpjh as a token of my thanks for your assistance with this issue.  Had our databases turned out to have been corrupted, I have no doubt that the solutions you each offered would have proven invaluable.

Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.