VSS writers in failed state

Here's the output of vssadmin list writers (sorry).

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {f9bfb908-81a2-4766-bfbc-28ad53c16fc5}
   State: [10] Failed
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {038d37f5-f24b-48d7-9d5a-e25988006088}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {4c5b5a5f-d0d0-4514-8608-df8813a5eda4}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {a5fa5515-ba84-4762-bc03-5c3b54285ad4}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {a0174dce-70c1-4093-b8e5-c4bd34b18a05}
   State: [1] Stable
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {c82ee7ad-ba23-463d-b689-c00c67945342}
   State: [10] Failed
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {1634865c-1fc3-48be-8606-98120254ff87}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {8ee6eb8c-fd84-4669-b0ab-3b97afc4a370}
   State: [10] Failed
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {499f74f5-31bc-449d-a417-55109f947b4f}
   State: [10] Failed
   Last error: No error

Writer name: 'WINS Jet Writer'
   Writer Id: {f08c1483-8407-4a26-8c26-6c267a629741}
   Writer Instance Id: {2d0c6fc5-46f9-4b23-9406-3c74873bb6fb}
   State: [10] Failed
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {22b5dc86-f02c-4b1d-89d1-0e6668fb32d0}
   State: [10] Failed
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {25a8e5ca-9113-4b7e-8a91-53e0aa8058ab}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {ea078b1e-4410-4935-bb15-8655d0e90d69}
   State: [10] Failed
   Last error: No error

Writer name: 'Dhcp Jet Writer'
   Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
   Writer Instance Id: {772095fe-140b-49dc-a509-7c926a8ad7c5}
   State: [10] Failed
   Last error: No error

Select allOpen in new window

Below are the new VSS Hotfixes for "Windows Server 2003 Service Pack 2,
x86-based versions" details are below.

http://support.microsoft.com/kb/940349
http://support.microsoft.com/kb/949391
http://support.microsoft.com/kb/951568
http://support.microsoft.com/kb/957778
http://support.microsoft.com/kb/968502
http://support.microsoft.com/kb/967551
http://support.microsoft.com/kb/969219

After each Hotfix Installation Reboot the Server once.

Hope this helps,
Shree

@shreedhar
I've already applied all of these updates, rebooting each time, relating to other VSS issues I've had with this particular server.
This server has ALWAYS had VSS issues.  I haven't been able to go more than 6 weeks w/o VSS failing for some different reason.

The backups completed last night w/o errors.  Here's the output of vssadmin list writers today:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Waiting for responses. 
These may be delayed if a shadow copy is being prepared.

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {f9bfb908-81a2-4766-bfbc-28ad53c16fc5}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {4c5b5a5f-d0d0-4514-8608-df8813a5eda4}
   State: [1] Stable
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {f25750f4-86c5-4b42-9d11-77d47024afe3}
   State: [1] Stable
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {c82ee7ad-ba23-463d-b689-c00c67945342}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {8ee6eb8c-fd84-4669-b0ab-3b97afc4a370}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {25a8e5ca-9113-4b7e-8a91-53e0aa8058ab}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {499f74f5-31bc-449d-a417-55109f947b4f}
   State: [1] Stable
   Last error: No error

Writer name: 'Dhcp Jet Writer'
   Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
   Writer Instance Id: {772095fe-140b-49dc-a509-7c926a8ad7c5}
   State: [1] Stable
   Last error: No error

Writer name: 'WINS Jet Writer'
   Writer Id: {f08c1483-8407-4a26-8c26-6c267a629741}
   Writer Instance Id: {2d0c6fc5-46f9-4b23-9406-3c74873bb6fb}
   State: [1] Stable
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {22b5dc86-f02c-4b1d-89d1-0e6668fb32d0}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {262ba344-1d44-4c76-a958-dbcec58f0f24}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {3e2c01cd-5baa-4df5-8919-72c8ecb2a7d6}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {ea078b1e-4410-4935-bb15-8655d0e90d69}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {c8690fcd-7687-4053-a80d-94456d930c80}
   State: [1] Stable
   Last error: No error

Select allOpen in new window

Hi,

All writers are stable. currently.

Are using any third party backup applicationon the server?

Also post the output of vssadmin list providers

-----------
Shree

VSSADMIN LIST PROVIDERS REPORTS:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

I run Macrium Reflect to make daily differential images of the server to ext HDD.  Weekly Full runs friday.
I also do incremental backups every 15 minutes of their companyfiles to a Terastation device.
No errors since this first one.

Hi,

As everything fine till now. That sound,s good.

Check with Macrium Reflect does have any issues with the VSS aware backups. Also check are there updates for the application.

Hope this helps,
Shree

It's come back again.  Same error, same failed state.
Macrium is up to date.  Their support forums don't have any related posts.

Just a shot, I inistallled KB 940349 to update VSS.
I may have before, but I don't remember.  I'll see if I can get a VS to spawn after the update.

Any update on this???

VSS worked last night.  I have no idea if it's fixed because it works, then doesn't, then works, then doesn't randomly.  I wish I could get a definitive answer on what the heck is wrong with it.

Same issue here. KB940349 is installed and it works after a server restart, but fails the next day(s).

I've got several failed writers but I'm not sure where to go from here.

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {53c0eeb3-7f50-4517-be0f-62f8a354292b}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {dacdf793-bcce-456c-a293-9dd56e35cc27}
   State: [9] Failed
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {43443e1c-9305-4750-b297-4a0d5b6f8675}
   State: [1] Stable
   Last error: No error

Writer name: 'TermServLicensing'
   Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
   Writer Instance Id: {375e6c0a-e16a-46e9-9552-1e27d2aedb76}
   State: [1] Stable
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {96fe45cd-5b13-4626-aa37-554967e2b92e}
   State: [9] Failed
   Last error: No error

Writer name: 'Dhcp Jet Writer'
   Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
   Writer Instance Id: {8f15aa0f-12a8-4238-92f3-d1b612f4f9d0}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {d94c826f-8746-4cb4-9b01-3d9a8b1bba5f}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {ccf5bf75-d397-4350-995f-6dddc47a036a}
   State: [1] Stable
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {4e90900e-73c8-4609-90a9-1b7601231a10}
   State: [9] Failed
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {4a445efa-1954-4fed-9f6a-883ca8f2d804}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {5ceba5b7-72ec-4c82-80a8-bdbb06d7740e}
   State: [9] Failed
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {6fe5b296-02de-4c66-9bb8-5a14dcce8a7d}
   State: [1] Stable
   Last error: No error

Writer name: 'WINS Jet Writer'
   Writer Id: {f08c1483-8407-4a26-8c26-6c267a629741}
   Writer Instance Id: {963cce81-57e2-488b-8aaf-0a0c572dd242}
   State: [1] Stable
   Last error: No error

Select allOpen in new window

Have you applied othe hotfixes suggested? If not then apply them.

Note: Reboot the server after installing each hotfix.

@shreedhar
I'm going to install these hotfixes when I get the chance.  I have to carve-out the time to reboot a in production server 6 times.

Just to put out some more information:
If I run the backup manually, there doesn't' seem to be a problem & it complete w/o error.
I have no idea why VSS is fine at 11am but when I try to run it late in the evening it fails.
There's  no virus scans or any other task running at this time.

Hi there,
I know this is a common issue for many different backup softwares, based on a problem in the Windows VSS technology and too much I/O activity. I'm having the same problem with IBM's TSM v6.1.3 client.

Try by applying this fix:

 - Install hotfixes 833167 and 867686
 - Set registry entry:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VolSnap\
              NoDiffAreaFill: REG_DWORD: 1
 - The event log VSS writer makes a second copy of its files to back up. If the event logs are sufficiently large, the timeout can occur when the writer is copying the files. To help avoid the timeout, keep the log sizes below 64 MB.

Good luck!  :)

@DannyOfSweden
I will try this asap & report back.  Looks promising!

@DannyOfSweden
Both hotfixes you listed errored stating that the Service Pack installed on the server was newer than the hotfixes & didn't need to be applied.
I added the reg entry you gave.
I'm not familiar on where I set the event log VSS writer log file size.  Can you tell me where this can be done?
Thanks.

Hi Bryan,

Have you installed the suggested hotfix on the server and rebooted it? If yes, then keep the backups under observation.

---------
Shree

@shreedhar
I've not been able to apply all of these hotfixes.  The server is in uses almost always in use & it's very painful to have it go down even for a reboot.
The latest logged failure is:

Creating shadow set {35660c08-9a45-4bbb-b910-a4f1f689ec42} ...
- Adding volume \\?\Volume{91071e29-78be-11db-9bb6-806e6f6e6963}\ [C:\] to the shadow set...
Preparing for backup ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Creating the shadow (DoSnapshotSet) ...
(Waiting for the asynchronous operation to finish...)
Error during the last asynchronous operation.
- Returned HRESULT = 0x80042306
- Error text: VSS_E_PROVIDER_VETO

After dealing with this for a long time, I'm more inclined to think that it's a time-out issue as DannyOfSweden is suggesting.  If I run the backup task & make a shadow copy any other time during the day it runs fine.  The scheduled job completes equidistantly.  It completes maybe 5-10 times, then fails.  If VSS was simply "broke" & needed hotfixes to correct it, I'd think it'd fail every time.  It's not.  This just seems like an issue with the state the server's in when the task runs..
I found how to change event log size & did reduce the Application event log to 64MB after the error.  I also set the backup time to 3am from 9pm to see if we can get the job running at a less busy time.
I'll report back.

Deleted with No refund.

I was able to find a solutions.  I appoligize for not posting this sooner.  I have posted the solution in the question's thread.

Bryanchandler - I've been having the same issue for a few clients of mine and I've applied the hotfixes and it seemed to do really well for a few days and then they fail again or they just keep saying waiting for completion even after I've restarted the Volume Shadow copy service.  I see that lowering the tasks priority has solved your issue and i'm hoping it will do the same with me, but what task are you speaking of? Should I be lowering the priority for the backup software or the writers? I know this is probably a stupid question, but thanks in advance.