VEEAM backup jobs periodically fail with VSS error

bizzie247
bizzie247 used Ask the Experts™
on
I am running VEEAM Backup & Replication 9.5 on a Windows Server 2012 R2. VEEAM is configured to backup VM's from 2 Hyper-V Servers that are running Windows Server Data Center 2012 R2. While all VM's backup successfully 'on occasion', often, I get the following error:

Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'
[/b]

I go to the VM Server identified, run the command "vssadmin list writers", and see that a number of VSS entries says 'timed out. I stop/start the Volume Shadow Copy service then restart the server. The backup jog then completes successfully. This issue will occur again after the 2nd or 3rd successful backup and again, I have to restart the server to get successful backups.

Obviously I cannot continue restarting these affected servers (all of which has databases including Exchange Server 2013). Is there anything that would likely cause the VSS services to periodically time out and fail a backup? Thanks
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Blue Street TechLast Knight
Distinguished Expert 2018
Commented:
Hi bizzie247,

Are these failures occurring job using Application-Aware Image Processing, e.g. Exchange, SQL, etc.?

This may be a known issue documented in this Microsoft KB: http://support2.microsoft.com/kb/2987610

If so, the solution is to run this query to see if the issue is present:
SELECT 1 FROM SYS.AVAILABILITY_REPLICAS AS AR INNER JOIN SYS.DATABASES AS D ON DB_ID(N'xxxx') = D.DATABASE_ID AND AR.REPLICA_ID = D.REPLICA_ID INNER JOIN SYS.DM_HADR_AVAILABILITY_REPLICA_STATES ST ON AR.REPLICA_ID = ST.REPLICA_ID AND ST.ROLE = 2
 
If you receive the following, this confirms the issue. Invalid object name 'SYS.AVAILABILITY_REPLICAS'.
 
This issue is resolved by the following Microsoft KB: http://support2.microsoft.com/kb/2983175


Also, if the other machine is your DC, verify that the NTDS VSS writer is stable:
From and elevated command prompt run the following command:
vssadmin list writers

Open in new window

The results will appear as:
Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {ee24b741-eaf7-4663-8f95-b92ae8c5e164}
   State: [1] Stable
   Last error: No error

If not listed as stable reboot the DC.
──────────────────────────────────────────────────────────
Verify that Automatic mounting of new volumes is enabled.
  1. From a Run command (Win+R) execute the command diskpart.
  2. From within Diskpart run the following command.
automount

Open in new window

If the results do not show “Automatic mounting of new volumes enabled.” Run the following command:
automount enable

Open in new window

──────────────────────────────────────────────────────────
Verify that there are no .bak keys in the ProfileList within the Registry.
From a Run command (Win+R) execute the command regedit.
Within the registry navigate to:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList
 
There will be a list of Keys, you must remove any ending in .bak

Let me know if either resolves your issue - if not there are other solutions.
Blue Street TechLast Knight
Distinguished Expert 2018

Commented:
i updated my last post...please refresh to see the changes.

Author

Commented:
Because we are not running SQL Server 2012, only instances for various programs and I do not currently have a test environment, I will run the registry check and look for .bak entries. Thanks

Author

Commented:
Nope, no .bak entries.

Author

Commented:
Yes, we have 'Application Aware' checked.
Blue Street TechLast Knight
Distinguished Expert 2018

Commented:
This may apply then, have you checked a known issue documented in this Microsoft KB: http://support2.microsoft.com/kb/2987610

If so, the solution is to run this query to see if the issue is present:
SELECT 1 FROM SYS.AVAILABILITY_REPLICAS AS AR INNER JOIN SYS.DATABASES AS D ON DB_ID(N'xxxx') = D.DATABASE_ID AND AR.REPLICA_ID = D.REPLICA_ID INNER JOIN SYS.DM_HADR_AVAILABILITY_REPLICA_STATES ST ON AR.REPLICA_ID = ST.REPLICA_ID AND ST.ROLE = 2
 
If you receive the following, this confirms the issue. Invalid object name 'SYS.AVAILABILITY_REPLICAS'.
 
This issue is resolved by the following Microsoft KB: http://support2.microsoft.com/kb/2983175

Author

Commented:
Yes, I ran the query and received the following error: Error:208, Severity:16, State:1
Invalid object name 'SYS.AVAILABILITY_REPLICAS'.

This VM Server is running on Windows 2008 R2. Is this kb suggesting that I install SQL Server 2012?
Blue Street TechLast Knight
Distinguished Expert 2018

Commented:
No, SQL runs on 2012. It is not suggesting you install it...it is a patch for it. So if you don't have it installed don't worry about it. I don't know your infrastructure so please disregard if you don't have SQL installed. I mentioned it because I've seen this issue with SQL server and it was a Microsoft issue.

Author

Commented:
Got it.... Thanks
Last Knight
Distinguished Expert 2018
Commented:
Are there any other errors. Anything talking about freeze failures. The error you described is mostly related to SQL from what I am seeing.

Anything like:
"VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec."
 
"VSSControl: Failed to freeze guest, wait timeout"

"Unfreeze Error"

"VSS_WS_FAILED_AT_FREEZE"

Troubleshooting
To source the VSS issues, open the Windows Event Viewer and check the Windows Logs > Application & System. Look for Disk, VSS, and VolSnap entries occurring at the time of backup or approximately the time when the error is reported in Veeam.

Next check the Hyper-V logs. It also helps to run search using your favorite search engine and search for: VOLSNAP 27 or VSS 8193. You basically use the Event ID and Source in your search to find additional information online on how to fix your particular error. Most of the time, however, the way to fix the error becomes obvious when reading the error message.
 
Hyper-V Volume Shadow Copy Errors
Additionally, you can search for Hyper-V VSS error information by navigating further down to: Applications and Services Logs > Microsoft > Windows > Hyper-V XXXXXXX

You'll find about a dozen different Hyper-V logs and more often than not you’ll find the answer to your problem right there.

Let me know how it goes!
A Q ChoudaryJunior Linux Engineer

Commented:
Veeam ahave a support team so it is better to lock a call (regeister a ticket)

here are some issue may you need to check first:
 1 it's too much backing up to drives at once we'd get that error "forced closed"
2 Replication of local backup and size.
3 Communication between your site and beckup point.
A Q ChoudaryJunior Linux Engineer

Commented:
Good job

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial