One of my Backup is failing

Backup failing:

Veeam Backup & Replication 9.5 update 3

One of my backups are failing with the following error message:

 Processing SERVER1 Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{632d02ad-fd33-495c-8ffa-bb2d8be70d8e}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Has anyone seen this error and can possible help me with correcting it?
NJ-EWhiteIT CoordinatorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Has anyone seen this error and can possible help me with correcting it?

Very common error here on EE!

Storage too slow, or SQL Server too active, e.g. High I/O

and it's got worse since EE members have upgraded to 9.5!

Please create a Manual Snapshot (Take Snapshot) with the Quicese option ticked, at the same time your Backup Job runs, does it fail.... are you also running the VM Backup Job backing up other VMs, e.g. more than a single VM is being backed up concurrently..?
NJ-EWhiteIT CoordinatorAuthor Commented:
I do have another jobs that starts before the job in questioned, but is still running when this job starts.

Should I space out the backup and make sure they are not running at the same time?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Please Confirm the snapshot test which will rule out VSS broke in VM
NJ-EWhiteIT CoordinatorAuthor Commented:
Hey,

Can I do this while the users are working?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes
NJ-EWhiteIT CoordinatorAuthor Commented:
So that I understand, you want me to kick off the backup now, while the users are online and try to take a stapshot while the Quicese option ticked?

I 'm going to do it right now.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No, not a backup.

Take a snapshot  with  Quicese option ticked?

and then delete it, delete ALL
NJ-EWhiteIT CoordinatorAuthor Commented:
OK!  I'm trying it right now.

I took a snapshot with the Quiesce tickedand it failed.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, this indicates that  Quiesce of the VM has failed, so therefore no backup will work correctly, because it uses the same API call.

So is VMware Tools installed ?

What is this server used for ?
NJ-EWhiteIT CoordinatorAuthor Commented:
Yes the VMware tools are installed and this Server is the Domain Controller and for Data Storage.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You will need to try and uninstall VMware Tools, and try and fix the VSS issue, or disable VSS on your backup for this VM.
NJ-EWhiteIT CoordinatorAuthor Commented:
OK!  I'm not sure how to resolve the VSS issue's.  Any suggestions?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Re-install VMware Tools (warning it's a difficult issue to resolve!)
NJ-EWhiteIT CoordinatorAuthor Commented:
I uninstalled VMware and reinstalled it on Wed.  The Backup Ran to 100% on Wed Night, but last night I got the same error message again.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Interesting because that level of randomization, suggests it's not VMware Tools in the VM, but possible slow down of the datastore.

have you tried

1. a different time for the backup job
2. create a back job - which does not run concurrently with any other backup job.
NJ-EWhiteIT CoordinatorAuthor Commented:
I'm going to push it back a little.

Would any Replications running at the same time, have any affect?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
yes, anuything which takes IOPS away from the datastore

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
NJ-EWhiteIT CoordinatorAuthor Commented:
OK!  I'm going to see what happens on Monday.  

I need my replication to do one full Replication this weekend and then I will test it on Monday eve.

Thanks.
NJ-EWhiteIT CoordinatorAuthor Commented:
I'm going to close this case, because right now, due to me setting up my backup & replication.

I'm waiting on a full replication of one server.  So, I'm going to close this case and come back to it after I've gotten a full replication of all my vm's.   Perhaps that that also could be what's causing the other Server to fail on backup and replication.  

Thanks for the help.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Veeam

From novice to tech pro — start learning today.