VSS Event ID 8229 - Hyper-V backups fail

Hello Everyone,

I have 2 Hyper-V Host servers (non clustered) that are throwing Backup errors when trying to backup the Hyper-V servers.  Our backup service is Barracuda Cloud...however they cannot help.

Event ID 8229 in the VSS

===============================
A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
   PrepareForSnapshot Event

Context:
   Execution Context: Writer
   Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Name: Microsoft Hyper-V VSS Writer
   Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308}
   Command Line: C:\Windows\system32\vmms.exe
   Process ID: 2840

=========================

Using VSSADMIN LIST WRITERS shows all writers with "No Error" on both the host machine and Hyper-V machines.  All Hyper-V files including the .avhdx files are on an iSCSI volume.

Right now our backups are failing for the Hyper V servers but Barracuda cannot help because they say it's a Microsoft Problem.  Any help?
Joe BrownIT ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
They should be providing you with more information than that in my opinion.

Try taking a backup using Veam Free or a trial of Acronis. See if that works.

Are there any more event messages? has this ever worked before? any changes in infrastructure recently?
0
Joe BrownIT ManagerAuthor Commented:
From Barracuda:

Virtual Machines we're failing to backup:
 
"Error with object with path 'Hyper-V Virtual Machines/111202B9-DD8A-4B61-BF38-31BBFF4143EE/' in share '15': Array"
 
"Error with object with path 'Hyper-V Virtual Machines/1E7F1608-A077-4297-936E-019419D8E5CF/' in share '16': Array"

"Error with object with path 'Hyper-V Virtual Machines/689B0BFA-9ED5-4359-9236-37FF5059E91D/' in share '16': Array"

Possible error code:
0x800423f3 or 0x800423f4

possible reason:
drive is mapped to a volume guid....which looks like the error we are getting...

To check this, you can verify using the mountvol.exe

The following KB articles may be able to assist you:

DPM faild to backup HyperV Gest VM Code: 0x800423F3 ID 30112
https://blogs.technet.microsoft.com/askcore/2008/10/29/configuring-storage-using-volume-guids-in-hyper-v/ 

Hyper-V Backup VSS Error 0x800423F3 – Event ID 22 and Event ID 12292
http://www.altaro.com/hyper-v/hyper-v-backup-vss-error-0x800423f3-event-id-22-and-event-id-12292/ 


The first link is mainly for clustering, although we ARE using mapped drives for the storage of our VMs (In this case its the H: drive).  The second link was a dead end as we have no 3rd party VSS providers.  No real changes in infrastructure.  A previous support agent removed the VSS writers and re enable them somehow but I am unable to reproduce what he did.  This worked for ONE backup then the next it failed again.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Where and what is the storage? I read a similar issue to this recently fixed by the guys switching to Dell Equalogic VSS they were using an Equalogic SAN.

Is it local storage in arrays? on a NAS? on a SAN?
0
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Ok so based on what you said it never worked? but an engineer did something to make it work one off? is that correct?
Is this newly installed? just curious if it ever worked consistently before or if it is a brand new install.

Do Barracuda do their own VSS kit by any chance? Do they have a recommended setup or guideline for this product?
Can we turn on verbose logging anywhere to give more visibility?

Downloading and installing VEAM free will help answer some Q's here.
Take a test backup.
0
Joe BrownIT ManagerAuthor Commented:
The storage is a Dell iSCSI array.  Seems to be working fine as far as that goes.
0
Joe BrownIT ManagerAuthor Commented:
Barracuda does not use their own 3rd party VSS...just the MS one.  Hence, why they said they could not help.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
I was just curious if they had something a long them lines.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
0
Joe BrownIT ManagerAuthor Commented:
Verbose:

0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur.  
   Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308} Command Line: C:\Windows\system32\vmms.exe Process ID: 2840
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
What happens if you shut a VM down and try to back up?

Are the latest integration services installed?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Also try this thread maybe helpful, I am more VMWare base than HyperV but am trying to upskill in Hyperv.

"
I have been fighting this error for the past 2 months!  I just posted our solution (mix of others) athttp://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/94a8bb64-e0ea-493f-8e29-01347b1fc6c7/:

Initially I triedhttp://support.microsoft.com/kb/2009272 but it did not help (I re-implemented later).
I re-registered the DLL’s as perhttp://social.technet.microsoft.com/Forums/da/winservergen/thread/b4bd0deb-45d0-4661-8c50-a2b45db41103 from NareshD. I do not believe this is necessary unless there is a corruption in the system but I am noting here just in case.
I tried to run The PSConfig but it was failing:I had to do the following in order to make it finish successfully:
(from same page athttp://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/94a8bb64-e0ea-493f-8e29-01347b1fc6c7/) :
On the migrated SBS2011 PSCDiagnostic Log I found no entry to detect or start the services SPWriterServiceInstance and SPTimerServiceInstance.
I run once again the command „PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures“ in the “Bin” Folder as an administrator console session.
In the moment I saw the status “Configuration task 5 of 6 is executed” I start to run the command “net start SPTraceV4 & net start SPWriterV4 & net start SPAdminV4 & net start SPSearch4 & net start SPTimerV4” ina parallel administrator console sessionand after that, the configuration task finished successfully.  (I just opened another cmd window and copy/paste the sequence below when I saw the status “Configuration Task x of y” on the screen:
net start SPTraceV4
net start SPWriterV4
net start SPAdminV4
net start SPSearch4
net start SPTimerV4
-Note SPSearch4 is disabled in our system…
Very awkward but it worked…
I implementedhttp://support.microsoft.com/kb/2009272 again.
Backup started working with no errors.
"
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
And obviously we have option of escalation to MS support but I hate that in general, we all need to learn. Keep me in the loop.
0
Joe BrownIT ManagerAuthor Commented:
Latest integration services might be the right track, how do I tell?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Install or update integration services
For hosts earlier than Windows Server 2016 and Windows 10, you'll need to manually install or update the integration services in the guest operating system. These steps can't be automated or done within a Windows PowerShell session.

Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.

Connect to the virtual machine. Right-click the virtual machine and click Connect.

From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive. Depending on the guest operating system, you might need to start the installation manually.

After the installation finishes, all integration services are available for use.

Taken from https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-integration-services
Good link that one
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Could you clarify this please:
...although we ARE using mapped drives for the storage of our VMs (In this case its the H: drive)...

What does that mean?

Is there an iSCSI LUN/ID set up on the storage that is hooked into the Hyper-V host and shows up as a partition there?

Where do mapped drives come in? It sounds confusing to me.

I have two very thorough EE articles on all things Hyper-V:

Some Hyper-V Hardware and Software Best Practices
Practical Hyper-V Performance Expectations
0
Joe BrownIT ManagerAuthor Commented:
There is a LUN set up as a partition on the Hyper-V host.  Currently it is the H: drive.  All guest servers are and will be hosted there.
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Okay, because "mapped drive" means a UNC path (\\Server\ShareName) mapped to a drive letter.

How big are the partitions? 1TB, 5TB, 10TB or more?
How full are they?
0
Joe BrownIT ManagerAuthor Commented:
Right now they are not big....around 500g.  About 20% used.
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Are the storage switches managed? What speed are they? Is there a console that will allow for monitoring for dropped packets?

The big thing about backups is the amount of I/O that they generate at the time of the snapshots.

If there's enough I/O happening when the backups run VSS can time-out.

In my experience, there are two key areas that cause VSS/Image based backups to fail:
1: Not enough I/O available
2: NTFS has less than 30% free in the partition
0
Joe BrownIT ManagerAuthor Commented:
They are managed 1gb switches.  I don't believe the I/O is a problem as the job fails almost immediately.  It almost seems like a permissions or similar issue.
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Ouch. 10GbE is not that expensive today.

I suggest this: Download a trial copy of StorageCraft ShadowProtect and run a backup using it. If it's successful then that eliminates Barracuda's excuse. SP runs at the guest level.
0
Joe BrownIT ManagerAuthor Commented:
Does StorageCraft or similar products use 3rd party VSS providers?  If so, they would still have that excuse since barracuda uses the MS ones.
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
They are all third party VSS providers. They hook into the built-in writers for the various components within Windows and the various servers that have writers such as Exchange and SQL.

We use both ShadowProtect at the guest level and Veeam at the host level. They just work.
0
Joe BrownIT ManagerAuthor Commented:
Looks like the updated integration services fixed the problem.  I got a little behind on patching and this fixed it up.  Thanks for reminding me rule #1:  keep up with the updates.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.