Avatar of Joe Brown
Joe Brown
 asked on

VSS Event ID 8229 - Hyper-V backups fail

Hello Everyone,

I have 2 Hyper-V Host servers (non clustered) that are throwing Backup errors when trying to backup the Hyper-V servers.  Our backup service is Barracuda Cloud...however they cannot help.

Event ID 8229 in the VSS

===============================
A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
   PrepareForSnapshot Event

Context:
   Execution Context: Writer
   Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Name: Microsoft Hyper-V VSS Writer
   Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308}
   Command Line: C:\Windows\system32\vmms.exe
   Process ID: 2840

=========================

Using VSSADMIN LIST WRITERS shows all writers with "No Error" on both the host machine and Hyper-V machines.  All Hyper-V files including the .avhdx files are on an iSCSI volume.

Right now our backups are failing for the Hyper V servers but Barracuda cannot help because they say it's a Microsoft Problem.  Any help?
Storage Software* iscsiWindows Server 2012* Volume Snapshot Service (VSS)Hyper-V

Avatar of undefined
Last Comment
Joe Brown

8/22/2022 - Mon
Mark Bill

They should be providing you with more information than that in my opinion.

Try taking a backup using Veam Free or a trial of Acronis. See if that works.

Are there any more event messages? has this ever worked before? any changes in infrastructure recently?
Joe Brown

ASKER
From Barracuda:

Virtual Machines we're failing to backup:
 
"Error with object with path 'Hyper-V Virtual Machines/111202B9-DD8A-4B61-BF38-31BBFF4143EE/' in share '15': Array"
 
"Error with object with path 'Hyper-V Virtual Machines/1E7F1608-A077-4297-936E-019419D8E5CF/' in share '16': Array"

"Error with object with path 'Hyper-V Virtual Machines/689B0BFA-9ED5-4359-9236-37FF5059E91D/' in share '16': Array"

Possible error code:
0x800423f3 or 0x800423f4

possible reason:
drive is mapped to a volume guid....which looks like the error we are getting...

To check this, you can verify using the mountvol.exe

The following KB articles may be able to assist you:

DPM faild to backup HyperV Gest VM Code: 0x800423F3 ID 30112
https://blogs.technet.microsoft.com/askcore/2008/10/29/configuring-storage-using-volume-guids-in-hyper-v/ 

Hyper-V Backup VSS Error 0x800423F3 – Event ID 22 and Event ID 12292
http://www.altaro.com/hyper-v/hyper-v-backup-vss-error-0x800423f3-event-id-22-and-event-id-12292/ 


The first link is mainly for clustering, although we ARE using mapped drives for the storage of our VMs (In this case its the H: drive).  The second link was a dead end as we have no 3rd party VSS providers.  No real changes in infrastructure.  A previous support agent removed the VSS writers and re enable them somehow but I am unable to reproduce what he did.  This worked for ONE backup then the next it failed again.
Mark Bill

Where and what is the storage? I read a similar issue to this recently fixed by the guys switching to Dell Equalogic VSS they were using an Equalogic SAN.

Is it local storage in arrays? on a NAS? on a SAN?
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Mark Bill

Ok so based on what you said it never worked? but an engineer did something to make it work one off? is that correct?
Is this newly installed? just curious if it ever worked consistently before or if it is a brand new install.

Do Barracuda do their own VSS kit by any chance? Do they have a recommended setup or guideline for this product?
Can we turn on verbose logging anywhere to give more visibility?

Downloading and installing VEAM free will help answer some Q's here.
Take a test backup.
Joe Brown

ASKER
The storage is a Dell iSCSI array.  Seems to be working fine as far as that goes.
Joe Brown

ASKER
Barracuda does not use their own 3rd party VSS...just the MS one.  Hence, why they said they could not help.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Mark Bill

I was just curious if they had something a long them lines.
Mark Bill

Joe Brown

ASKER
Verbose:

0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur.  
   Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308} Command Line: C:\Windows\system32\vmms.exe Process ID: 2840
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
ASKER CERTIFIED SOLUTION
Mark Bill

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Mark Bill

Also try this thread maybe helpful, I am more VMWare base than HyperV but am trying to upskill in Hyperv.

"
I have been fighting this error for the past 2 months!  I just posted our solution (mix of others) athttp://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/94a8bb64-e0ea-493f-8e29-01347b1fc6c7/:

Initially I triedhttp://support.microsoft.com/kb/2009272 but it did not help (I re-implemented later).
I re-registered the DLL’s as perhttp://social.technet.microsoft.com/Forums/da/winservergen/thread/b4bd0deb-45d0-4661-8c50-a2b45db41103 from NareshD. I do not believe this is necessary unless there is a corruption in the system but I am noting here just in case.
I tried to run The PSConfig but it was failing:I had to do the following in order to make it finish successfully:
(from same page athttp://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/94a8bb64-e0ea-493f-8e29-01347b1fc6c7/) :
On the migrated SBS2011 PSCDiagnostic Log I found no entry to detect or start the services SPWriterServiceInstance and SPTimerServiceInstance.
I run once again the command „PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures“ in the “Bin” Folder as an administrator console session.
In the moment I saw the status “Configuration task 5 of 6 is executed” I start to run the command “net start SPTraceV4 & net start SPWriterV4 & net start SPAdminV4 & net start SPSearch4 & net start SPTimerV4” ina parallel administrator console sessionand after that, the configuration task finished successfully.  (I just opened another cmd window and copy/paste the sequence below when I saw the status “Configuration Task x of y” on the screen:
net start SPTraceV4
net start SPWriterV4
net start SPAdminV4
net start SPSearch4
net start SPTimerV4
-Note SPSearch4 is disabled in our system…
Very awkward but it worked…
I implementedhttp://support.microsoft.com/kb/2009272 again.
Backup started working with no errors.
"
Mark Bill

And obviously we have option of escalation to MS support but I hate that in general, we all need to learn. Keep me in the loop.
Joe Brown

ASKER
Latest integration services might be the right track, how do I tell?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Mark Bill

Install or update integration services
For hosts earlier than Windows Server 2016 and Windows 10, you'll need to manually install or update the integration services in the guest operating system. These steps can't be automated or done within a Windows PowerShell session.

Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.

Connect to the virtual machine. Right-click the virtual machine and click Connect.

From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive. Depending on the guest operating system, you might need to start the installation manually.

After the installation finishes, all integration services are available for use.

Taken from https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-integration-services
Good link that one
SOLUTION
Philip Elder

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Joe Brown

ASKER
There is a LUN set up as a partition on the Hyper-V host.  Currently it is the H: drive.  All guest servers are and will be hosted there.
Philip Elder

Okay, because "mapped drive" means a UNC path (\\Server\ShareName) mapped to a drive letter.

How big are the partitions? 1TB, 5TB, 10TB or more?
How full are they?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
Joe Brown

ASKER
Right now they are not big....around 500g.  About 20% used.
Philip Elder

Are the storage switches managed? What speed are they? Is there a console that will allow for monitoring for dropped packets?

The big thing about backups is the amount of I/O that they generate at the time of the snapshots.

If there's enough I/O happening when the backups run VSS can time-out.

In my experience, there are two key areas that cause VSS/Image based backups to fail:
1: Not enough I/O available
2: NTFS has less than 30% free in the partition
Joe Brown

ASKER
They are managed 1gb switches.  I don't believe the I/O is a problem as the job fails almost immediately.  It almost seems like a permissions or similar issue.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Philip Elder

Ouch. 10GbE is not that expensive today.

I suggest this: Download a trial copy of StorageCraft ShadowProtect and run a backup using it. If it's successful then that eliminates Barracuda's excuse. SP runs at the guest level.
Joe Brown

ASKER
Does StorageCraft or similar products use 3rd party VSS providers?  If so, they would still have that excuse since barracuda uses the MS ones.
Philip Elder

They are all third party VSS providers. They hook into the built-in writers for the various components within Windows and the various servers that have writers such as Exchange and SQL.

We use both ShadowProtect at the guest level and Veeam at the host level. They just work.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Joe Brown

ASKER
Looks like the updated integration services fixed the problem.  I got a little behind on patching and this fixed it up.  Thanks for reminding me rule #1:  keep up with the updates.