I have 2 Hyper-V Host servers (non clustered) that are throwing Backup errors when trying to backup the Hyper-V servers. Our backup service is Barracuda Cloud...however they cannot help.
Event ID 8229 in the VSS
=============================== A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried,
the error may not reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.
Operation:
PrepareForSnapshot Event
Context:
Execution Context: Writer
Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Name: Microsoft Hyper-V VSS Writer
Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308}
Command Line: C:\Windows\system32\vmms.exe
Process ID: 2840
=========================
Using VSSADMIN LIST WRITERS shows all writers with "No Error" on both the host machine and Hyper-V machines. All Hyper-V files including the .avhdx files are on an iSCSI volume.
Right now our backups are failing for the Hyper V servers but Barracuda cannot help because they say it's a Microsoft Problem. Any help?
Storage Software* iscsiWindows Server 2012* Volume Snapshot Service (VSS)Hyper-V
Last Comment
Joe Brown
8/22/2022 - Mon
Mark Bill
They should be providing you with more information than that in my opinion.
Try taking a backup using Veam Free or a trial of Acronis. See if that works.
Are there any more event messages? has this ever worked before? any changes in infrastructure recently?
Joe Brown
ASKER
From Barracuda:
Virtual Machines we're failing to backup:
"Error with object with path 'Hyper-V Virtual Machines/111202B9-DD8A-4B61-BF38-31BBFF4143EE/' in share '15': Array"
"Error with object with path 'Hyper-V Virtual Machines/1E7F1608-A077-4297-936E-019419D8E5CF/' in share '16': Array"
"Error with object with path 'Hyper-V Virtual Machines/689B0BFA-9ED5-4359-9236-37FF5059E91D/' in share '16': Array"
Possible error code:
0x800423f3 or 0x800423f4
possible reason:
drive is mapped to a volume guid....which looks like the error we are getting...
To check this, you can verify using the mountvol.exe
The following KB articles may be able to assist you:
The first link is mainly for clustering, although we ARE using mapped drives for the storage of our VMs (In this case its the H: drive). The second link was a dead end as we have no 3rd party VSS providers. No real changes in infrastructure. A previous support agent removed the VSS writers and re enable them somehow but I am unable to reproduce what he did. This worked for ONE backup then the next it failed again.
Mark Bill
Where and what is the storage? I read a similar issue to this recently fixed by the guys switching to Dell Equalogic VSS they were using an Equalogic SAN.
Is it local storage in arrays? on a NAS? on a SAN?
Ok so based on what you said it never worked? but an engineer did something to make it work one off? is that correct?
Is this newly installed? just curious if it ever worked consistently before or if it is a brand new install.
Do Barracuda do their own VSS kit by any chance? Do they have a recommended setup or guideline for this product?
Can we turn on verbose logging anywhere to give more visibility?
Downloading and installing VEAM free will help answer some Q's here.
Take a test backup.
Joe Brown
ASKER
The storage is a Dell iSCSI array. Seems to be working fine as far as that goes.
Joe Brown
ASKER
Barracuda does not use their own 3rd party VSS...just the MS one. Hence, why they said they could not help.
0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur.
Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {7ce2a182-2c5e-4e8f-afb9-b94d68791308} Command Line: C:\Windows\system32\vmms.exe Process ID: 2840
Initially I triedhttp://support.microsoft.com/kb/2009272 but it did not help (I re-implemented later).
I re-registered the DLL’s as perhttp://social.technet.microsoft.com/Forums/da/winservergen/thread/b4bd0deb-45d0-4661-8c50-a2b45db41103 from NareshD. I do not believe this is necessary unless there is a corruption in the system but I am noting here just in case.
I tried to run The PSConfig but it was failing:I had to do the following in order to make it finish successfully:
(from same page athttp://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/94a8bb64-e0ea-493f-8e29-01347b1fc6c7/) :
On the migrated SBS2011 PSCDiagnostic Log I found no entry to detect or start the services SPWriterServiceInstance and SPTimerServiceInstance.
I run once again the command „PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures“ in the “Bin” Folder as an administrator console session.
In the moment I saw the status “Configuration task 5 of 6 is executed” I start to run the command “net start SPTraceV4 & net start SPWriterV4 & net start SPAdminV4 & net start SPSearch4 & net start SPTimerV4” ina parallel administrator console sessionand after that, the configuration task finished successfully. (I just opened another cmd window and copy/paste the sequence below when I saw the status “Configuration Task x of y” on the screen:
net start SPTraceV4
net start SPWriterV4
net start SPAdminV4
net start SPSearch4
net start SPTimerV4
-Note SPSearch4 is disabled in our system…
Very awkward but it worked…
I implementedhttp://support.microsoft.com/kb/2009272 again.
Backup started working with no errors.
"
Mark Bill
And obviously we have option of escalation to MS support but I hate that in general, we all need to learn. Keep me in the loop.
Joe Brown
ASKER
Latest integration services might be the right track, how do I tell?
Install or update integration services
For hosts earlier than Windows Server 2016 and Windows 10, you'll need to manually install or update the integration services in the guest operating system. These steps can't be automated or done within a Windows PowerShell session.
Open Hyper-V Manager. From the Tools menu of Server Manager, click Hyper-V Manager.
Connect to the virtual machine. Right-click the virtual machine and click Connect.
From the Action menu of Virtual Machine Connection, click Insert Integration Services Setup Disk. This action loads the setup disk in the virtual DVD drive. Depending on the guest operating system, you might need to start the installation manually.
After the installation finishes, all integration services are available for use.
Right now they are not big....around 500g. About 20% used.
Philip Elder
Are the storage switches managed? What speed are they? Is there a console that will allow for monitoring for dropped packets?
The big thing about backups is the amount of I/O that they generate at the time of the snapshots.
If there's enough I/O happening when the backups run VSS can time-out.
In my experience, there are two key areas that cause VSS/Image based backups to fail:
1: Not enough I/O available
2: NTFS has less than 30% free in the partition
Joe Brown
ASKER
They are managed 1gb switches. I don't believe the I/O is a problem as the job fails almost immediately. It almost seems like a permissions or similar issue.
I suggest this: Download a trial copy of StorageCraft ShadowProtect and run a backup using it. If it's successful then that eliminates Barracuda's excuse. SP runs at the guest level.
Joe Brown
ASKER
Does StorageCraft or similar products use 3rd party VSS providers? If so, they would still have that excuse since barracuda uses the MS ones.
Philip Elder
They are all third party VSS providers. They hook into the built-in writers for the various components within Windows and the various servers that have writers such as Exchange and SQL.
We use both ShadowProtect at the guest level and Veeam at the host level. They just work.
Looks like the updated integration services fixed the problem. I got a little behind on patching and this fixed it up. Thanks for reminding me rule #1: keep up with the updates.
Try taking a backup using Veam Free or a trial of Acronis. See if that works.
Are there any more event messages? has this ever worked before? any changes in infrastructure recently?