Veeam not backing up due to VSS not working correctly from the OS

What is the Hypervisor?

guessing windows server with hyper-v role
you really have to fix vss
vssadmin list writers

AFAIK it doesn't matter about the VM's but the ACTUAL Host must have a working VSS

@Andrew and David,

I am honored to have both of you helping on this one. It's been a rocky road with Veeam and Microsoft.

Yeah, it is Windows Server 2019 Standard as the Hyper-V host and both VMs are 2016. One is WSE, the other RDS. I always worry since somehow one of my VMs is spanning two drives.

Anyway, Veeam NEVER works due to VSS not working. Microsoft generally works when doing a bare metal backup, but if I try just backing up the VMs, it is a toss up.

Just tough sometimes working with Microsoft as they pass the ticket from department to department, and it takes three more emails to actually set up a time. But, I digress.

on the host what is the result of

vssadmin list writers

Select allOpen in new window

C:\Users\Administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {2b3647c3-36a8-42e7-b87d-f9b912c121f4}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {e905fe31-fd87-4356-9efb-7eaa516d39c7}
   State: [5] Waiting for completion
   Last error: Unexpected error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {bff33909-d442-49f0-b55e-e438fb63296e}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {4f839414-4ebe-4e06-9d82-c7743f31f0df}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {f7c58654-26c9-4f19-8817-d17ec6c72b4f}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {642abc0f-b6d2-4b52-ac9a-cb9ab67db677}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {94691d8f-582c-4eba-a5e5-3b1a78edadcd}
   State: [1] Stable
   Last error: No error

restart the Hyper-V VMMS service
check the writers status again

Microsoft Windows [Version 10.0.17763.2028]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\Administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {2b3647c3-36a8-42e7-b87d-f9b912c121f4}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {17933560-3069-4f62-bf0f-2b9fa546dc68}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {38a39e22-e32b-4295-890d-9069dd5bcbc3}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {e0ebf024-a1a8-4142-9819-a9c46a99c42e}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {21d8401c-ca8f-43d5-b2a0-a2ef822fd102}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {94691d8f-582c-4eba-a5e5-3b1a78edadcd}
   State: [1] Stable
   Last error: No error

so does the backup still fail?

Yes. Veeam remoted in. Microsoft was a no show. I could use agents in the VMs, but I would need a higher version. I can also back up if I don't enable application-aware processing, which doesn't make sense because the rest of the backup isn't using VSS.

if you don't use -  "enable application-aware processing" you just end up with crash-consistent backups which is fine because the OS cannot be quiesced. (because VSS is broke!)

So, when and if Microsoft gets around to a) admitting it is their problems and b) fixing it, I will need to enable application-aware processing?

you should fix vss
try
dism /online /cleanup-image /restorehealth
sfc /scannow

reboot and retest

upgrade install of the OS

What is upgrade install of the OS?

And, I don't know how, because I thought I had the text file saved, but I lost the command for PowerShell to delete the checkpoint. Would have done it a while ago.

But, the very nice text from Veeam told me to run the command only after backing everything up. So, let me get this right. Run the command, but it could screw things up so be sure to back up everything before running the command although I can't do backups.

you should fix vss
try
dism /online /cleanup-image /restorehealth
sfc /scannow

reboot and retest

upgrade install of the OS

OK, so I run the dism /online /cleanup-image /restorehealth as one line? this is at a admin prompt?
One that ends in system32?

Sorry having difficulties with the commands

where vm = name of the vm that can't be deleted?

if you run get-vm you will get a listing of the vmnames

what I gave you removes all snapshots in all vm's

Just a question. As I am forced to use Windows Server Backup to backup my VMs and/or server, more than 2 out of 3 times it fails due to VSS not making a shadow copy correctly. But 1 out of 3 times it does. What explains that?

have you tried the over install? or the dism /online /cleanup-image /restorehealth  ?  Its almost time to reinstall from scratch and start anew if you've already done these

OK, so I run the

 dism /online /cleanup-image /restorehealth

Select allOpen in new window

as one line? this is at a admin prompt?
One that ends in system32?
yes it is one line from an command prompt run as administrator

OK. I uninstalled Veeam and reinstalled it. All that did was help me learn how to uninstall and reinstall much more quickly than before. Thought I would try.

I do have the media of Windows Server 2019 Standard to use for the Hyper-V. I wonder if I could just install over and choose to not change anything. I think David mentioned that.

Also, right next to the server is a Lenovo server, about 2 1/2 years old, only used for the Veeam Backup Server. Yes, they highly recommend installing the software on a separate server.

Could't I move the two VMs to that server. Of course, if I could I could just use that one except it is  32GBs of RAM vs 96. But, if I did that, I could wipe the Dell clean and start over when moving the VMs back. Of course, this is one of the advantages of VM, but I wouldn't feel too good about it especially with one WSB to restore.

@David,

Yeah, I have been dreading starting anew. :(

I agree. That is the way to go. And, I was thinking of that.

Then, my long last IT guy from California couldn't resist the challenge and decided to remote in and do what Veeam and Microsoft couldn't do. Fix VSS and fix everything. So now everything works. Microsoft should be ashamed of themselves. They:

1. Did not keep appointments to work on it
2. When they worked on it, they did one WBS backup which happened to work. I don't know how it would work 40% of the time, but Veeam would fail 100% of the time. But, the tech happened to catch that one time and said it was fixed.
3. Constantly call me with that auto-call before the tech gets on 2 minutes later during work and start right in with a conversation without even considering I was at work.
4. Made an appointment with Veeam and me to work together to fix it and didn't show up.
5. Every time Microsoft was even close to doing a remote, the lead tech would say he couldn't make it and to just call his team leaders who never returned the call.
6. At least three times changed me to another support team, because they realized they weren't the right team.

This is the second time my IT guy has remoted in from California to fix a problem Microsoft couldn't fix.

what was the fix.. hopefully you recorded it for posterity and to help others

Sandeep is an interesting guy. He used to be my full-time IT (which gives him a bit of an upper hand on the system). Then one day I didn't hear from him. For months. One day, after an email, he emailed me the solution. Didn't hear from him for a while.

I had emailed him about this and offered to pay him his going rate. Nothing. Then all of a sudden I get an email saying he couldn't remote in anymore, so I left him. I could see him working on it, but he wouldn't talk to me on IM or text or email. Then the following day he states he could get RDS VM to work but not Ess, but within hours all was fixed including the orphaned checkpoint.

Then he backed out, and I have emailed him three times just asking what he did so I could do the very thing you are recommending. Weird. I really think things are not going well for him but every once in a while he pops in and fixes something that becomes a dilemma.

I am rather confident that those two recommendations (almost the same thing) would fix the problem. Still don't know what Sandeep did.