Cloud Backup of System State blocked

When trying to configure a cloud backup of the System State on a Hyper-V VM of Server 2008 R2 sp1 I get the following error; VSS writer "System Writer" is missing. Please ensure that "Cryptographic Services" service has enough rights." I have checked the Cryptographic Service permissions. It appears to be identical to another VM on the same host which is working correctly. I attempted to check for System Writer by opening an elevated command prompt and running vssadmin list writers. the command fails to run with the following error "This program is blocked by group policy. For more information, contact your system administrator." Next I ran "gpresult /h result.html" from the same administrator command prompt. The results don't make sense to me. I'm hoping someone can help me interpret the results (attached) or point me in a different direction if I'm off in the weeds. This backup was working then began failing a few days ago. I tried to reconfigure the backup as a troubleshooting step but now it generates the error that System Writer is missing as described above. The only change made to the server recently is MS updates applied.
result.html
LVL 7
rettif9DaleAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Patrick BogersDatacenter platform engineer LindowsCommented:
Hi

Have a look at stop exe app data group policy. i suspect it is preventing you to run vssadmin.exe

Cheers
rettif9DaleAuthor Commented:
I looked at that. It seems like it would prevent almost everything from running. Perhaps I'm misunderstanding the scope of the GPO. It hasn't been recently added by anyone in the administrators group. So I'm asking myself where did it come from and if it has been there a long time why the recent changed behavior. As I mentioned earlier, there are 4 VM's on this host. Only this VM has a failing System State backup. The other VM backup jobs are running normally on this server and on the other VMs. System state backup job is running normally on the other servers but the GPO appears to be a domain GPO.
rettif9DaleAuthor Commented:
Vssadmin runs without any error on the other VMs one of which is a DC, so I have to assume that the GPO I posted isn't causing the problem, it is a domain GPO.

Searching through the registry I discovered this registry key;

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\safer\codeidentifiers\0\Paths\{5dd9d132-a16f-4113-bbb6-13374eb7066f}
Description         CryptoLocker Prevention
ItemData             vssadmin.exe

Event Viewer - application log
Access to C:\Windows\system32\vssadmin.exe has been restricted by your Administrator by location with policy rule {5dd9d132-a16f-4113-bbb6-13374eb7066f} placed on path vssadmin.exe.

There are other keys near this one that also seem to have to do with vssadmin. The same key is not found in the registry of the other VMs. I suspect this may be the reason vssadmin won't run but I'm unsure what generated this key or how to go about correcting it. Any suggestions?
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Hello ThereSystem AdministratorCommented:
Turn off antivirus and apps like this and try again.
rettif9DaleAuthor Commented:
All computers and servers in the domain have the same AV. The AV is not interfering with backup of the other servers. I think that eliminates AV as a cause. As I mentioned earlier VSS writers are successfully called during the backup of SQL data base and the Files and Folders backup on this server. What other app are you suggesting?
kenfcampCommented:
Actually your software update appears to be pending

Software Installation      Pending      2/25/2018 1:42:00 PM

Software Installation did not complete policy processing because a system restart is required for the settings to be applied.
Group Policy will attempt to apply the settings the next time the computer is restarted.

Additional information may have been logged. Review the Policy Events tab in the console or the application event log for events between 2/25/2018 1:42:00 PM and 2/25/2018 1:42:00 PM.

Have you rebooted??
Hello ThereSystem AdministratorCommented:
The AV is not interfering with backup of the other servers.

It doesn't mean that your server is ok.
Recently we had a situation when AV was preventing  from System State Backup on one device...
So please turn AV off and try again.
rettif9DaleAuthor Commented:
I had to wait for a maintenance window. The server has now been rebooted. Same behavior.
rettif9DaleAuthor Commented:
I decided to test by disabling the GPO overnight to make sure the change was applied to all computers in the domain. The behavior is unchanged so the GPO doesn't appear to be the cause.
rettif9DaleAuthor Commented:
I have disabled the AV and run vssadmin list writers. same behavior. An early comment suggested that the stop exe GPO might be the cause. I have disabled the GPO and left it disabled over night. The next morning I ran vssadmin list writers. The behavior was unchanged so I re-enabled the GPO.
Hypercat (Deb)Commented:
Are you running Applocker or Cryptoprevent or similar app on this server?  I found this EE post that refers to these and similar applications causing this behavior:

https://www.experts-exchange.com/questions/28611838/VSS-Issue-error-8193-vssadmin-writers-list-blocked-by-group-policy.html
rettif9DaleAuthor Commented:
The output of; gpresult /h result.html is attached to the OP. As I previously mentioned the GPO doesn't seem to be related to the registry key. Disabling the GPO had no effect. The file result.html would indicate any Applocker configuration wouldn't it? The key mentioned above;

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\safer\codeidentifiers\0\Paths\{5dd9d132-a16f-4113-bbb6-13374eb7066f}
Description         CryptoLocker Prevention
ItemData             vssadmin.exe

doesn't appear in GPO. I'm wondering if it is something that was left behind in an uninstall. i could edit the key but I would prefer to find what generated the key if possible.
Hypercat (Deb)Commented:
I assume you're logging on to the server with administrative rights.  Try adding/editing this policy in the registry if it's not already set properly:

HKLM\Software\Policies\Microsoft\Windows\Safer\CodeIdentifiers
Key: REG_DWORD
Name: PolicyScope
Value: 1

Then restart the server.  This value should exclude administrative users from any software restriction policies.
rettif9DaleAuthor Commented:
I had to wait for a maintenance window. The key was there but the value was 0. I changed it to 1 and restarted. "vssadmin list writers" is running successfully now but is only listing 7 writers, "system writer" is missing. Another 2008 VM on the same host has 11 writers. When I try to configure System State backup it fails with this error - VSS writer "System Writer" is missing. Please ensure that "Cryptographic Services" service has enough rights.
Cryptographic Services is set to automatic and configured to log on as Network Service.
Patrick BogersDatacenter platform engineer LindowsCommented:
Hi

I have disabled the GPO and left it disabled over night. The next morning I ran vssadmin list writers. The behavior was unchanged so I re-enabled the GPO

After disabling this policy and update group policy on the host with GPUPDATE did you check if GPRESULT.exe came back without this policy? It can happen you need to logoff and logon for some policies to become (un)active.

Cheers
Hypercat (Deb)Commented:
I found this related article describing how to use Procmon to troubleshoot this issue.  I thought it might be helpful:

https://blogs.msdn.microsoft.com/ntdebugging/2013/08/27/missing-system-writer-case-explained/
rettif9DaleAuthor Commented:
@ Hypercat (Deb)

I followed the instructions. I have attached a snip.  I'd like you to review the results rather than try to puzzle it out on my own. Suggestions?
ProcMon_Capture.PNG
rettif9DaleAuthor Commented:
@ Patrick Bogers

As a precaution I disabled the GPO, ran gpupdate, log off/ log on, verified that
GPO state was disabled, ran gpupdate again, ran gpresult /f /h result.html, ran vssadmin list writers on DC where I changed the GPO, System Writer stable no error, ran vssadmin list writers on member server with failing System State backup, System Writer not found. Suggestions?
rettif9DaleAuthor Commented:
I had previously sent an inquiry to foolishit.com support;

 
The reason I reached out to you is that I’m having a problem with a server backing up the system state. In troubleshooting I have found a registry key and event log entry that both point to the problem;
 
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\safer\codeidentifiers\0\Paths\{5dd9d132-a16f-4113-bbb6-13374eb7066f}
Description         CryptoLocker Prevention
ItemData             vssadmin.exe

Event Viewer - application log
Access to C:\Windows\system32\vssadmin.exe has been restricted by your Administrator by location with policy rule {5dd9d132-a16f-4113-bbb6-13374eb7066f} placed on path vssadmin.exe.
 
Researching the registry key led me to a thread on spiceworks that suggested that the foolishit crypto protect software might me involved. Just checking to see if it is related.


I just received this reply from foolishit.com support;

yes, vssadmin.exe is a setting in CryptoPrevent (originally denoted by "CryptoLocker Protection" in the event) which you can disable in this instance.  In fact, although this has been in there for years, we just posted a blog explaining what vssadmin is for.  https://www.foolishit.com/2018/03/cryptoprevent-ransomware-threat-mitigation-and-vssadmin-exe/   in any case, you'll just need to disable that setting and restart the server.  

See post above where I found a registry key identified as "CryptoLocker Prevention" The number didn't match up with numbers in the GPO so I was unsure how the key was generated. I suspect that the cryptoprevent software from foolishit.com may have been installed on this computer at one time. Possibly when it was uninstalled this key wasn't deleted. This is just speculation but if it is true I should be able to delete the key and make the problem stop. What do you more experienced commenters think?
Hypercat (Deb)Commented:
My opinion:  System state backups are a very important part of a complete backup solution.  If you need to delete the CryptoLocker prevention key to allow them to work, then my advice is to do it.  You should make a backup of the key first just in case it causes something else unforeseen to tank and you need to put it back.  The FoolishIT article was interesting but flawed IMO because it concentrates on justifying their reasons for locking vssadmin rather than addressing the many reasons you might have to have it running and advising other precautions to take if you do.  These are the main precautions that I think will address the vssadmin considerations mentioned in the article:

1.  Make sure that NO ONE in your organization except for a few trusted admins has access to anything but necessary user work product on your servers.
2.  Make sure those admins have complex passwords and change them frequently.
3.  Have a backup system that does full or full/differential backups on a weekly rotating schedule and runs reliably every day.
4.  Maintain local and cloud copies of your backups, or create local duplicate media that are taken offsite daily.
5.  Rotate your physical media so that you always have at least one full local backup (the most recent one) that is NOT directly connected.
6.  Keep all your servers up-to-date on security updates and make sure you have good perimeter and device security that can prevent or mitigate potential exploits.

If you aren't already doing these things, then you might want to revisit your backup policies.  I don't see any reason not to use the vssadmin writers as they are still required by many high-end backup software packages in order to do system state and database backups such as SQL and Exchange.
Hypercat (Deb)Commented:
P.S.  You CAN find local and local-to-cloud based backup solutions that don't require the vssadmin writers.  The one that I use is Storagecraft SPX/Storagecraft ShadowProtect.  There are a number of them around, so you might also want to look at a different cloud solution that would eliminate your need for the system writer.
rettif9DaleAuthor Commented:
@Hypercat (Deb)

I am providing backup services for this client using SolarWindsMSP Backup and Recovery. This customer was previously supported by a different MSP. I have no knowledge of what they may have done to this server. I have several SQL servers being backed up for a number of customers. This is the only server experiencing this problem. Interestingly the backup was running without error until about a week ago. I am very puzzled by that because I can't identify any change that could have triggered this behavior. SolarWindsMSP does use various vss writers but I'm not particularly concerned by that. After all it is the mechanism provided by Microsoft to backup online DBs. I am already using the hardening steps you mentioned to protect backups or a variation of the concept. I'm going to backup the registry and then delete the key(s). There are a group of keys found using F3 (search) in regedit clustered together that seem to be related all. I'll approach it cautiously. I already have a backup of the VHD so I can do a full restore if necessary. We backup the Host and the VMs.
rettif9DaleAuthor Commented:
The Domain GPO I mentioned is modeled after this thread (I think) by the previous MSP. Lengthy but interesting read. https://community.spiceworks.com/topic/389016-need-help-with-gpo-to-block-exe-s-in-appdata-folder
Hypercat (Deb)Commented:
We're an MSP also and have an MSP agreement with Storagecraft which is why I mentioned it. I don't know how it compares with what SolarWinds backup offers. We've been very happy with Storagecraft, though. Here's a link if you want to take a look:

https://www.storagecraft.com/partnerships
rettif9DaleAuthor Commented:
Rather than delete the registry key I made two changes to it. I changed the ItemData value to vssadmin.ded and changed the SaferFlags value from 0 to 1 after restarting the server I checked to verify that the registry change had been applied. I ran vssadmin list writers in an elevated command prompt. The behavior is unchanged. "System Writer" is still missing. The steps you provided earlier using ProcMon demonstrate that "System Writer" does start. Do you have any insight about the process that runs just before it closes? See the screen capture I included above.
Hypercat (Deb)Commented:
From re-reading the article I posted above along with looking at your ProcMon capture, it looks like a possible problem with one of the three setupapi.ev* files that run directly before the System Writer attempts to start.  I did some research on this and did find an article describing the same problem.  The ProcMon capture posted with that article looked very similar to yours.  The poster fixed the issue by replacing these three files with files from another server: C:\windows\inf\setupapi.ev1, setupapi.ev2 and setupapi.ev3.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
rettif9DaleAuthor Commented:
@ Hypercat (Deb)

After replacing the three files with copies from another 2008 server I ran vssadmin list writers. System Writer appeared in the list. I configured the System State backup in the backup software successfully. I ran a manual backup which completed successfully. Apparently this was the solution. Thanks for staying with the problem and finding the solution. Your help is much appreciated.
rettif9DaleAuthor Commented:
To others who provided input; I appreciate your efforts but as your comments were not part of the solution I felt compelled to award all solution points to Hypercat. I learned several things while searching for the solution to this problem. I hope others also benefited.
Hypercat (Deb)Commented:
Glad we finally got it figured out.  That's to me the best of EE - we work together on a difficult problem and we all learn something in the process.

Cheers!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.