NTFS Errors In Volume Shadow Copy Volumes During Backup

Hello,

We have a client with a VMware 5.5 host running several Windows 2012R2 virtual machines.  They are using Backup Exec 2014 for backup, and the actual backups succeed or fail inconsistently.  During the backup, all servers are throwing NTFS errors on the HarddiskVolumeShadowCopy volumes, such as:

Event 98, Ntfs:  Volume System Reserved (\Device\HarddiskVolumeShadowCopy20) requires an Online Scan.  An Online Scan will automatically run as part of the next scheduled maintenance task.  Alternatively you may run "CHKDSK /SCAN" locally via the command line, or run "REPAIR-VOLUME <drive:> -SCAN" locally or remotely via PowerShell.

and

Event 55: A corruption was discovered in the file system structure on volume ??.

The exact nature of the corruption is unknown.  The file system structures need to be scanned online.

A few days ago we had one of the VM's fail to boot because of registry corruption, but  it's unclear if that was because of NTFS corruption or something to do with Windows updates.  We restored a couple registry hives from the backup folder and got the server up.  We don't see any NTFS errors other than when the backups run.

I've assigned a drive letter to the System Reserved Volumes and ran a chkdsk against them and they come up clean.  Occasionally we will see a failed VSS NTDS writer.  I thought shadow copies were stored in the System Volume Information folder, but they are not there.  I've searched Symantec's knowledgebase and Google'd for clues on the errors and am coming up empty.

Can anyone lend advice on what to look for?

Thanks.
SafetyNet-TCAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

pjamCommented:
I imagine this is on a RAID 5 or higher volume.  Are any of the disks showing amber or red LEDs?
sounds like a failing or failed disk to me.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what is the underlying datastore on the ESXi host ?

is it SATA?

Network Storage, e.g. NAS or SAN?

is it fatest enough, do you have performance issues ?
0
SafetyNet-TCAuthor Commented:
VMware datastore is running on 15K SAS disks in a RAID10 configuration.  No amber lights, iDRAC shows volume is good.  We upgraded the PERC controller firmware the other day.
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

pjamCommented:
Is that when it started?  firmware update?
0
SafetyNet-TCAuthor Commented:
No, it occurred before the firmware update.  We updated the firmware thinking maybe the updated version would make it better.
0
gheistCommented:
You need to have vmware tools installed in virtual machines. That rises SCSI timeouts to the level acceptable for e.g. failing vmotion etc.
0
SafetyNet-TCAuthor Commented:
VMware tools are installed.
0
gheistCommented:
then how it detected bad blocks on virtual disk? vmware should be blinking all in red already...
0
SafetyNet-TCAuthor Commented:
NTFS errors within the OS.  Doesn't seem like actual volumes have bad blocks, just the shadow copy volumes.
0
gheistCommented:
Clean up al shadow copies and reinstall.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SafetyNet-TCAuthor Commented:
There has been no significant new corruption since the controller upgrade.  We will likely have to rebuild all the machines reporting corrupt registry files at various times.  Closing thread.  Thanks for all the suggestions.
0
SafetyNet-TCAuthor Commented:
Awarding points for effort.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.