NTFS Errors In Volume Shadow Copy Volumes During Backup

Hello,

We have a client with a VMware 5.5 host running several Windows 2012R2 virtual machines.  They are using Backup Exec 2014 for backup, and the actual backups succeed or fail inconsistently.  During the backup, all servers are throwing NTFS errors on the HarddiskVolumeShadowCopy volumes, such as:

Event 98, Ntfs:  Volume System Reserved (\Device\HarddiskVolumeShadowCopy20) requires an Online Scan.  An Online Scan will automatically run as part of the next scheduled maintenance task.  Alternatively you may run "CHKDSK /SCAN" locally via the command line, or run "REPAIR-VOLUME <drive:> -SCAN" locally or remotely via PowerShell.

and

Event 55: A corruption was discovered in the file system structure on volume ??.

The exact nature of the corruption is unknown.  The file system structures need to be scanned online.

A few days ago we had one of the VM's fail to boot because of registry corruption, but  it's unclear if that was because of NTFS corruption or something to do with Windows updates.  We restored a couple registry hives from the backup folder and got the server up.  We don't see any NTFS errors other than when the backups run.

I've assigned a drive letter to the System Reserved Volumes and ran a chkdsk against them and they come up clean.  Occasionally we will see a failed VSS NTDS writer.  I thought shadow copies were stored in the System Volume Information folder, but they are not there.  I've searched Symantec's knowledgebase and Google'd for clues on the errors and am coming up empty.

Can anyone lend advice on what to look for?

Thanks.
SafetyNet-TCAsked:
Who is Participating?
 
gheistCommented:
Clean up al shadow copies and reinstall.
0
 
pjamCommented:
I imagine this is on a RAID 5 or higher volume.  Are any of the disks showing amber or red LEDs?
sounds like a failing or failed disk to me.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what is the underlying datastore on the ESXi host ?

is it SATA?

Network Storage, e.g. NAS or SAN?

is it fatest enough, do you have performance issues ?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
SafetyNet-TCAuthor Commented:
VMware datastore is running on 15K SAS disks in a RAID10 configuration.  No amber lights, iDRAC shows volume is good.  We upgraded the PERC controller firmware the other day.
0
 
pjamCommented:
Is that when it started?  firmware update?
0
 
SafetyNet-TCAuthor Commented:
No, it occurred before the firmware update.  We updated the firmware thinking maybe the updated version would make it better.
0
 
gheistCommented:
You need to have vmware tools installed in virtual machines. That rises SCSI timeouts to the level acceptable for e.g. failing vmotion etc.
0
 
SafetyNet-TCAuthor Commented:
VMware tools are installed.
0
 
gheistCommented:
then how it detected bad blocks on virtual disk? vmware should be blinking all in red already...
0
 
SafetyNet-TCAuthor Commented:
NTFS errors within the OS.  Doesn't seem like actual volumes have bad blocks, just the shadow copy volumes.
0
 
SafetyNet-TCAuthor Commented:
There has been no significant new corruption since the controller upgrade.  We will likely have to rebuild all the machines reporting corrupt registry files at various times.  Closing thread.  Thanks for all the suggestions.
0
 
SafetyNet-TCAuthor Commented:
Awarding points for effort.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.