VMwre VCB Backup problems

Backing up allot of VM's through Commvault using VCB.

Getting errors on a few. The server in question is reporting the following:

Commvault error: Warning Unable to quiesce guest file system during snapshot creation

I am seeing a lot of VSS errors in the event logs of the VM at the same time as the backup. The events are all 6013 but vary in the message. The most common is:

Error state: 1, Severity: 16
Source: Microsoft OLE DB Provider for SQL Server
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 945
Error state: 2, Severity: 14
Source: Microsoft OLE DB Provider for SQL Server
Error message: Database 'ISALOG_20110915_FWS_000' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

This server is an ISA server.

If I open the SQL error logs I can also see the following entries at the same time as the backup:

2011-10-03 23:40:21.99 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}1' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.14 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110901_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}5' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}4' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}3' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}2' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.38 spid12    This SQL Server has been optimized for 8 concurrent queries. This limit has been exceeded by 74 queries and performance may be adversely affected.

....

2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}65' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110929_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}64' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.07 spid135   Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:24.40 spid128   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}67'. Operating system error 995(error not found).
2011-10-03 23:40:24.40 spid128   Internal I/O request 0x40125C78: Op: Write, pBuffer: 0x11340400, Size: 1024, Position: 0, UMS: Internal: 0x0, InternalHigh: 0x0, Offset: 0x0, OffsetHigh: 0x0, m_buf: 0x00000000, m_len: 0, m_actualBytes: 0, m_errcode: 995, BackupFile: {AC551847-12D6-42EB-8DDF-7F0514E88278}67
2011-10-03 23:40:24.41 spid134   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}69'. Operating system error 995(error not found).


.....

2011-10-03 23:40:24.72 backup    BACKUP failed to complete the command BACKUP DATABASE [master] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}77' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.02 spid135   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}68'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.03 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20111001_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}68' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.05 spid129   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}66'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.05 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}66' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:36.00 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:40:36.49 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:40:37.74 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:40:38.89 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:39.10 spid52    Starting up database 'ISALOG_20111002_WEB_000'.
2011-10-03 23:41:09.53 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:41:09.72 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:41:10.05 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:41:10.21 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:41:10.33 spid52    Starting up database


There seems to be some kind of VSS issue and the snapshot that VCB is trying to create seems to fail.

Any help is much appreciated
LVL 1
Lamont77Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
A few checks, can you create a manual "take snapshot" with the following settings of the VM with the issues

 Test-if-Snapshot-Broken.jpg
A little background, all backup products rely on the Storage API, Snapshot process to Backup VMs. The problem is that this process, has always had issues, with quiesce guest file systems, due to the following:-

1.VMware Tools broken in the virtual machine, uninstalling VMware Tools completely, and re-installing after a reboot may help.

2. Slow datastore, multiple VMs on the same datastore, heavily loaded datastore, slow datastore subsystem, 7,200rpm, SATA RAID 5 or single disks.

3. A VM under heavy I/O, or Active Domain Controllers, Oracle and SQL databases, or database based servers such as Exchange, can also have issues.
0
Lamont77Author Commented:
There is no problem with manually creating snapshots in VC.

- The Data store is a 2TB store with roughyl 25 VM's on it. It is in alert as there is only 100GB left
- Cant confirm the type of disks at the moment
- The server in question is an ISA server and most of the errors are comming from VSS / SQL

Another question comes to mind. The Commvault logs are telling me that when it fails to create a snapshot due to quiesing ... it then tries to create a snapshot without quiescing and seems to be succesful:

5288   5 10/04 00:27:28 20783 ### CreateSnapshot_Task --- Started Create Snapshot task task-61876 for VM VMNAME 5288   5 10/04 00:28:41 20783 ### _WaitForTask --- Task [task-61876] status waited for [00:01:13.0853078] task took [00:01:12.0297270]   Wait [04/10/2011 00:27:28 to 04/10/2011 00:28:41]   Task [04/10/2011 00:27:31 to 04/10/2011 00:28:43]
5288   5 10/04 00:28:41 20783 ### CreateSnapshot_Task --- Failed to create Snapshot from VM VMNAME - Operation timed out.
5288 1d90 10/04 00:28:41 20783 CVMWareInfo::_CreateVMSnapshot() - Unable to create snapshot of VM [VMNAME], trying without quiesing filesystem

I think this means that it is still backing up OK. I need to investigate this a bit further though.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So you've checked creating Snapshot as above?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, so the issue is possible with tools, of the frozen wait time has exceeded, and it's aborting snapshot.
0
Netman66Commented:
If you say you have only 100GB left in that datastore, then that's likely your issue.

We had similar problems with low disk space - every VM that kicks off the backup needs a snapshot which is equivalent to the volume size it is snapping.  Too many VMs snapping at the same time consumed all the datastore capacity as the snapshots are saved with the VM files by default.

Try adding a datastore and reconfiguring the snapshots to use it instead to see if your issue goes away.
0
Lamont77Author Commented:
The main issue was caused by an out of date VMware component on the Comcell. The VDDK should have been at version 1.2 and was sitting at 1.1. After this update the backups were transformed and started behaving normally.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Lamont77Author Commented:
THis was the right answer
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.