[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

VMwre VCB Backup problems

Posted on 2011-10-04
7
Medium Priority
?
2,140 Views
Last Modified: 2012-09-03
Backing up allot of VM's through Commvault using VCB.

Getting errors on a few. The server in question is reporting the following:

Commvault error: Warning Unable to quiesce guest file system during snapshot creation

I am seeing a lot of VSS errors in the event logs of the VM at the same time as the backup. The events are all 6013 but vary in the message. The most common is:

Error state: 1, Severity: 16
Source: Microsoft OLE DB Provider for SQL Server
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 945
Error state: 2, Severity: 14
Source: Microsoft OLE DB Provider for SQL Server
Error message: Database 'ISALOG_20110915_FWS_000' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

This server is an ISA server.

If I open the SQL error logs I can also see the following entries at the same time as the backup:

2011-10-03 23:40:21.99 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}1' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.14 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110901_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}5' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}4' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}3' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}2' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.38 spid12    This SQL Server has been optimized for 8 concurrent queries. This limit has been exceeded by 74 queries and performance may be adversely affected.

....

2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}65' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110929_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}64' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.07 spid135   Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:24.40 spid128   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}67'. Operating system error 995(error not found).
2011-10-03 23:40:24.40 spid128   Internal I/O request 0x40125C78: Op: Write, pBuffer: 0x11340400, Size: 1024, Position: 0, UMS: Internal: 0x0, InternalHigh: 0x0, Offset: 0x0, OffsetHigh: 0x0, m_buf: 0x00000000, m_len: 0, m_actualBytes: 0, m_errcode: 995, BackupFile: {AC551847-12D6-42EB-8DDF-7F0514E88278}67
2011-10-03 23:40:24.41 spid134   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}69'. Operating system error 995(error not found).


.....

2011-10-03 23:40:24.72 backup    BACKUP failed to complete the command BACKUP DATABASE [master] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}77' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.02 spid135   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}68'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.03 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20111001_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}68' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.05 spid129   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}66'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.05 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}66' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:36.00 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:40:36.49 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:40:37.74 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:40:38.89 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:39.10 spid52    Starting up database 'ISALOG_20111002_WEB_000'.
2011-10-03 23:41:09.53 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:41:09.72 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:41:10.05 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:41:10.21 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:41:10.33 spid52    Starting up database


There seems to be some kind of VSS issue and the snapshot that VCB is trying to create seems to fail.

Any help is much appreciated
0
Comment
Question by:Lamont77
  • 3
  • 3
7 Comments
 
LVL 124
ID: 36910953
A few checks, can you create a manual "take snapshot" with the following settings of the VM with the issues

 Test-if-Snapshot-Broken.jpg
A little background, all backup products rely on the Storage API, Snapshot process to Backup VMs. The problem is that this process, has always had issues, with quiesce guest file systems, due to the following:-

1.VMware Tools broken in the virtual machine, uninstalling VMware Tools completely, and re-installing after a reboot may help.

2. Slow datastore, multiple VMs on the same datastore, heavily loaded datastore, slow datastore subsystem, 7,200rpm, SATA RAID 5 or single disks.

3. A VM under heavy I/O, or Active Domain Controllers, Oracle and SQL databases, or database based servers such as Exchange, can also have issues.
0
 
LVL 1

Author Comment

by:Lamont77
ID: 36911149
There is no problem with manually creating snapshots in VC.

- The Data store is a 2TB store with roughyl 25 VM's on it. It is in alert as there is only 100GB left
- Cant confirm the type of disks at the moment
- The server in question is an ISA server and most of the errors are comming from VSS / SQL

Another question comes to mind. The Commvault logs are telling me that when it fails to create a snapshot due to quiesing ... it then tries to create a snapshot without quiescing and seems to be succesful:

5288   5 10/04 00:27:28 20783 ### CreateSnapshot_Task --- Started Create Snapshot task task-61876 for VM VMNAME 5288   5 10/04 00:28:41 20783 ### _WaitForTask --- Task [task-61876] status waited for [00:01:13.0853078] task took [00:01:12.0297270]   Wait [04/10/2011 00:27:28 to 04/10/2011 00:28:41]   Task [04/10/2011 00:27:31 to 04/10/2011 00:28:43]
5288   5 10/04 00:28:41 20783 ### CreateSnapshot_Task --- Failed to create Snapshot from VM VMNAME - Operation timed out.
5288 1d90 10/04 00:28:41 20783 CVMWareInfo::_CreateVMSnapshot() - Unable to create snapshot of VM [VMNAME], trying without quiesing filesystem

I think this means that it is still backing up OK. I need to investigate this a bit further though.
0
 
LVL 124
ID: 36911184
So you've checked creating Snapshot as above?
0
Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

 
LVL 124
ID: 36911194
Yes, so the issue is possible with tools, of the frozen wait time has exceeded, and it's aborting snapshot.
0
 
LVL 51

Expert Comment

by:Netman66
ID: 36959740
If you say you have only 100GB left in that datastore, then that's likely your issue.

We had similar problems with low disk space - every VM that kicks off the backup needs a snapshot which is equivalent to the volume size it is snapping.  Too many VMs snapping at the same time consumed all the datastore capacity as the snapshots are saved with the VM files by default.

Try adding a datastore and reconfiguring the snapshots to use it instead to see if your issue goes away.
0
 
LVL 1

Accepted Solution

by:
Lamont77 earned 0 total points
ID: 37524683
The main issue was caused by an out of date VMware component on the Comcell. The VDDK should have been at version 1.2 and was sitting at 1.1. After this update the backups were transformed and started behaving normally.
0
 
LVL 1

Author Closing Comment

by:Lamont77
ID: 38360302
THis was the right answer
0

Featured Post

NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Giving access to ESXi shell console is always an issue for IT departments to other Teams, or Projects. We need to find a way so that teams can use ESXTOP for their POCs, or tests without giving them the access to ESXi host shell console with a root …
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
Suggested Courses

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question