Start Free TrialLog in
Avatar of Lamont77
Lamont77

asked on 

VMwre VCB Backup problems

Backing up allot of VM's through Commvault using VCB.

Getting errors on a few. The server in question is reporting the following:

Commvault error: Warning Unable to quiesce guest file system during snapshot creation

I am seeing a lot of VSS errors in the event logs of the VM at the same time as the backup. The events are all 6013 but vary in the message. The most common is:

Error state: 1, Severity: 16
Source: Microsoft OLE DB Provider for SQL Server
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 945
Error state: 2, Severity: 14
Source: Microsoft OLE DB Provider for SQL Server
Error message: Database 'ISALOG_20110915_FWS_000' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

This server is an ISA server.

If I open the SQL error logs I can also see the following entries at the same time as the backup:

2011-10-03 23:40:21.99 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}1' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.14 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110901_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}5' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}4' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110831_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}3' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.16 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110830_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}2' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:22.38 spid12    This SQL Server has been optimized for 8 concurrent queries. This limit has been exceeded by 74 queries and performance may be adversely affected.

....

2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_FWS_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}65' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.00 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110929_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}64' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:23.07 spid135   Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:24.40 spid128   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}67'. Operating system error 995(error not found).
2011-10-03 23:40:24.40 spid128   Internal I/O request 0x40125C78: Op: Write, pBuffer: 0x11340400, Size: 1024, Position: 0, UMS: Internal: 0x0, InternalHigh: 0x0, Offset: 0x0, OffsetHigh: 0x0, m_buf: 0x00000000, m_len: 0, m_actualBytes: 0, m_errcode: 995, BackupFile: {AC551847-12D6-42EB-8DDF-7F0514E88278}67
2011-10-03 23:40:24.41 spid134   BackupMedium::ReportIoError: write failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}69'. Operating system error 995(error not found).


.....

2011-10-03 23:40:24.72 backup    BACKUP failed to complete the command BACKUP DATABASE [master] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}77' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.02 spid135   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}68'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.03 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20111001_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}68' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:25.05 spid129   BackupVirtualDeviceSet::Initialize: Open failure on backup device '{AC551847-12D6-42EB-8DDF-7F0514E88278}66'. Operating system error 0x80070002(error not found).
2011-10-03 23:40:25.05 backup    BACKUP failed to complete the command BACKUP DATABASE [ISALOG_20110930_WEB_000] TO VIRTUAL_DEVICE='{AC551847-12D6-42EB-8DDF-7F0514E88278}66' WITH SNAPSHOT,BUFFERCOUNT=1,BLOCKSIZE=1024
2011-10-03 23:40:36.00 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:40:36.49 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:40:37.74 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:40:38.89 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:40:39.10 spid52    Starting up database 'ISALOG_20111002_WEB_000'.
2011-10-03 23:41:09.53 spid52    Starting up database 'ISALOG_20111001_FWS_000'.
2011-10-03 23:41:09.72 spid52    Starting up database 'ISALOG_20111002_FWS_000'.
2011-10-03 23:41:10.05 spid52    Starting up database 'ISALOG_20110930_WEB_000'.
2011-10-03 23:41:10.21 spid52    Starting up database 'ISALOG_20111001_WEB_000'.
2011-10-03 23:41:10.33 spid52    Starting up database


There seems to be some kind of VSS issue and the snapshot that VCB is trying to create seems to fail.

Any help is much appreciated
VMwareMicrosoft Server OS
Avatar of undefined
Last Comment
Lamont77
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image
A few checks, can you create a manual "take snapshot" with the following settings of the VM with the issues

 User generated image
A little background, all backup products rely on the Storage API, Snapshot process to Backup VMs. The problem is that this process, has always had issues, with quiesce guest file systems, due to the following:-

1.VMware Tools broken in the virtual machine, uninstalling VMware Tools completely, and re-installing after a reboot may help.

2. Slow datastore, multiple VMs on the same datastore, heavily loaded datastore, slow datastore subsystem, 7,200rpm, SATA RAID 5 or single disks.

3. A VM under heavy I/O, or Active Domain Controllers, Oracle and SQL databases, or database based servers such as Exchange, can also have issues.
Avatar of Lamont77
Lamont77

ASKER

There is no problem with manually creating snapshots in VC.

- The Data store is a 2TB store with roughyl 25 VM's on it. It is in alert as there is only 100GB left
- Cant confirm the type of disks at the moment
- The server in question is an ISA server and most of the errors are comming from VSS / SQL

Another question comes to mind. The Commvault logs are telling me that when it fails to create a snapshot due to quiesing ... it then tries to create a snapshot without quiescing and seems to be succesful:

5288   5 10/04 00:27:28 20783 ### CreateSnapshot_Task --- Started Create Snapshot task task-61876 for VM VMNAME 5288   5 10/04 00:28:41 20783 ### _WaitForTask --- Task [task-61876] status waited for [00:01:13.0853078] task took [00:01:12.0297270]   Wait [04/10/2011 00:27:28 to 04/10/2011 00:28:41]   Task [04/10/2011 00:27:31 to 04/10/2011 00:28:43]
5288   5 10/04 00:28:41 20783 ### CreateSnapshot_Task --- Failed to create Snapshot from VM VMNAME - Operation timed out.
5288 1d90 10/04 00:28:41 20783 CVMWareInfo::_CreateVMSnapshot() - Unable to create snapshot of VM [VMNAME], trying without quiesing filesystem

I think this means that it is still backing up OK. I need to investigate this a bit further though.
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image
So you've checked creating Snapshot as above?
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image
Yes, so the issue is possible with tools, of the frozen wait time has exceeded, and it's aborting snapshot.
Avatar of Netman66
Netman66
Flag of Canada image
If you say you have only 100GB left in that datastore, then that's likely your issue.

We had similar problems with low disk space - every VM that kicks off the backup needs a snapshot which is equivalent to the volume size it is snapping.  Too many VMs snapping at the same time consumed all the datastore capacity as the snapshots are saved with the VM files by default.

Try adding a datastore and reconfiguring the snapshots to use it instead to see if your issue goes away.
ASKER CERTIFIED SOLUTION
Avatar of Lamont77
Lamont77
Blurred text
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Avatar of Lamont77
Lamont77

ASKER

THis was the right answer
Microsoft Server OS
Microsoft Server OS

The Microsoft Server topic includes all of the legacy versions of the operating system, including the Windows NT 3.1, NT 3.5, NT 4.0 and Windows 2000 and Windows Home Server versions.

59K
Questions
--
Followers
--
Top Experts
Get a personalized solution from industry experts
Ask the experts

  • "Invaluable tool for our organization"

    Josh Regalski

  • "Your help has saved me hundreds of hours of internet surfing."

    @fblack61

  • "Just a dang good service!"

    @golferski

  • "Worth every penny."

    Steve Hougom

  • "It’s been a life saver."

    Maria Halt

  • "Best support site I’ve seen!"

    Bob Schneider

  • "I would be lost without them."

    @fblack61

  • "Saved my job multiple times."

    Walt Forbes

  • "I love this site."

    Maria Duwe

  • "Friendly respectful help."

    Andrew Kowtalo

  • "Such top-notch experts."

    @magento

  • "The best money I have ever spent."

    @rwheeler23

  • "Beyond any comprehensible value"

    George Morris

  • "Invaluable tool for our organization"

    Josh Regalski

Read over 600 more reviews

TRUSTED BY

IBM logoIntel logoMicrosoft logoUbisoft logoSAP logo
Qualcomm logoCitrix Systems logoWorkday logoErnst & Young logo
High performer badgeUsers love us badge
LinkedIn logoFacebook logoX logoInstagram logoTikTok logoYouTube logo
© 1996-2023 Experts Exchange, LLC. All rights reserved. Covered by US Patent