[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

VMWare ESX 4 and using VCB Backup Snapshot remove failed

Posted on 2011-05-10
43
Medium Priority
?
3,058 Views
Last Modified: 2012-07-18
Hello,

I am trying to configure VMWare VCB backup in our environment however I am running at a "dead end" when trying to get a "sucesufll" backup using the VCB Framework.

VMWare ESX v.4.0.0 (B: 258672)
VMWare VCenter Server v4.0.0 (B: 258672)

Backup server is using VMWare Consolidated Backup Framework v1.5.0 (B: 226297), Windows 2008R2 x64 SP1 Enterprise OS.

The backup VM is on the same virtual infrastructure as the VM being backed up (the VM we are trying to backup is a fresh Windows 2008 R2 x64 SP1 with VMWare tools installed).

Before I start using our backup software (CA ArcServe r15) I am testing this at the VCB level to get a "successful" backup however it is failing and here is the log included:

C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework>vcbMounter.exe -h 172.16.4.17:9443 -u "vand1\administrator" -p J@ng0F3tt -a ipaddr:172.16.5.101 -r C:\EDI03 -t fullvm -m nbd

[2011-05-10 12:44:17.957 'App' 4072 info] Current working directory: C:\ProgramFiles (x86)\VMware\VMware Consolidated Backup Framework
[2011-05-10 12:44:17.959 'BaseLibs' 4072 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 1 numThreadsPerCore 1.
[2011-05-10 12:44:17.960 'BaseLibs' 4072 info] HOSTINFO: This machine has 2 physical CPUS, 2 total cores, and 2 logical CPUs.
[2011-05-10 12:44:18.498 'BaseLibs' 4072 info] Using system libcrypto, version 90709F
[2011-05-10 12:44:20.140 'BaseLibs' 4072 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: van-monitor.vand1.oppy.com vs 172.16.4.17
[2011-05-10 12:44:20.141 'BaseLibs' 4072 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

* The host name used for the connection does not match the subject name on the host certificate

* The host certificate chain is not complete.

[2011-05-10 12:44:20.143 'BaseLibs' 4072 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite theerror
Copying "[Fibre4] Van-EDI03/Van-EDI03.vmx":
        0%=====================50%=====================100%
        **************************************************

Copying "[Fibre4] Van-EDI03/Van-EDI03-aux.xml":
        0%=====================50%=====================100%
        **************************************************

Copying "[Fibre4] Van-EDI03/Van-EDI03.nvram":
        0%=====================50%=====================100%
        **************************************************

Copying "[Fibre4] Van-EDI03//vmware-1.log":
        0%=====================50%=====================100%
        *************************************************

Copying "[Fibre4] Van-EDI03//vmware-2.log":
        0%=====================50%=====================100%
        *************************************************

Copying "[Fibre4] Van-EDI03//vmware-3.log":
        0%=====================50%=====================100%
        **************************************************

Copying "[Fibre4] Van-EDI03//vmware-4.log":
        0%=====================50%=====================100%
        *************************************************

Copying "[Fibre4] Van-EDI03//vmware.log":
        0%=====================50%=====================100%
        **************************************************

Converting "C:\EDI03\scsi0-0-0-Van-EDI03.vmdk" (compact file):
        0%=====================50%=====================100%
        **************************************************

Converting "C:\EDI03\scsi0-1-0-Van-EDI03_1.vmdk" (compact file):
        0%=====================50%=====================100%
        **************************************************

[2011-05-10 13:30:52.567 'vcbMounter' 4072 warning] Snapshot deletion failed. Attempting to clean up snapshot database...
[2011-05-10 13:31:03.011 'BaseLibs' 2980 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: van-monitor.vand1.oppy.com vs 172.16.4.17
[2011-05-10 13:31:03.011 'BaseLibs' 2980 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

* The host name used for the connection does not match the subject name on the host certificate

* The host certificate chain is not complete.

[2011-05-10 13:31:03.011 'BaseLibs' 2980 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite theerror
[2011-05-10 13:31:25.479 'vcbMounter' 4072 error] Error: Other error encountered: Snapshot remove failed: Unable to access file <unspecified filename> since it is locked
[2011-05-10 13:31:25.479 'vcbMounter' 4072 error] An error occurred, cleaning up...
[2011-05-10 13:31:25.963 'vcbMounter' 4072 warning] Snapshot deletion failed. Attempting to clean up snapshot database...

Deleted directory C:\EDI03

C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework>

Open in new window


I think for the vcbConverter to consider a backup successful it tries to remove the VM snapshot, however this is the failing part; something is locking it on the VM Host? Thought, no idea what that is.

I do know that even if I go back to my vSphere Client and look at the snapshots for the particular VM, it shows there is a snapshot however it does not show it in the snapshot manager. If I create a new snapshot I can then see something different (screenshot attached).

To successfuly remove all the snapshots I turn the box off, move it to another VM Host, take a snapshot and then go back to Snapshot Manager and select "Delete All".

Anyone out there with a similar issue or any idea on how I can fix this locking issue? I have been on VMWare support for the last two days back and forth without any progress; wanted to run this by EE :)
0
Comment
Question by:mirde
  • 22
  • 18
  • 2
  • +1
43 Comments
 
LVL 124
ID: 35733076
Something to check here....

Can you manually take a Snapshot of the Virtual Machine, with the following options.

1. No tick in Snapshot Memory
2. Tick in the Quiece Virtual Machine

Let me know the result?
0
 
LVL 124
ID: 35733091
you've also got about 10+ Snapshots?

has this been caused by the VCB?
0
Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

 

Author Comment

by:mirde
ID: 35733096
Yes from VCB tests; nothing manual.
0
 
LVL 124
ID: 35733099
Is the VM on a SAN?
0
 
LVL 124
ID: 35733101
Can you manually take a Snapshot of the Virtual Machine, with the following options.

1. No tick in Snapshot Memory
2. Tick in the Quiece Virtual Machine

Let me know the result?
0
 

Author Comment

by:mirde
ID: 35733117
Both VM are the SAN (backup VM and the VM being backed up (VAN-EDI03).

The EDI03 is on Fibre, the Backup is on SATA.

Will let you know the results of VM Snapshot with the specific options shortly.
0
 
LVL 124
ID: 35733132
what is the SAN?

Does this happen with ALL VMs being backed up?

0
 

Author Comment

by:mirde
ID: 35733154
I have been using VAN-EDI03 as my test case, which is a fresh copy of Windows 2008 R2 x64 SP1.

Could try building another VM as I have Win2008R2 templates to deploy from.
0
 
LVL 124
ID: 35733157
what size are the LUNs on the SAN?
0
 

Author Comment

by:mirde
ID: 35733261
The LUN is are 2TB. Multiple LUNs in different Raid configuration (Raid 5, Raid 0).

Mix of FC Hard Drives and SATA Hard Drives for different LUNs.

Still waiting for a process to finish before I try the snapshot you requested.
0
 
LVL 124
ID: 35733273
size of vmdk?
0
 
LVL 124
ID: 35733285
okay 50 and 100GB, I can see that.

it's odd, something has a lock on the vmdk file, snapshot file which is preventing it from being deleted.
0
 

Author Comment

by:mirde
ID: 35733288
There are two disks:

DISK 1 - C:\, VMDK = 50GB Thick Provisioning
DISK 2 = D:\ VMDK = 100GB Thick Provisioning
0
 

Author Comment

by:mirde
ID: 35733292
Its almost as if the vcbConverter.exe that runs and creates the snapshot never releases it for the delete process.
0
 

Author Comment

by:mirde
ID: 35733316
Also, I did a Snapshot with:

1. No tick in Snapshot Memory
2. Tick in the Quiece Virtual Machine

Successful, and I can verify now by going to VM Properties the Hard Disk 1 & 2 are pointing to the VAN-EDI03-000001.vmdk and EDI03_1-000001.vmdk files.
0
 
LVL 124
ID: 35733325
well vcbconverter interfaces to the storage APIs, and the storage API creates and committs the snapshot.

just check these disks are not part of any other VM
0
 

Author Comment

by:mirde
ID: 35733339
Strange thing is, taking that snapshot and deleting it through vSphere Client has no issues, no locking error.

I think its isolated to vcbCenter and the storage API.

Is there a way to find out what is locking the file being logged into the VM Host of the VM Instance?
0
 
LVL 124
ID: 35733350
has VMware asked for a vm-support, or Support Log Bundle yet?

have they examine the logs?
0
 
LVL 124
ID: 35733356
you have have to start looking at logs.
0
 

Author Comment

by:mirde
ID: 35733366
Not yet they have not, initially I had issues bypassing the "SSL Handshake" issue which turns out because my port for where VCenter Server is is not default (443) bur rather 9443; now stuck on this error.

Going to push for escalation as I think I am still stuck in the loop of their primary /first line of support.
0
 

Author Comment

by:mirde
ID: 35733374
Any particular logs I should be looking at for this?
0
 
LVL 124
ID: 35733431
I'm surprised they've not had a moan at you about using ESX 4.0.0 and vCenter 4.0.0!

/var/log/vmkernel

/var/log/vmkwarning

0
 

Author Comment

by:mirde
ID: 35733448
We will be upgrading to the latest in the next few weeks; the server software was on a 2003 x86 box and 4.1 I think needs to be on something higher (OS or x64) which we can finally do now that best practice suggest VCenter software should reside on the virtual infrastructure itself and not a bare bone hardware.
0
 
LVL 124
ID: 35733458
you could do a tail -f kernel (latest kernel file)

and just before you do this, kick off a manual vcbMounter.exe
0
 
LVL 124
ID: 35733467
Yes VC does require 64 bit now.

We deploy all VCs on 64 bit VMs.
0
 
LVL 124
ID: 35733496
so this is 1.5 Build 2 of VCB?
0
 

Author Comment

by:mirde
ID: 35733500
Correct, the latest version of VCB Framework.
0
 
LVL 124
ID: 35733513
Have VMware agreed VCB 1.5 Build 2 is okay with ESX 4.0.0?

do you know if they have tested or confirmed this shouldn't be an issue?
0
 

Author Comment

by:mirde
ID: 35733524
I have relayed your last message to my open case with them, that is a good question; in a perfect world it should be backwards compatible right ? :) :)
0
 
LVL 124
ID: 35733528
yes, it's okay.
0
 
LVL 124
ID: 35733544
Well, what I mean it's suggested is compatible with ESX 4.0 and VC 4.0.

I knew that this was created for ESX 4.1 U1, because users complained because they are pulling support for the product.
0
 

Author Comment

by:mirde
ID: 35733558
Pulling support for which product? VCB?
0
 
LVL 124

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 35733620
Yes, there were.....

New: VMware has extended the end of availability timeline for VCB, and has added VCB support for vSphere 4.1. VMware will continue to support VCB 1.5 Update 2 for vSphere 4.1 and its subsequent update and patch releases through the end of their lifecycles. Support is extended in accordance with the VMware support policy for the VMware Infrastructure 3 platform. New major or minor versions of the vSphere platform beyond vSphere 4.1 will not be supported with VCB.

So after 4.1 no VCB, so with the next release of vSphere 5.0 (July/August 2011 (maybe slipping)).

End of Support for VCB was last year July 2010, but they've released new VCB 1.5 B2 for 4.0 U1/U2 and 4.1.

So now were are on Extended Support until - 2013/07/25

Technical Guidance Until - 2015/07/25.

But that's only helpful if you are going to stay on 4.1 until 2013!
0
 
LVL 124
ID: 35733637
probably need to sift through you logs. if there is a process lock on the disk file.
0
 

Author Comment

by:mirde
ID: 35733696
Interesting, though I tough VCB is a new replacement for VDDK right?

Maybe I am not using the right protocol for my backup? Any better suggestion? VCB? VDDK? VADP?

The product that will be ultimately running the nightly back-up job is CA ArcServe r15 (SP1) and is pretty current (2011). They are currently in beta with r16 that supports CBT (Change Block Tracking).

Maybe I should use VADP? Might have better luck with that.
0
 
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 35733766
VCB is as old as the hills.

It was VMware attempt of providing a backup.

But its not a backup product, just an enabler. it entered the market too late, third parties, vizioncore entered the market long before veeam with esxranger and took the market, many found it too complicated, slow, unreliable, and looked for alternatives, veeam and phd entered the market, with disk to disk based products which people could use to archivr to tape if neefbe, and now symantec and arcserve have products that can backup vms to tape without agents using the storage api.

vmware launch vdr, in esx 4.0, so where does that leave vcb.

unwanted and unsed. maybe.
0
 
LVL 16

Expert Comment

by:Danny McDaniel
ID: 35733843
the other nail poised to be hammered into VCB's coffin is that it depends upon Converter to do restores with and the last release of Converter was supposed to have removed support for VCB (but I can't recall if it went to release that way or not)
0
 

Author Comment

by:mirde
ID: 35734653
Interesting feedback, I think I just crossed off VCB off my list as its already started giving me trouble before I even tried backing up with ArcServe.

Though I tough ArcServe makes use of VCB to back-up VMWare VM.. will need to look at their guide to see what else I have available as I just renewed our maintenance and added ArcServe VMWare Backup Agents.
0
 
LVL 124
ID: 35734687
Don't get me wrong VCB is okay, I think it's just been overtaken by other vendors products.

It was never a full backup product. But remember this VMware *SNAPSHOTS* are a function of ALL Third Party backup products if you want backups of VMs. (at block level, out side of the VM). They all interface with the APIs to call backups.

Due to this very issue, years ago, we stopped using products that used the Snapshot process, because Snapshots are always troublesome. (they stick, stop, don't quices the VM), heavily loaded VMs can cause issues.

So we backup machines using SAN Based Snapshots, and we don't bother snapshotting the VM at the same time, and some people will flame and moan about this, and state your backups are therefore only crash consistent, e.g. as good as if the machine has crashed!

but that works for us, may not work for everybody, but it works for us. (and others!)
0
 

Author Closing Comment

by:mirde
ID: 35739775
Though not a "fix" for my issue, it pointed me to the right process; taking a look at the logs and pushing VMWare support to look at them.

Also, suggested alternatives, such as VDDK and described the technology as it evolved.

If I could add more than 500pts to you I would.

Thanks.
0
 

Expert Comment

by:koolkid1976
ID: 38011931
Your files are being locked by the CA services on your vCenter server. To delet the snapshots, go into your vCenter server and stop all services starting with CA. Then use the vCenter console to take another snapshot of the VM that has the locked snapshot file. Then go to Snapshot Manger and "Delete All". This should delete the snapshot you just made, as well as the other previously locked snapshot/s.
If that does not work, you'll have to migrate your VM to another ESX host, make another snapshot, then select "Delete All."
0
 

Expert Comment

by:koolkid1976
ID: 38198839
Updating to ESX 5.0 resolved it all!
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Microsoft will be releasing the Windows 10 Creators Update in just a matter of weeks. Are you prepared? Follow these steps to ensure everything goes smoothly and you don't lose valuable data on your PC.
This article shows how to use a free utility called 'Parkdale' to easily test the performance and benchmark any Hard Drive(s) installed in your computer. We also look at RAM Disks and their speed comparisons.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question