Solved

Unable to backup Virtual Machine, quiescing!

Posted on 2013-06-19
60
5,699 Views
Last Modified: 2014-03-20
Hello

I have a problem trying to backup a VM.  THe error message happens in VSphere when it tries to take a snapshot

"The guest OS has reported an error during quiescing. The error code was: 4 The error message was: Quiesce aborted."

I did some research online and discovered that its a problem with the VM Tools.

I've tried:

>Restarting VM Tools
>Uninstall and reinstall
>Unintstall and reinstall using Custom and excluding the VSS option

Any idea how I can fix this error?

Thanks in advance
0
Comment
Question by:Tony
  • 30
  • 17
  • 13
60 Comments
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39258860
are you using any third party backup application ?

can you also post VSS writers status for the vm

to check vss writer status please run below command on command prompt

vssadmin list writers
0
 
LVL 1

Author Comment

by:Tony
ID: 39258870
Yes, we are using Netvault Backup

Its the first time we have come across this error message, and seems to have happened after the upgrade to vpshere 5.1

Which server do I run the command on?  THe one i'm trying to backup or the vcenter, or...?
0
 
LVL 118
ID: 39258879
This can be prepared, but also check any errors at the time of the snapshot, in the Event Logs of the OS.

Also does this error occur when you manually Snapshot the VM with the Quiesce option Ticked.

This can be resolved, but requires, Installation/Uninstallion of VMware Tools, at least 3 times.
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39258880
the one which you are trying to backup.
0
 
LVL 1

Author Comment

by:Tony
ID: 39259043
Attached is the VSS writers

It doesn't occur when I manually Snapshot the CM with the Quiesce option ticked

SHould I try removing VM Tools 3 times?


Edit:  I'm not able to upload for some reason...

Text for the writers is:

vssadmin 1.1 - Outil ligne de commande d'administration du service
de cliché instantané de volume
(C) Copyright 2001-2005 Microsoft Corp.

Nom du rédacteur : 'System Writer'
   ID du rédacteur : {e8132975-6f93-4464-a53e-1050253ae220}
   ID de l'instance du rédacteur : {e61f4d61-5a30-4973-8000-d158825b2529}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'ASR Writer'
   ID du rédacteur : {be000cbe-11fe-4426-9c58-531aa6355fc4}
   ID de l'instance du rédacteur : {6ffe3cf8-a4d7-4676-a340-17dbf0f0081a}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'Shadow Copy Optimization Writer'
   ID du rédacteur : {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   ID de l'instance du rédacteur : {51f44640-f1b1-4713-95e4-892cfd5aab1d}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'BITS Writer'
   ID du rédacteur : {4969d978-be47-48b0-b100-f328f07ac1e0}
   ID de l'instance du rédacteur : {b4e26107-1b3c-428b-82bf-0acac88ee05d}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'WMI Writer'
   ID du rédacteur : {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   ID de l'instance du rédacteur : {e18feac0-b606-4aaf-9d0a-858ec7e05c58}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'Registry Writer'
   ID du rédacteur : {afbab4a2-367d-4d15-a586-71dbb18f8485}
   ID de l'instance du rédacteur : {aa39de51-bc71-4172-aee0-f2e8b8969409}
   État : [1] Écurie
   Dernière erreur : Aucune erreur

Nom du rédacteur : 'COM+ REGDB Writer'
   ID du rédacteur : {542da469-d3e1-473c-9f4f-7847f01fc64f}
   ID de l'instance du rédacteur : {3b0ab6e5-bddb-467f-a3d2-91851f89801a}
   État : [1] Écurie
   Dernière erreur : Aucune erreur
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39259052
can not find attachment ,

yes you can try removing VM tools 3 times..
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39259059
All writers are stable . .are you seeing any errors in event log of the Vm ?


Also have you tried check disk on the vm ?
0
 
LVL 118
ID: 39259116
Yes,

1. Remove VMware Tools
2. Restart
3. Install VMware Tools

repeat above 3 times....
0
 
LVL 1

Author Comment

by:Tony
ID: 39259191
Thanks for the help

I've removed and reinstalled 3 times

The backup will run overnight tonight, and i'll report back tomorrow morning how it went

Thanks again
0
 
LVL 118
ID: 39259283
you should be able to check with a manual snapshot, because the backup will use the same function!
0
 
LVL 1

Author Comment

by:Tony
ID: 39265102
SOrry for the delay

After much testing it still isn't working.

The following is what I have tried in the past 2 days:

1) Remove/ReInstall VMTools approx 4 times in a row
2) Manual Snapshot works
3) Manual backup works
4) When scheduled backup ran overnight, the snapshot failed
5) In the morning, manual snapshot fails
6) Remove/Reinstall VM Tools again
7) Manual Snapshot works
8) Backup overnight fails again
9) This morning Manual Snapshot works fine

I can't work out why it seems to work during the day but not at night when the scheduled backup runs.

Is it worth just cloaning the machine to see if that works?
0
 
LVL 118
ID: 39265122
So Manual Snapshot with quiescing ticked? - always works?

are any other backups occuring at the same time on the same datastore?

any errors in vCenter Server in the events log?
0
 
LVL 1

Author Comment

by:Tony
ID: 39265171
It's intermittent.  Sometimes manual snapshot works, sometimes not.

Almost the entire vmware infrastructure is backed up nightly

No errors in vCenter Server at the start, during, or end of the nightly backups
0
 
LVL 118
ID: 39265281
We have seen issues when a datastore becomes under "pressure" high-load from Backups, VMware Snapshotting (which is used by many third party backup solutions) fails.

Also this can occur, if the VM is very busy, and has many transactions occurring, because VMware Snapshotting,  tries to "stun" freeze the VM writes to the disk.

What is the VM?
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265372
What is role of VM.. verify is any other backup schedule on the same time.. looks like 2 backups jobs are getting conflict with each other..
0
 
LVL 1

Author Comment

by:Tony
ID: 39265539
The server is for some software called "Numara".  It's basically a software distribution and inventory server, and isn't very busy at 11pm when the backup runs

We use backup system called "Netvault Backup", which is now owned by Dell.  It is a single backup job that backs up around 10 virtual machines, but does them one at a time
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265545
Okay can you check if Shadow copies are enable ?
0
 
LVL 1

Author Comment

by:Tony
ID: 39265578
Sorry, not sure what a Shadow copy is?  Do you mean check this on the vcenter server or the netvault backup server?

sorry...
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265591
Nope check if below setting enabled on the virtual machine

shadowcopies
0
 
LVL 1

Author Comment

by:Tony
ID: 39265597
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265601
in task manager any schedule is configured ? or running ?
0
 
LVL 1

Author Comment

by:Tony
ID: 39265614
No scheduled tasks configured or running
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265625
Last check if any error events occur during 10:50 to 11:15 PM in event viewer?
0
 
LVL 1

Author Comment

by:Tony
ID: 39265654
There are some error messages but the times don't match up with the times of the backup job
ScreenHunter-136-Jun.-21-14.27.jpg
ScreenHunter-137-Jun.-21-14.27.jpg
ScreenHunter-138-Jun.-21-14.27.jpg
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265680
backup job is schedule at 11 PM isnt?  but vm tool showing it got aborted at 0.52.. somgthing is really strange.. it should not take that much time if its schedule at 11pm
0
 
LVL 1

Author Comment

by:Tony
ID: 39265710
Backup job starts @ 22:00

The VM fails @ 22:55

Entire backup job finishes @ 01:45

Some of the VM's are quite large, and when the job finishes, it is only the single VM that has failed, all the rest are fine
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265730
after viewing vmtool error logs i found below link please check if that helps..

http://communities.vmware.com/thread/309844?start=15&tstart=0

refer markusepp comment :
0
 
LVL 1

Author Comment

by:Tony
ID: 39265802
Thanks, I followed the instructions in markusepp's comment, but the option is already set as FALSE for that VM

I've read through the entire posts, and there doesn't seem to be a solution
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39265876
What you think about @mulo reply ...  ( on same link )

installing Vmtool without VSS drivers.

As this server does not have any database ( SQL or MSDE) you can disable this try..

rest i am blank now .. do not find any other option to suggest.. or let other expert analyze this and respond.
0
 
LVL 1

Author Comment

by:Tony
ID: 39265886
Thanks very much for your help, but in my first post i've stated that i've already tried reinstalling the VM Tools without the VSS driver

Once again, thanks for your help though, it is appreciated

Need help from others :)
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 1

Author Comment

by:Tony
ID: 39270461
Can anyone else help with this issue?

At the weekend we take a backup of everything, and we have discovered that some VM's (that are only backed up at weekends) have the same problem as well
0
 
LVL 118
ID: 39270487
if you manually set a job to backup this VM - now, does it work?

The unity error message in the log, is normal, that's a VMware Tools (issue and bug)
0
 
LVL 1

Author Comment

by:Tony
ID: 39271083
Yes it works

I've deleted the entire backup and recreated it, will test overnight to see if that makes any difference
0
 
LVL 1

Author Comment

by:Tony
ID: 39289329
OK, very sorry for the delay in coming back to you, its been a busy week!

The scheduled backup overnight failed on the same vm with the same error message again.

We have another backup job that runs only at weekends, which backs up several VM's, and 6 of those failed with the same message as well.

I am specifically trying to resolve the problem on one VM first.  If i run a manual backup using netvault right now it fails with the snapshot error message.  If i try to manually take a snapshot in vsphere, that works fine.

Again i've tried removing vmware tools and reinstalling

Is it worth trying to clone the machine to see if i have the same problem on the clone?

I'm really running out of ideas and we now have 7 VM's that won't backup, all are critical...

Once again, sorry for the delay in responding, I will keep on top of it now
0
 
LVL 1

Author Comment

by:Tony
ID: 39289336
On another note, it appears that if I take a manual snapshot in vsphere, then delete that snapshot, then try the backup with netvault, it works fine and the snapshot with netvault works...
0
 
LVL 118
ID: 39289431
So is the issue, multiple backups?

datastore broadcast storm? (it does happen, if you saturate the datastore with backups!)
0
 
LVL 1

Author Comment

by:Tony
ID: 39289468
Yes, multiple backups.

Monday - Thursday:  we have a single backup job that does aprox 10 VM's, with 1 failing every day.

Friday Night:  We backup all VM's.  This is done in 2 backup jobs, starting 30 min apart from each other.  We have two backup drives, so each "job" has its own drive.

It has been like this for a few years but only recently, after upgrading to vsphere 5.1 have we seen this particular error message.

Sorry, what is a datastore broadcast storm?
0
 
LVL 22

Expert Comment

by:Haresh Nikumbh
ID: 39289475
in my first post i have mention this is getting failed because of multiple backup running for the vm.

you need to verify if both backups jobs are getting conflict. or make schedule tht way it wont conflict with each other.
0
 
LVL 118
ID: 39289476
datastore broadcast storm = datastore cannot cope with the amount or read and writes, and concurrent operations.

we have seen this many times, with backup applications, and have reduce the number of concurrent operations in the job, and create two or three jobs with less VM backups per job.

it may be worth experimenting with this.
0
 
LVL 1

Author Comment

by:Tony
ID: 39289488
Thanks

It just seems really odd that the problem is happening now.

The daily backup only does 10 VM's.  When it gets to the VM that always has the problem, it fails, and then moves on to the next VM without any issues.

Surely if it was a storm issue it would fail on the VM and any VM's after that one.

Even when the big Friday night backup runs, it may backup all the VM's but it only takes the snapshot and backup of a VM one at a time.  It performs a snapshot of VM-1, then back ups VM-1, then moves onto VM-2 and does the same thing.

With only 1 or 2 MAX VM's being backed up, why would this cause problems with the datastore?
0
 
LVL 118
ID: 39289492
that can depends on how busy that VM is, what it's doing, how many transactions that VM needs to write to disk, if the datastore is extremely busy, and VMware Tools, get's instruction to pause the VM (stun), it will abort, because it cannot pause the VM long enough or quick enough to gets the writes to disk.

depends on what configuration has changed, what SAN datastore you have.

are all VMs on the same LUN/datastore, are you using Jumbo Frames, multipath, iSCSI etc


@takecoffee in your first post....do not see any reference there!

it's likely this is the number of VMs being backuped at the same time, e.g. concurrent read and writes to the same datastore, e.g. datastore contention.
0
 
LVL 1

Author Comment

by:Tony
ID: 39289498
Yes it could be.

If I change the jobs, perhaps create more jobs with fewer VM's in each, isn't it going to be the same problem? If i have 1 job with 100 VM's, or 10 jobs with 10 VM's, isn't it going to cause the same problem.  Not to mention it would be a nightmare to work out when to schedule the start of each job for 100 VM's over a weekend
0
 
LVL 118
ID: 39289533
You need to evaluate your environment.
0
 
LVL 1

Author Comment

by:Tony
ID: 39299229
Rather embarrassingly I have discovered that all the VM's that have the problem, from either the weekday or weekend backups, are all on the same ESXi Host!

If i perform a test backup of one of the VM's it fails with the same error message.  I then migrate it to another host, run the backup job again, and it works fine.  If i then move the VM back to the original host, run the backup job again, it now works fine as well.

I've tried rebooting the host, without success, is it worth just formatting the hosts hdd and reinstalling 5.1?
0
 
LVL 118
ID: 39299244
how does this host differ to your other hosts?
0
 
LVL 1

Author Comment

by:Tony
ID: 39299250
It doesn't, its exactly the same hardware, software, and configuration

Only difference is that it obviously has a different name
0
 
LVL 118
ID: 39299350
I would rebuild. re-install, but before harvest logs.
0
 
LVL 1

Author Comment

by:Tony
ID: 39313493
An update to this:

I reinstalled the host, still have the problem

I then ran a complete format on the host's HDD, and installed esxi again, still have the same problem.

If we move the VM's to another host, they all backup on, so it is the host but I can't work out what it is on the host thats causing the problem.

Any ideas what to try next?
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 39313498
look at the logs, on the host, if anything is wrong, that you can see.
0
 
LVL 1

Author Comment

by:Tony
ID: 39316855
I've managed to find the log files, never looked before, but there are approx 15 different log files and most are pretty large.

Not sure which log file i should be looking at or even what I'm actually looking for...
0
 
LVL 118
ID: 39316891
what logs have you found?
0
 
LVL 1

Author Comment

by:Tony
ID: 39316986
auth.log
configRP.log
dhclient.log
esxupdate.log
fdm.log
hostd.log
hostprofiletrace.log
lacp.log
shell.log
storagerm.log
sysboot.log
syslog.log
usb.log
vmauthd.log
vmkdevmgr.log
vmkernel.log
vmkeventd.log
vmksummary.log
vmkwarning.log
vobd.log
vprobed.log
vpxa.log
0
 
LVL 118
ID: 39317066
lets have a look at the vmware.log in the vmware machine folder.
0
 
LVL 1

Author Comment

by:Tony
ID: 39317222
SOrry, how do I access that?

I'm using this to access the above log files:

https://hostname/host
0
 
LVL 118
ID: 39317296
if you browse the datastore, using the vSphere Client?
0
 
LVL 1

Author Comment

by:Tony
ID: 39317306
Yes, sorry

The latest log file of one of the vm's is attached
vmware.log
0
 
LVL 1

Author Comment

by:Tony
ID: 39328839
Sorry to be a pain but were you able to find anything in the log?
0
 
LVL 118
ID: 39328852
nothing that could explain the backup issue.
0
 
LVL 1

Author Comment

by:Tony
ID: 39328997
So I still don't know how to fix this problem...

I've now had to move all the VM's from the host to other hosts so that they can backup ok.  Which leaves me with a host not being used.

I've tried switching the IBM blade into another socket on the Management Module, just to check that it is a problem with the host, and yes it is.

Any ideas?
0
 
LVL 1

Author Closing Comment

by:Tony
ID: 39942576
All sorted
0

Featured Post

Free Gift Card with Acronis Backup Purchase!

Backup any data in any location: local and remote systems, physical and virtual servers, private and public clouds, Macs and PCs, tablets and mobile devices, & more! For limited time only, buy any Acronis backup products and get a FREE Amazon/Best Buy gift card worth up to $200!

Join & Write a Comment

Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
HOW TO: Install and Configure VMware vSphere Hypervisor 6.5 (ESXi 6.5), Step by Step Tutorial with screenshots. From Download, Checking Media, to Completed Installation.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now