Vsphere consolidation fails on a powered-down VM

Hi,

Got a consolidation warning on a recent clone of a VM this afternoon.  No snapshots are defined; tried consolidating and it threw an error about a file lock.  I powered it down and tried it again.  Got a slightly different error (attached).  

Did have a Veeam replication fail earlier at about the same timeframe.  It locked up the Veeam server so I can't tell which VM it was on at the time.  Not sure if that is pertinent but it might be.

How do I diagnose this one?  vSphere 6.0 environment.

Thanks!

--Ben
vsphere60.JPG
Ben ConnerCTO, SAS developerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Is the parent disk attached to the VM Server ? and hence why you cannot consolidate it, because it's locked e.g. the Veeam server has the lock on the parent VMDK.

Otherwise you will have to find the process which has the lock, or restart the HOST server to clear the lock.
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Just listed the files on that VM's folder and got the results below.  Definitely a lock there.  What's interesting is the timestamp on it--23:29.  Is it safe to try to delete that lock file?  

--Ben

-rw-------    1 root     root     8589934592 Nov 19 23:29 SQL12-7ce255fa.vswp
-rw-r--r--    1 root     root            13 Nov 19 23:06 SQL12-aux.xml
-rw-------    1 root     root          8684 Nov 19 23:30 SQL12.nvram
-rw-------    1 root     root            77 Nov 19 23:05 SQL12.vmsd
-rwx------    1 root     root          7270 Nov 19 23:29 SQL12.vmx
-rw-------    1 root     root             0 Nov 19 23:29 SQL12.vmx.lck
-rw-------    1 root     root          3370 Nov 19 18:18 SQL12.vmxf
-rwx------    1 root     root          7268 Nov 19 23:29 SQL12.vmx~
-rw-------    1 root     root       4260352 Nov 19 23:18 SQL2005B_4-000001-ctk.v
-rw-------    1 root     root      16912384 Nov 19 23:05 SQL2005B_4-000001-delta
-rw-------    1 root     root           416 Nov 19 23:05 SQL2005B_4-000001.vmdk
-rw-------    1 root     root       4260352 Nov 19 23:18 SQL2005B_4-000002-ctk.v
-rw-------    1 root     root      16912384 Nov 19 23:05 SQL2005B_4-000002-delta
-rw-------    1 root     root           423 Nov 19 23:05 SQL2005B_4-000002.vmdk
-rw-------    1 root     root       4260352 Nov 19 23:30 SQL2005B_4-000003-ctk.v
-rw-------    1 root     root      33689600 Nov 19 23:43 SQL2005B_4-000003-delta
-rw-------    1 root     root           400 Nov 19 23:29 SQL2005B_4-000003.vmdk
-rw-------    1 root     root       4260352 Nov 19 22:14 SQL2005B_4-ctk.vmdk
-rw-------    1 root     root     69793218560 Nov 19 22:14 SQL2005B_4-flat.vmdk
-rw-------    1 root     root           611 Nov 19 23:05 SQL2005B_4.vmdk
-rw-------    1 root     root       4915712 Nov 19 23:30 SQL2005B_5-ctk.vmdk
-rw-------    1 root     root     161061273600 Nov 19 23:43 SQL2005B_5-flat.vmdk
-rw-------    1 root     root           612 Nov 19 23:30 SQL2005B_5.vmdk
-rw-------    1 root     root        521190 Nov 19 23:28 vmware-1.log
-rw-------    1 root     root        236076 Nov 19 23:38 vmware.log
-rw-------    1 root     root     193986560 Nov 19 23:29 vmx-SQL12-2095207930-1.
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
The disks on this VM are local to the host it runs on.  DAS datastore on a RAID card.  And I've rebooted the Veeam server in the interim as well.

--Ben
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You have to look at the processes to check if there is a lock, not just check the files!

those files with -00000x.delta are snapshots!

and there are 3!

check disk settings of the VM...see my EE Article to check for if your current VM is writing to a snapshot disk....

HOW TO: VMware Snapshots :- Be Patient
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Yes, there are no snapshots listed in the Snapshot manager for this VM.  Are the best options at this point to clone the VM to a new one?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Please do not rely on looking in Snapshot Manager, it will not list them.

Please read my EE Article, which will show you how to check if your VM is currently writing to a snapshot!

Are the best options at this point to clone the VM to a new one?

Not going to help you I'm afraid at this point, you need to check if your VM is running on a snapshot.

and once you have that information, we can continue to debug your issue.

If you just keep randomly hitting buttons, things will get worse than better!
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
LOL...understood.

I am indeed running off snapshots on thsi VM.  It has 2 disks:

disk1: [HE8] SQL12_h/SQL2005B_4-000003.vmdk
disk2: [HE8] SQL12_h/SQL2005B_5.vmdk

Just peeked at the VM this was cloned from and it had the same issue; just didn't know it at the time.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, so the VM has two disks, and the VM is currently writing to snapshots, which I assume are shown in disk settings.

When the CONSOLIDATION WARNING appears, it's DUMB, it's just a watchdog, and if a file called 00000x.vmdk appears, it pops up the warning....

Just clicking consolidate without knowing WHY, can make it worse!

Please check your VEEAM VM, and check to see if it has the parent disks attached to it....

SQL2005B_4.vmdk
SQL2005B_5.vmdk

and remove them from the VM (do not delete from disk!)
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Hm.  How do I see if they are attached?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Right Click the VM, Edit Settings, and Look at the disks, like you did to see the snapshots attached to your SQL server!?
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Yes, that is where I got the 2 .vmdk filenames from...
The full descriptions for them are:
Disk 1: [HE8] SQL12_h/SQL2005B_4-000003.vmdk
Disk 2: [HE8] SQL12_h/SQL2005B_5.vmdk

I'm guessing just Disk 1 has the snapshot issue?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So now you need to check if the Veeam VM which does your backups, has the parent virtual disks attached.... (this is what could be causing the LOCK!)

Veeam has them attached...

So check your VM, which has Veeam installed, I assume it's a virtual machine ?

SQL2005B_4
SQL2005B_5
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Yes, but on a different host.  Let me check a couple of things...  I see an issue within Veeam for one of the managed servers it uses on the host shared with the SQL12 VM.  Let me take a look at that for a couple of minutes.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If no backups or replications are currently running on your Veeam VM, it should have no disks attached to it, other than it's own!

If the Veeam Server does not have these SQL parent disks attached, then the lock is elsewhere, and you will need to release the lock on these files before you can clear the snapshot.
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Ok, that's taken care of.  I have 2 Windows servers Veeam uses on that host.  Neither has any drives attached that aren't their own.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So it's time to research the locks on the files.

https://kb.vmware.com/s/article/10051

Until the locks have gone, you'll not be able to remove the snapshots.

(Restarting the hosts can sometimes be faster!)
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
I went through the configuration of all the VMs on this host and none of them have a disk attached other than their own.
Looking at that article now...
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
One of the things it suggests is looking through the vmware.log file; I see 2 in it, one with a -1.log suffix.  The one without the suffix has the most recent timestamp on it.  I didn't see anything obvious in either.
vmware.log
vmware-1.log
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you need to look at

Locating the lock and removing it, it can take some time, and sometimes all your effort is not fruitful, and restarting all ESXI hosts can be quicker!

Time is approaching 2.00GMT here in the UK, so I'm off to my bed, but will be back in 6 hours!
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
I assume I needed to run the vmkfstools command on the snapshot file name.  I ran the command:

vmkfstools -D /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-000003.vmdk

and got back:

Lock [type 10c00001 offset 230909952 v 2051, hb offset 4173824
gen 3, mode 0, owner 00000000-00000000-0000-000000000000 mtime 383126
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 357, 165>, gen 2035, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 400, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

Must need to do something else as the owner field is all 0s.  Also see no Read Only owner
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Following that article down further, I just ran the command:
vmkvsitools lsof | grep  SQL12_h

and got back:
138449      vmx                   FILE                       50   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL12.vmx.lck
138449      vmx                   FILE                       51   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL12.vmx
138449      vmx                   FILE                       52   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL12.vmx~
138449      vmx                   FILE                       53   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/vmware.log
138449      vmx                   FILE                       85   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-000003-delta.vmdk
138449      vmx                   FILE                       86   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_5-flat.vmdk
138449      vmx                   FILE                       87   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_5-ctk.vmdk
138449      vmx                   FILE                       88   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-000002-delta.vmdk
138449      vmx                   FILE                       89   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-000001-delta.vmdk
138449      vmx                   FILE                       91   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-flat.vmdk
138449      vmx                   FILE                       93   /vmfs/volumes/5a0ff70b-484f0b37-3be5-20cf301f235f/SQL12_h/SQL2005B_4-000003-ctk.vmdk

I'm not sure what to do with the output though.
0
 
Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:
Before doing anything on the VM check if there any VM virtual disks atteched in the Veeam or Veeam Proxy. This can happen and then there is lot of not deleted snaptshots and also is not possible to consolidate.

Check my article regarding this type of issue

https://www.experts-exchange.com/articles/29387/Veeam-Proxy-issue-Removing-Veeam-ghost-snapshots.html
0

Experts Exchange Solution brought to you by ConnectWise

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
 
Ben ConnerCTO, SAS developerAuthor Commented:
Hi Luciano,

That is working!  YES!   I'm guessing Veeam had a bad day weeks/months ago on this VM and it just now became an issue.  It is consolidating the snapshots now.

I will wait out the consolidation and then power it back up.

Much appreciated to both you and Andrew!  I always learn something each time I have to ask for help.

--Ben
0
 
Ben ConnerCTO, SAS developerAuthor Commented:
Thanks to both of you for pulling me out of the fire.  Again.

Great end to a stressful weekend.

--Ben
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.