Can't ping/console into VMs : disk space issue

I right-click a few UAT VMs last night, click Snapshot & then click "Take Snapshot":
most of the snapshot backup fails due to insufficient storage space.
At this point in time, the VMs are still pingable & can be RDP'ed into.

Then between app01 & app02 VMs I shared out the E: drives to each
other to take a Bare metal backup to save each other's Windows 2008
R2 bare metal backup to each other.

Then both the app01 & app02 VMs hanged & can't be pinged into anymore.
This morning, from vCenter, noticed I can't console into them as well,
giving an error message.

Can't Reboot, Reset, Shutdown nor Power Off the VMs from vCenter.
I tried to delete the Snapshot files (right click VMs, select Snapshot
Manager, select the snapshot file created last night & select Delete:
 it did not give any error but the Snapshot files are still there).

I ssh login to the ESXi host.  Below is a list of files for app01 VM
(yes, I short-formed the ESXi hostname to app01/02) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l |more
-rw-------    1 root     root         2919342080 Sep 28  2011 HHXXXVS2APP01-000001-delta.v
mdk
-rw-------    1 root     root                353 May 30 04:14 HHXXXVS2APP01-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01-000002-delta.v
mdk
-rw-------    1 root     root                330 Aug 13  2011 HHXXXVS2APP01-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01-000003-delta.v
mdk
-rw-------    1 root     root                330 Aug 16  2011 HHXXXVS2APP01-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01-000004-delta.v
mdk
-rw-------    1 root     root                330 Sep  3  2011 HHXXXVS2APP01-000004.vmdk
-rw-------    1 root     root        35383255040 May 29 10:10 HHXXXVS2APP01-000005-delta.v
mdk
-rw-------    1 root     root                360 May 30 04:14 HHXXXVS2APP01-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01-000006-delta.v
mdk
-rw-------    1 root     root                337 Sep 28  2011 HHXXXVS2APP01-000006.vmdk
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.v
mdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-r--r--    1 root     root                 65 Apr 26 11:28 HHXXXVS2APP01-6c60162a.hlog
-rw-------    1 root     root         8589934592 Aug 31  2012 HHXXXVS2APP01-6c60162a.vswp
-rw-------    1 root     root         8599829899 May 29 10:12 HHXXXVS2APP01-Snapshot13.vmsn
-rw-r--r--    1 root     root                 13 Aug 16  2011 HHXXXVS2APP01-aux.xml
-rw-------    1 root     root        53687091200 Sep  3  2011 HHXXXVS2APP01-flat.vmdk
-rw-------    1 root     root               8684 May 29 10:10 HHXXXVS2APP01.nvram
-rw-------    1 root     root                550 May 30 04:14 HHXXXVS2APP01.vmdk
-rw-r--r--    1 root     root                673 May 29 10:10 HHXXXVS2APP01.vmsd
-rwxr-xr-x    1 root     root               3905 May 29 10:10 HHXXXVS2APP01.vmx
-rw-r--r--    1 root     root                268 Apr 26 11:28 HHXXXVS2APP01.vmxf
-rw-------    1 root     root           67215360 Sep 24  2011 HHXXXVS2APP01_1-000001-delta.vmdk
-rw-------    1 root     root                357 May 30 04:14 HHXXXVS2APP01_1-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01_1-000002-delta.vmdk
-rw-------    1 root     root                334 Aug 13  2011 HHXXXVS2APP01_1-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01_1-000003-delta.vmdk
-rw-------    1 root     root                334 Aug 16  2011 HHXXXVS2APP01_1-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01_1-000004-delta.vmdk
-rw-------    1 root     root                334 Sep  3  2011 HHXXXVS2APP01_1-000004.vmdk
-rw-------    1 root     root        54073073664 May 29 10:10 HHXXXVS2APP01_1-000005-delta.vmdk
-rw-------    1 root     root                364 May 30 04:14 HHXXXVS2APP01_1-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01_1-000006-delta.vmdk
-rw-------    1 root     root                341 Sep 28  2011 HHXXXVS2APP01_1-000006.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root        53687091200 Aug 26  2011 HHXXXVS2APP01_1-flat.vmdk
-rw-------    1 root     root                552 May 30 04:14 HHXXXVS2APP01_1.vmdk
-rw-------    1 root     root           16801792 Sep 24  2011 HHXXXVS2APP01_2-000001-delta.vmdk
-rw-------    1 root     root                356 May 30 04:14 HHXXXVS2APP01_2-000001.vmdk
-rw-------    1 root     root           16801792 Aug 13  2011 HHXXXVS2APP01_2-000002-delta.vmdk
-rw-------    1 root     root                333 Aug 13  2011 HHXXXVS2APP01_2-000002.vmdk
-rw-------    1 root     root           16801792 Aug 16  2011 HHXXXVS2APP01_2-000003-delta.vmdk
-rw-------    1 root     root                333 Aug 16  2011 HHXXXVS2APP01_2-000003.vmdk
-rw-------    1 root     root           16801792 Sep  3  2011 HHXXXVS2APP01_2-000004-delta.vmdk
-rw-------    1 root     root                333 Sep  3  2011 HHXXXVS2APP01_2-000004.vmdk
-rw-------    1 root     root           16801792 May 29 10:10 HHXXXVS2APP01_2-000005-delta.vmdk
-rw-------    1 root     root                363 May 30 04:14 HHXXXVS2APP01_2-000005.vmdk
-rw-------    1 root     root           16801792 Sep 28  2011 HHXXXVS2APP01_2-000006-delta.vmdk
-rw-------    1 root     root                340 Sep 28  2011 HHXXXVS2APP01_2-000006.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk
-rw-------    1 root     root        10737418240 Aug 26  2011 HHXXXVS2APP01_2-flat.vmdk
-rw-------    1 root     root                551 May 30 04:14 HHXXXVS2APP01_2.vmdk
-rw-r--r--    1 root     root             436721 Aug 22  2012 vmware-10.log
-rw-r--r--    1 root     root             577066 Aug 31  2012 vmware-11.log
-rw-r--r--    1 root     root             458842 Dec 18 02:45 vmware-12.log
-rw-r--r--    1 root     root             153691 Jan  3 06:24 vmware-13.log
-rw-r--r--    1 root     root             193737 Apr 10 08:51 vmware-14.log
-rw-r--r--    1 root     root             161305 Apr 26 11:28 vmware-15.log
-rw-r--r--    1 root     root             169297 May 30 03:14 vmware.log
-rw-------    1 root     root           52428800 Apr 26 11:25 vmx-HHXXXVS2APP01-1818236458-1.vswp


How should I go about recovering the VMs?  Tried Storage vMotion as
well but no joy
sunhuxAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
You cannot expand disks as you have stated, that is only for VMware Workstation.

Also, you cannot expand disks which have a snapshot attached.

Please see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would check if the current VM is still in Snapshot Mode, e.g. writing to a Snapshot disk, see my EE Article, and check.

and then we can work with you to get out of Snapshot Hell!
0
 
sunhuxAuthor Commented:
Is it safe to expand the files' sizes & which files (all from the list
below? ) should I expand:

vmware-vdiskmanager -x 10MB myDisk.vmdk


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l *0007*
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.vmdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk

Is it safe to try the suggestions given in link below:
  http://www.howtoforge.com/how-to-shrink-vmware-virtual-disk-files-vmdk
0
 
sunhuxAuthor Commented:
So as precaution, before doing Snapshot, have to check the free disk space
before commencing Snapshot?

Ok, for 3 of the VMs, under Summary tab, we click on "Cancel" & OK
& quickly go to Snapshot Manager to do "Delete All" the snapshots
& was able to boot up & recovered the VMs.

However, the fourth VM which has multiple snapshots, it took almost
2 hrs to do "Delete All" its snapshots (stuck at 99% for very long) &
after that's done, the VM can't boot up with 'insufficient space' as
ack'ed at the ESXi SSH command prompt:
mfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1

/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #

Now trying to do Datastore vMotion out one of the VM to free
up space in this datastore
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
*ALWAYS* have at least 20% of the datastore FREE!

Be careful using Snapshots, intentionally!

Yes, that's normal for Snapshot dseletion to take a long time, hence my Article BE PATIENT!

If you have run out of storage space, when Deleting a Snapshot, it's likely it could now be corrupted or needs additional handling!

Increase storage space, and then we can proceed!
0
 
sunhuxAuthor Commented:
Ok, the datastore vMotion frees up about 3-4% of space &
I could now boot up the last VM ;   phew

Just 1 last question & I'll close this thread:
Why is it after deleting away all my UAT Snapshots, I'm now
left with less than 3% free space (& this is after I've done
datastore vMotion to move out one UAT VM to a Prod datastore) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1059808346112  39434846208  96% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1068498944000  30744248320  97% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Wait, and Be Patient, it can take a while for the datastore space to be refreshed.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.