Solved

Can't ping/console into VMs : disk space issue

Posted on 2013-05-29
6
667 Views
Last Modified: 2013-05-30
I right-click a few UAT VMs last night, click Snapshot & then click "Take Snapshot":
most of the snapshot backup fails due to insufficient storage space.
At this point in time, the VMs are still pingable & can be RDP'ed into.

Then between app01 & app02 VMs I shared out the E: drives to each
other to take a Bare metal backup to save each other's Windows 2008
R2 bare metal backup to each other.

Then both the app01 & app02 VMs hanged & can't be pinged into anymore.
This morning, from vCenter, noticed I can't console into them as well,
giving an error message.

Can't Reboot, Reset, Shutdown nor Power Off the VMs from vCenter.
I tried to delete the Snapshot files (right click VMs, select Snapshot
Manager, select the snapshot file created last night & select Delete:
 it did not give any error but the Snapshot files are still there).

I ssh login to the ESXi host.  Below is a list of files for app01 VM
(yes, I short-formed the ESXi hostname to app01/02) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l |more
-rw-------    1 root     root         2919342080 Sep 28  2011 HHXXXVS2APP01-000001-delta.v
mdk
-rw-------    1 root     root                353 May 30 04:14 HHXXXVS2APP01-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01-000002-delta.v
mdk
-rw-------    1 root     root                330 Aug 13  2011 HHXXXVS2APP01-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01-000003-delta.v
mdk
-rw-------    1 root     root                330 Aug 16  2011 HHXXXVS2APP01-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01-000004-delta.v
mdk
-rw-------    1 root     root                330 Sep  3  2011 HHXXXVS2APP01-000004.vmdk
-rw-------    1 root     root        35383255040 May 29 10:10 HHXXXVS2APP01-000005-delta.v
mdk
-rw-------    1 root     root                360 May 30 04:14 HHXXXVS2APP01-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01-000006-delta.v
mdk
-rw-------    1 root     root                337 Sep 28  2011 HHXXXVS2APP01-000006.vmdk
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.v
mdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-r--r--    1 root     root                 65 Apr 26 11:28 HHXXXVS2APP01-6c60162a.hlog
-rw-------    1 root     root         8589934592 Aug 31  2012 HHXXXVS2APP01-6c60162a.vswp
-rw-------    1 root     root         8599829899 May 29 10:12 HHXXXVS2APP01-Snapshot13.vmsn
-rw-r--r--    1 root     root                 13 Aug 16  2011 HHXXXVS2APP01-aux.xml
-rw-------    1 root     root        53687091200 Sep  3  2011 HHXXXVS2APP01-flat.vmdk
-rw-------    1 root     root               8684 May 29 10:10 HHXXXVS2APP01.nvram
-rw-------    1 root     root                550 May 30 04:14 HHXXXVS2APP01.vmdk
-rw-r--r--    1 root     root                673 May 29 10:10 HHXXXVS2APP01.vmsd
-rwxr-xr-x    1 root     root               3905 May 29 10:10 HHXXXVS2APP01.vmx
-rw-r--r--    1 root     root                268 Apr 26 11:28 HHXXXVS2APP01.vmxf
-rw-------    1 root     root           67215360 Sep 24  2011 HHXXXVS2APP01_1-000001-delta.vmdk
-rw-------    1 root     root                357 May 30 04:14 HHXXXVS2APP01_1-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01_1-000002-delta.vmdk
-rw-------    1 root     root                334 Aug 13  2011 HHXXXVS2APP01_1-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01_1-000003-delta.vmdk
-rw-------    1 root     root                334 Aug 16  2011 HHXXXVS2APP01_1-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01_1-000004-delta.vmdk
-rw-------    1 root     root                334 Sep  3  2011 HHXXXVS2APP01_1-000004.vmdk
-rw-------    1 root     root        54073073664 May 29 10:10 HHXXXVS2APP01_1-000005-delta.vmdk
-rw-------    1 root     root                364 May 30 04:14 HHXXXVS2APP01_1-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01_1-000006-delta.vmdk
-rw-------    1 root     root                341 Sep 28  2011 HHXXXVS2APP01_1-000006.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root        53687091200 Aug 26  2011 HHXXXVS2APP01_1-flat.vmdk
-rw-------    1 root     root                552 May 30 04:14 HHXXXVS2APP01_1.vmdk
-rw-------    1 root     root           16801792 Sep 24  2011 HHXXXVS2APP01_2-000001-delta.vmdk
-rw-------    1 root     root                356 May 30 04:14 HHXXXVS2APP01_2-000001.vmdk
-rw-------    1 root     root           16801792 Aug 13  2011 HHXXXVS2APP01_2-000002-delta.vmdk
-rw-------    1 root     root                333 Aug 13  2011 HHXXXVS2APP01_2-000002.vmdk
-rw-------    1 root     root           16801792 Aug 16  2011 HHXXXVS2APP01_2-000003-delta.vmdk
-rw-------    1 root     root                333 Aug 16  2011 HHXXXVS2APP01_2-000003.vmdk
-rw-------    1 root     root           16801792 Sep  3  2011 HHXXXVS2APP01_2-000004-delta.vmdk
-rw-------    1 root     root                333 Sep  3  2011 HHXXXVS2APP01_2-000004.vmdk
-rw-------    1 root     root           16801792 May 29 10:10 HHXXXVS2APP01_2-000005-delta.vmdk
-rw-------    1 root     root                363 May 30 04:14 HHXXXVS2APP01_2-000005.vmdk
-rw-------    1 root     root           16801792 Sep 28  2011 HHXXXVS2APP01_2-000006-delta.vmdk
-rw-------    1 root     root                340 Sep 28  2011 HHXXXVS2APP01_2-000006.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk
-rw-------    1 root     root        10737418240 Aug 26  2011 HHXXXVS2APP01_2-flat.vmdk
-rw-------    1 root     root                551 May 30 04:14 HHXXXVS2APP01_2.vmdk
-rw-r--r--    1 root     root             436721 Aug 22  2012 vmware-10.log
-rw-r--r--    1 root     root             577066 Aug 31  2012 vmware-11.log
-rw-r--r--    1 root     root             458842 Dec 18 02:45 vmware-12.log
-rw-r--r--    1 root     root             153691 Jan  3 06:24 vmware-13.log
-rw-r--r--    1 root     root             193737 Apr 10 08:51 vmware-14.log
-rw-r--r--    1 root     root             161305 Apr 26 11:28 vmware-15.log
-rw-r--r--    1 root     root             169297 May 30 03:14 vmware.log
-rw-------    1 root     root           52428800 Apr 26 11:25 vmx-HHXXXVS2APP01-1818236458-1.vswp


How should I go about recovering the VMs?  Tried Storage vMotion as
well but no joy
0
Comment
Question by:sunhux
  • 3
  • 3
6 Comments
 

Author Comment

by:sunhux
Comment Utility
Is it safe to expand the files' sizes & which files (all from the list
below? ) should I expand:

vmware-vdiskmanager -x 10MB myDisk.vmdk


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l *0007*
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.vmdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk

Is it safe to try the suggestions given in link below:
  http://www.howtoforge.com/how-to-shrink-vmware-virtual-disk-files-vmdk
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 490 total points
Comment Utility
You cannot expand disks as you have stated, that is only for VMware Workstation.

Also, you cannot expand disks which have a snapshot attached.

Please see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would check if the current VM is still in Snapshot Mode, e.g. writing to a Snapshot disk, see my EE Article, and check.

and then we can work with you to get out of Snapshot Hell!
0
 

Author Comment

by:sunhux
Comment Utility
So as precaution, before doing Snapshot, have to check the free disk space
before commencing Snapshot?

Ok, for 3 of the VMs, under Summary tab, we click on "Cancel" & OK
& quickly go to Snapshot Manager to do "Delete All" the snapshots
& was able to boot up & recovered the VMs.

However, the fourth VM which has multiple snapshots, it took almost
2 hrs to do "Delete All" its snapshots (stuck at 99% for very long) &
after that's done, the VM can't boot up with 'insufficient space' as
ack'ed at the ESXi SSH command prompt:
mfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1

/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #

Now trying to do Datastore vMotion out one of the VM to free
up space in this datastore
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 490 total points
Comment Utility
*ALWAYS* have at least 20% of the datastore FREE!

Be careful using Snapshots, intentionally!

Yes, that's normal for Snapshot dseletion to take a long time, hence my Article BE PATIENT!

If you have run out of storage space, when Deleting a Snapshot, it's likely it could now be corrupted or needs additional handling!

Increase storage space, and then we can proceed!
0
 

Author Comment

by:sunhux
Comment Utility
Ok, the datastore vMotion frees up about 3-4% of space &
I could now boot up the last VM ;   phew

Just 1 last question & I'll close this thread:
Why is it after deleting away all my UAT Snapshots, I'm now
left with less than 3% free space (& this is after I've done
datastore vMotion to move out one UAT VM to a Prod datastore) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1059808346112  39434846208  96% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1068498944000  30744248320  97% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 490 total points
Comment Utility
Wait, and Be Patient, it can take a while for the datastore space to be refreshed.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Join & Write a Comment

Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now