?
Solved

Can't ping/console into VMs : disk space issue

Posted on 2013-05-29
6
Medium Priority
?
690 Views
Last Modified: 2013-05-30
I right-click a few UAT VMs last night, click Snapshot & then click "Take Snapshot":
most of the snapshot backup fails due to insufficient storage space.
At this point in time, the VMs are still pingable & can be RDP'ed into.

Then between app01 & app02 VMs I shared out the E: drives to each
other to take a Bare metal backup to save each other's Windows 2008
R2 bare metal backup to each other.

Then both the app01 & app02 VMs hanged & can't be pinged into anymore.
This morning, from vCenter, noticed I can't console into them as well,
giving an error message.

Can't Reboot, Reset, Shutdown nor Power Off the VMs from vCenter.
I tried to delete the Snapshot files (right click VMs, select Snapshot
Manager, select the snapshot file created last night & select Delete:
 it did not give any error but the Snapshot files are still there).

I ssh login to the ESXi host.  Below is a list of files for app01 VM
(yes, I short-formed the ESXi hostname to app01/02) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l |more
-rw-------    1 root     root         2919342080 Sep 28  2011 HHXXXVS2APP01-000001-delta.v
mdk
-rw-------    1 root     root                353 May 30 04:14 HHXXXVS2APP01-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01-000002-delta.v
mdk
-rw-------    1 root     root                330 Aug 13  2011 HHXXXVS2APP01-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01-000003-delta.v
mdk
-rw-------    1 root     root                330 Aug 16  2011 HHXXXVS2APP01-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01-000004-delta.v
mdk
-rw-------    1 root     root                330 Sep  3  2011 HHXXXVS2APP01-000004.vmdk
-rw-------    1 root     root        35383255040 May 29 10:10 HHXXXVS2APP01-000005-delta.v
mdk
-rw-------    1 root     root                360 May 30 04:14 HHXXXVS2APP01-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01-000006-delta.v
mdk
-rw-------    1 root     root                337 Sep 28  2011 HHXXXVS2APP01-000006.vmdk
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.v
mdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-r--r--    1 root     root                 65 Apr 26 11:28 HHXXXVS2APP01-6c60162a.hlog
-rw-------    1 root     root         8589934592 Aug 31  2012 HHXXXVS2APP01-6c60162a.vswp
-rw-------    1 root     root         8599829899 May 29 10:12 HHXXXVS2APP01-Snapshot13.vmsn
-rw-r--r--    1 root     root                 13 Aug 16  2011 HHXXXVS2APP01-aux.xml
-rw-------    1 root     root        53687091200 Sep  3  2011 HHXXXVS2APP01-flat.vmdk
-rw-------    1 root     root               8684 May 29 10:10 HHXXXVS2APP01.nvram
-rw-------    1 root     root                550 May 30 04:14 HHXXXVS2APP01.vmdk
-rw-r--r--    1 root     root                673 May 29 10:10 HHXXXVS2APP01.vmsd
-rwxr-xr-x    1 root     root               3905 May 29 10:10 HHXXXVS2APP01.vmx
-rw-r--r--    1 root     root                268 Apr 26 11:28 HHXXXVS2APP01.vmxf
-rw-------    1 root     root           67215360 Sep 24  2011 HHXXXVS2APP01_1-000001-delta.vmdk
-rw-------    1 root     root                357 May 30 04:14 HHXXXVS2APP01_1-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01_1-000002-delta.vmdk
-rw-------    1 root     root                334 Aug 13  2011 HHXXXVS2APP01_1-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01_1-000003-delta.vmdk
-rw-------    1 root     root                334 Aug 16  2011 HHXXXVS2APP01_1-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01_1-000004-delta.vmdk
-rw-------    1 root     root                334 Sep  3  2011 HHXXXVS2APP01_1-000004.vmdk
-rw-------    1 root     root        54073073664 May 29 10:10 HHXXXVS2APP01_1-000005-delta.vmdk
-rw-------    1 root     root                364 May 30 04:14 HHXXXVS2APP01_1-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01_1-000006-delta.vmdk
-rw-------    1 root     root                341 Sep 28  2011 HHXXXVS2APP01_1-000006.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root        53687091200 Aug 26  2011 HHXXXVS2APP01_1-flat.vmdk
-rw-------    1 root     root                552 May 30 04:14 HHXXXVS2APP01_1.vmdk
-rw-------    1 root     root           16801792 Sep 24  2011 HHXXXVS2APP01_2-000001-delta.vmdk
-rw-------    1 root     root                356 May 30 04:14 HHXXXVS2APP01_2-000001.vmdk
-rw-------    1 root     root           16801792 Aug 13  2011 HHXXXVS2APP01_2-000002-delta.vmdk
-rw-------    1 root     root                333 Aug 13  2011 HHXXXVS2APP01_2-000002.vmdk
-rw-------    1 root     root           16801792 Aug 16  2011 HHXXXVS2APP01_2-000003-delta.vmdk
-rw-------    1 root     root                333 Aug 16  2011 HHXXXVS2APP01_2-000003.vmdk
-rw-------    1 root     root           16801792 Sep  3  2011 HHXXXVS2APP01_2-000004-delta.vmdk
-rw-------    1 root     root                333 Sep  3  2011 HHXXXVS2APP01_2-000004.vmdk
-rw-------    1 root     root           16801792 May 29 10:10 HHXXXVS2APP01_2-000005-delta.vmdk
-rw-------    1 root     root                363 May 30 04:14 HHXXXVS2APP01_2-000005.vmdk
-rw-------    1 root     root           16801792 Sep 28  2011 HHXXXVS2APP01_2-000006-delta.vmdk
-rw-------    1 root     root                340 Sep 28  2011 HHXXXVS2APP01_2-000006.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk
-rw-------    1 root     root        10737418240 Aug 26  2011 HHXXXVS2APP01_2-flat.vmdk
-rw-------    1 root     root                551 May 30 04:14 HHXXXVS2APP01_2.vmdk
-rw-r--r--    1 root     root             436721 Aug 22  2012 vmware-10.log
-rw-r--r--    1 root     root             577066 Aug 31  2012 vmware-11.log
-rw-r--r--    1 root     root             458842 Dec 18 02:45 vmware-12.log
-rw-r--r--    1 root     root             153691 Jan  3 06:24 vmware-13.log
-rw-r--r--    1 root     root             193737 Apr 10 08:51 vmware-14.log
-rw-r--r--    1 root     root             161305 Apr 26 11:28 vmware-15.log
-rw-r--r--    1 root     root             169297 May 30 03:14 vmware.log
-rw-------    1 root     root           52428800 Apr 26 11:25 vmx-HHXXXVS2APP01-1818236458-1.vswp


How should I go about recovering the VMs?  Tried Storage vMotion as
well but no joy
0
Comment
Question by:sunhux
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 

Author Comment

by:sunhux
ID: 39206802
Is it safe to expand the files' sizes & which files (all from the list
below? ) should I expand:

vmware-vdiskmanager -x 10MB myDisk.vmdk


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l *0007*
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.vmdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk

Is it safe to try the suggestions given in link below:
  http://www.howtoforge.com/how-to-shrink-vmware-virtual-disk-files-vmdk
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1960 total points
ID: 39206823
You cannot expand disks as you have stated, that is only for VMware Workstation.

Also, you cannot expand disks which have a snapshot attached.

Please see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would check if the current VM is still in Snapshot Mode, e.g. writing to a Snapshot disk, see my EE Article, and check.

and then we can work with you to get out of Snapshot Hell!
0
 

Author Comment

by:sunhux
ID: 39207008
So as precaution, before doing Snapshot, have to check the free disk space
before commencing Snapshot?

Ok, for 3 of the VMs, under Summary tab, we click on "Cancel" & OK
& quickly go to Snapshot Manager to do "Delete All" the snapshots
& was able to boot up & recovered the VMs.

However, the fourth VM which has multiple snapshots, it took almost
2 hrs to do "Delete All" its snapshots (stuck at 99% for very long) &
after that's done, the VM can't boot up with 'insufficient space' as
ack'ed at the ESXi SSH command prompt:
mfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1

/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #

Now trying to do Datastore vMotion out one of the VM to free
up space in this datastore
0
Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1960 total points
ID: 39207122
*ALWAYS* have at least 20% of the datastore FREE!

Be careful using Snapshots, intentionally!

Yes, that's normal for Snapshot dseletion to take a long time, hence my Article BE PATIENT!

If you have run out of storage space, when Deleting a Snapshot, it's likely it could now be corrupted or needs additional handling!

Increase storage space, and then we can proceed!
0
 

Author Comment

by:sunhux
ID: 39207152
Ok, the datastore vMotion frees up about 3-4% of space &
I could now boot up the last VM ;   phew

Just 1 last question & I'll close this thread:
Why is it after deleting away all my UAT Snapshots, I'm now
left with less than 3% free space (& this is after I've done
datastore vMotion to move out one UAT VM to a Prod datastore) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1059808346112  39434846208  96% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1068498944000  30744248320  97% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #
0
 
LVL 123

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1960 total points
ID: 39207244
Wait, and Be Patient, it can take a while for the datastore space to be refreshed.
0

Featured Post

Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A look into Log Analysis and Effective Critical Alerting.
Is your phone running out of space to hold pictures?  This article will show you quick tips on how to solve this problem.
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question