Solved

Can't ping/console into VMs : disk space issue

Posted on 2013-05-29
6
680 Views
Last Modified: 2013-05-30
I right-click a few UAT VMs last night, click Snapshot & then click "Take Snapshot":
most of the snapshot backup fails due to insufficient storage space.
At this point in time, the VMs are still pingable & can be RDP'ed into.

Then between app01 & app02 VMs I shared out the E: drives to each
other to take a Bare metal backup to save each other's Windows 2008
R2 bare metal backup to each other.

Then both the app01 & app02 VMs hanged & can't be pinged into anymore.
This morning, from vCenter, noticed I can't console into them as well,
giving an error message.

Can't Reboot, Reset, Shutdown nor Power Off the VMs from vCenter.
I tried to delete the Snapshot files (right click VMs, select Snapshot
Manager, select the snapshot file created last night & select Delete:
 it did not give any error but the Snapshot files are still there).

I ssh login to the ESXi host.  Below is a list of files for app01 VM
(yes, I short-formed the ESXi hostname to app01/02) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l |more
-rw-------    1 root     root         2919342080 Sep 28  2011 HHXXXVS2APP01-000001-delta.v
mdk
-rw-------    1 root     root                353 May 30 04:14 HHXXXVS2APP01-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01-000002-delta.v
mdk
-rw-------    1 root     root                330 Aug 13  2011 HHXXXVS2APP01-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01-000003-delta.v
mdk
-rw-------    1 root     root                330 Aug 16  2011 HHXXXVS2APP01-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01-000004-delta.v
mdk
-rw-------    1 root     root                330 Sep  3  2011 HHXXXVS2APP01-000004.vmdk
-rw-------    1 root     root        35383255040 May 29 10:10 HHXXXVS2APP01-000005-delta.v
mdk
-rw-------    1 root     root                360 May 30 04:14 HHXXXVS2APP01-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01-000006-delta.v
mdk
-rw-------    1 root     root                337 Sep 28  2011 HHXXXVS2APP01-000006.vmdk
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.v
mdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-r--r--    1 root     root                 65 Apr 26 11:28 HHXXXVS2APP01-6c60162a.hlog
-rw-------    1 root     root         8589934592 Aug 31  2012 HHXXXVS2APP01-6c60162a.vswp
-rw-------    1 root     root         8599829899 May 29 10:12 HHXXXVS2APP01-Snapshot13.vmsn
-rw-r--r--    1 root     root                 13 Aug 16  2011 HHXXXVS2APP01-aux.xml
-rw-------    1 root     root        53687091200 Sep  3  2011 HHXXXVS2APP01-flat.vmdk
-rw-------    1 root     root               8684 May 29 10:10 HHXXXVS2APP01.nvram
-rw-------    1 root     root                550 May 30 04:14 HHXXXVS2APP01.vmdk
-rw-r--r--    1 root     root                673 May 29 10:10 HHXXXVS2APP01.vmsd
-rwxr-xr-x    1 root     root               3905 May 29 10:10 HHXXXVS2APP01.vmx
-rw-r--r--    1 root     root                268 Apr 26 11:28 HHXXXVS2APP01.vmxf
-rw-------    1 root     root           67215360 Sep 24  2011 HHXXXVS2APP01_1-000001-delta.vmdk
-rw-------    1 root     root                357 May 30 04:14 HHXXXVS2APP01_1-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01_1-000002-delta.vmdk
-rw-------    1 root     root                334 Aug 13  2011 HHXXXVS2APP01_1-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01_1-000003-delta.vmdk
-rw-------    1 root     root                334 Aug 16  2011 HHXXXVS2APP01_1-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01_1-000004-delta.vmdk
-rw-------    1 root     root                334 Sep  3  2011 HHXXXVS2APP01_1-000004.vmdk
-rw-------    1 root     root        54073073664 May 29 10:10 HHXXXVS2APP01_1-000005-delta.vmdk
-rw-------    1 root     root                364 May 30 04:14 HHXXXVS2APP01_1-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01_1-000006-delta.vmdk
-rw-------    1 root     root                341 Sep 28  2011 HHXXXVS2APP01_1-000006.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root        53687091200 Aug 26  2011 HHXXXVS2APP01_1-flat.vmdk
-rw-------    1 root     root                552 May 30 04:14 HHXXXVS2APP01_1.vmdk
-rw-------    1 root     root           16801792 Sep 24  2011 HHXXXVS2APP01_2-000001-delta.vmdk
-rw-------    1 root     root                356 May 30 04:14 HHXXXVS2APP01_2-000001.vmdk
-rw-------    1 root     root           16801792 Aug 13  2011 HHXXXVS2APP01_2-000002-delta.vmdk
-rw-------    1 root     root                333 Aug 13  2011 HHXXXVS2APP01_2-000002.vmdk
-rw-------    1 root     root           16801792 Aug 16  2011 HHXXXVS2APP01_2-000003-delta.vmdk
-rw-------    1 root     root                333 Aug 16  2011 HHXXXVS2APP01_2-000003.vmdk
-rw-------    1 root     root           16801792 Sep  3  2011 HHXXXVS2APP01_2-000004-delta.vmdk
-rw-------    1 root     root                333 Sep  3  2011 HHXXXVS2APP01_2-000004.vmdk
-rw-------    1 root     root           16801792 May 29 10:10 HHXXXVS2APP01_2-000005-delta.vmdk
-rw-------    1 root     root                363 May 30 04:14 HHXXXVS2APP01_2-000005.vmdk
-rw-------    1 root     root           16801792 Sep 28  2011 HHXXXVS2APP01_2-000006-delta.vmdk
-rw-------    1 root     root                340 Sep 28  2011 HHXXXVS2APP01_2-000006.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk
-rw-------    1 root     root        10737418240 Aug 26  2011 HHXXXVS2APP01_2-flat.vmdk
-rw-------    1 root     root                551 May 30 04:14 HHXXXVS2APP01_2.vmdk
-rw-r--r--    1 root     root             436721 Aug 22  2012 vmware-10.log
-rw-r--r--    1 root     root             577066 Aug 31  2012 vmware-11.log
-rw-r--r--    1 root     root             458842 Dec 18 02:45 vmware-12.log
-rw-r--r--    1 root     root             153691 Jan  3 06:24 vmware-13.log
-rw-r--r--    1 root     root             193737 Apr 10 08:51 vmware-14.log
-rw-r--r--    1 root     root             161305 Apr 26 11:28 vmware-15.log
-rw-r--r--    1 root     root             169297 May 30 03:14 vmware.log
-rw-------    1 root     root           52428800 Apr 26 11:25 vmx-HHXXXVS2APP01-1818236458-1.vswp


How should I go about recovering the VMs?  Tried Storage vMotion as
well but no joy
0
Comment
Question by:sunhux
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 

Author Comment

by:sunhux
ID: 39206802
Is it safe to expand the files' sizes & which files (all from the list
below? ) should I expand:

vmware-vdiskmanager -x 10MB myDisk.vmdk


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l *0007*
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.vmdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk

Is it safe to try the suggestions given in link below:
  http://www.howtoforge.com/how-to-shrink-vmware-virtual-disk-files-vmdk
0
 
LVL 120

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39206823
You cannot expand disks as you have stated, that is only for VMware Workstation.

Also, you cannot expand disks which have a snapshot attached.

Please see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would check if the current VM is still in Snapshot Mode, e.g. writing to a Snapshot disk, see my EE Article, and check.

and then we can work with you to get out of Snapshot Hell!
0
 

Author Comment

by:sunhux
ID: 39207008
So as precaution, before doing Snapshot, have to check the free disk space
before commencing Snapshot?

Ok, for 3 of the VMs, under Summary tab, we click on "Cancel" & OK
& quickly go to Snapshot Manager to do "Delete All" the snapshots
& was able to boot up & recovered the VMs.

However, the fourth VM which has multiple snapshots, it took almost
2 hrs to do "Delete All" its snapshots (stuck at 99% for very long) &
after that's done, the VM can't boot up with 'insufficient space' as
ack'ed at the ESXi SSH command prompt:
mfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1

/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #

Now trying to do Datastore vMotion out one of the VM to free
up space in this datastore
0
Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

 
LVL 120

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39207122
*ALWAYS* have at least 20% of the datastore FREE!

Be careful using Snapshots, intentionally!

Yes, that's normal for Snapshot dseletion to take a long time, hence my Article BE PATIENT!

If you have run out of storage space, when Deleting a Snapshot, it's likely it could now be corrupted or needs additional handling!

Increase storage space, and then we can proceed!
0
 

Author Comment

by:sunhux
ID: 39207152
Ok, the datastore vMotion frees up about 3-4% of space &
I could now boot up the last VM ;   phew

Just 1 last question & I'll close this thread:
Why is it after deleting away all my UAT Snapshots, I'm now
left with less than 3% free space (& this is after I've done
datastore vMotion to move out one UAT VM to a Prod datastore) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1059808346112  39434846208  96% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1068498944000  30744248320  97% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #
0
 
LVL 120

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39207244
Wait, and Be Patient, it can take a while for the datastore space to be refreshed.
0

Featured Post

2017 Webroot Threat Report

MSPs: Get the facts you need to protect your clients.
The 2017 Webroot Threat Report provides a uniquely insightful global view into the analysis and discoveries made by the Webroot® Threat Intelligence Platform to provide insights on key trends and risks as seen by our users.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If your vDisk VHD file gets deleted from the image store accidentally or on purpose, you won't be able to remove the vDisk from the PVS console. There is a known workaround that is solid.
This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
Teach the user how to use create log bundles for vCenter Server or ESXi hosts Open vSphere Web Client: Generate vCenter Server and ESXi host log bundle:  Open vCenter Server Appliance Web Management interface and generate log bundle: Open vCenter Se…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question