Solved

Can't ping/console into VMs : disk space issue

Posted on 2013-05-29
6
682 Views
Last Modified: 2013-05-30
I right-click a few UAT VMs last night, click Snapshot & then click "Take Snapshot":
most of the snapshot backup fails due to insufficient storage space.
At this point in time, the VMs are still pingable & can be RDP'ed into.

Then between app01 & app02 VMs I shared out the E: drives to each
other to take a Bare metal backup to save each other's Windows 2008
R2 bare metal backup to each other.

Then both the app01 & app02 VMs hanged & can't be pinged into anymore.
This morning, from vCenter, noticed I can't console into them as well,
giving an error message.

Can't Reboot, Reset, Shutdown nor Power Off the VMs from vCenter.
I tried to delete the Snapshot files (right click VMs, select Snapshot
Manager, select the snapshot file created last night & select Delete:
 it did not give any error but the Snapshot files are still there).

I ssh login to the ESXi host.  Below is a list of files for app01 VM
(yes, I short-formed the ESXi hostname to app01/02) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l |more
-rw-------    1 root     root         2919342080 Sep 28  2011 HHXXXVS2APP01-000001-delta.v
mdk
-rw-------    1 root     root                353 May 30 04:14 HHXXXVS2APP01-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01-000002-delta.v
mdk
-rw-------    1 root     root                330 Aug 13  2011 HHXXXVS2APP01-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01-000003-delta.v
mdk
-rw-------    1 root     root                330 Aug 16  2011 HHXXXVS2APP01-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01-000004-delta.v
mdk
-rw-------    1 root     root                330 Sep  3  2011 HHXXXVS2APP01-000004.vmdk
-rw-------    1 root     root        35383255040 May 29 10:10 HHXXXVS2APP01-000005-delta.v
mdk
-rw-------    1 root     root                360 May 30 04:14 HHXXXVS2APP01-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01-000006-delta.v
mdk
-rw-------    1 root     root                337 Sep 28  2011 HHXXXVS2APP01-000006.vmdk
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.v
mdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-r--r--    1 root     root                 65 Apr 26 11:28 HHXXXVS2APP01-6c60162a.hlog
-rw-------    1 root     root         8589934592 Aug 31  2012 HHXXXVS2APP01-6c60162a.vswp
-rw-------    1 root     root         8599829899 May 29 10:12 HHXXXVS2APP01-Snapshot13.vmsn
-rw-r--r--    1 root     root                 13 Aug 16  2011 HHXXXVS2APP01-aux.xml
-rw-------    1 root     root        53687091200 Sep  3  2011 HHXXXVS2APP01-flat.vmdk
-rw-------    1 root     root               8684 May 29 10:10 HHXXXVS2APP01.nvram
-rw-------    1 root     root                550 May 30 04:14 HHXXXVS2APP01.vmdk
-rw-r--r--    1 root     root                673 May 29 10:10 HHXXXVS2APP01.vmsd
-rwxr-xr-x    1 root     root               3905 May 29 10:10 HHXXXVS2APP01.vmx
-rw-r--r--    1 root     root                268 Apr 26 11:28 HHXXXVS2APP01.vmxf
-rw-------    1 root     root           67215360 Sep 24  2011 HHXXXVS2APP01_1-000001-delta.vmdk
-rw-------    1 root     root                357 May 30 04:14 HHXXXVS2APP01_1-000001.vmdk
-rw-------    1 root     root           16883712 Aug 13  2011 HHXXXVS2APP01_1-000002-delta.vmdk
-rw-------    1 root     root                334 Aug 13  2011 HHXXXVS2APP01_1-000002.vmdk
-rw-------    1 root     root           16883712 Aug 16  2011 HHXXXVS2APP01_1-000003-delta.vmdk
-rw-------    1 root     root                334 Aug 16  2011 HHXXXVS2APP01_1-000003.vmdk
-rw-------    1 root     root           16883712 Sep  3  2011 HHXXXVS2APP01_1-000004-delta.vmdk
-rw-------    1 root     root                334 Sep  3  2011 HHXXXVS2APP01_1-000004.vmdk
-rw-------    1 root     root        54073073664 May 29 10:10 HHXXXVS2APP01_1-000005-delta.vmdk
-rw-------    1 root     root                364 May 30 04:14 HHXXXVS2APP01_1-000005.vmdk
-rw-------    1 root     root           16883712 Sep 28  2011 HHXXXVS2APP01_1-000006-delta.vmdk
-rw-------    1 root     root                341 Sep 28  2011 HHXXXVS2APP01_1-000006.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root        53687091200 Aug 26  2011 HHXXXVS2APP01_1-flat.vmdk
-rw-------    1 root     root                552 May 30 04:14 HHXXXVS2APP01_1.vmdk
-rw-------    1 root     root           16801792 Sep 24  2011 HHXXXVS2APP01_2-000001-delta.vmdk
-rw-------    1 root     root                356 May 30 04:14 HHXXXVS2APP01_2-000001.vmdk
-rw-------    1 root     root           16801792 Aug 13  2011 HHXXXVS2APP01_2-000002-delta.vmdk
-rw-------    1 root     root                333 Aug 13  2011 HHXXXVS2APP01_2-000002.vmdk
-rw-------    1 root     root           16801792 Aug 16  2011 HHXXXVS2APP01_2-000003-delta.vmdk
-rw-------    1 root     root                333 Aug 16  2011 HHXXXVS2APP01_2-000003.vmdk
-rw-------    1 root     root           16801792 Sep  3  2011 HHXXXVS2APP01_2-000004-delta.vmdk
-rw-------    1 root     root                333 Sep  3  2011 HHXXXVS2APP01_2-000004.vmdk
-rw-------    1 root     root           16801792 May 29 10:10 HHXXXVS2APP01_2-000005-delta.vmdk
-rw-------    1 root     root                363 May 30 04:14 HHXXXVS2APP01_2-000005.vmdk
-rw-------    1 root     root           16801792 Sep 28  2011 HHXXXVS2APP01_2-000006-delta.vmdk
-rw-------    1 root     root                340 Sep 28  2011 HHXXXVS2APP01_2-000006.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk
-rw-------    1 root     root        10737418240 Aug 26  2011 HHXXXVS2APP01_2-flat.vmdk
-rw-------    1 root     root                551 May 30 04:14 HHXXXVS2APP01_2.vmdk
-rw-r--r--    1 root     root             436721 Aug 22  2012 vmware-10.log
-rw-r--r--    1 root     root             577066 Aug 31  2012 vmware-11.log
-rw-r--r--    1 root     root             458842 Dec 18 02:45 vmware-12.log
-rw-r--r--    1 root     root             153691 Jan  3 06:24 vmware-13.log
-rw-r--r--    1 root     root             193737 Apr 10 08:51 vmware-14.log
-rw-r--r--    1 root     root             161305 Apr 26 11:28 vmware-15.log
-rw-r--r--    1 root     root             169297 May 30 03:14 vmware.log
-rw-------    1 root     root           52428800 Apr 26 11:25 vmx-HHXXXVS2APP01-1818236458-1.vswp


How should I go about recovering the VMs?  Tried Storage vMotion as
well but no joy
0
Comment
Question by:sunhux
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 

Author Comment

by:sunhux
ID: 39206802
Is it safe to expand the files' sizes & which files (all from the list
below? ) should I expand:

vmware-vdiskmanager -x 10MB myDisk.vmdk


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca/HHXXXVS2APP01 # ls -l *0007*
-rw-------    1 root     root           83992576 May 30 03:14 HHXXXVS2APP01-000007-delta.vmdk
-rw-------    1 root     root                337 May 29 10:10 HHXXXVS2APP01-000007.vmdk
-rw-------    1 root     root          234987520 May 29 13:24 HHXXXVS2APP01_1-000007-delta.vmdk
-rw-------    1 root     root                341 May 29 10:11 HHXXXVS2APP01_1-000007.vmdk
-rw-------    1 root     root              24576 May 29 10:10 HHXXXVS2APP01_2-000007-delta.vmdk
-rw-------    1 root     root                340 May 29 10:10 HHXXXVS2APP01_2-000007.vmdk

Is it safe to try the suggestions given in link below:
  http://www.howtoforge.com/how-to-shrink-vmware-virtual-disk-files-vmdk
0
 
LVL 121

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39206823
You cannot expand disks as you have stated, that is only for VMware Workstation.

Also, you cannot expand disks which have a snapshot attached.

Please see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I would check if the current VM is still in Snapshot Mode, e.g. writing to a Snapshot disk, see my EE Article, and check.

and then we can work with you to get out of Snapshot Hell!
0
 

Author Comment

by:sunhux
ID: 39207008
So as precaution, before doing Snapshot, have to check the free disk space
before commencing Snapshot?

Ok, for 3 of the VMs, under Summary tab, we click on "Cancel" & OK
& quickly go to Snapshot Manager to do "Delete All" the snapshots
& was able to boot up & recovered the VMs.

However, the fourth VM which has multiple snapshots, it took almost
2 hrs to do "Delete All" its snapshots (stuck at 99% for very long) &
after that's done, the VM can't boot up with 'insufficient space' as
ack'ed at the ESXi SSH command prompt:
mfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1

/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #

Now trying to do Datastore vMotion out one of the VM to free
up space in this datastore
0
Free NetCrunch network monitor licenses!

Only on Experts-Exchange: Sign-up for a free-trial and we'll send you your permanent license!

Here is what you get: 30 Nodes | Unlimited Sensors | No Time Restrictions | Absolutely FREE!

Act now. This offer ends July 14, 2017.

 
LVL 121

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39207122
*ALWAYS* have at least 20% of the datastore FREE!

Be careful using Snapshots, intentionally!

Yes, that's normal for Snapshot dseletion to take a long time, hence my Article BE PATIENT!

If you have run out of storage space, when Deleting a Snapshot, it's likely it could now be corrupted or needs additional handling!

Increase storage space, and then we can proceed!
0
 

Author Comment

by:sunhux
ID: 39207152
Ok, the datastore vMotion frees up about 3-4% of space &
I could now boot up the last VM ;   phew

Just 1 last question & I'll close this thread:
Why is it after deleting away all my UAT Snapshots, I'm now
left with less than 3% free space (& this is after I've done
datastore vMotion to move out one UAT VM to a Prod datastore) :


/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1099243192320            0 100% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1059808346112  39434846208  96% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca # df | grep UAT
VMFS-3     1099243192320 1068498944000  30744248320  97% /vmfs/volumes/UAT_DATASTORE_1
/vmfs/volumes/4e1a713f-86524876-0f29-5cf3fc4a38ca #
0
 
LVL 121

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 490 total points
ID: 39207244
Wait, and Be Patient, it can take a while for the datastore space to be refreshed.
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If your vDisk VHD file gets deleted from the image store accidentally or on purpose, you won't be able to remove the vDisk from the PVS console. There is a known workaround that is solid.
This article outlines why you need to choose a backup solution that protects your entire environment – including your VMware ESXi and Microsoft Hyper-V virtualization hosts – not just your virtual machines.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:

717 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question