Why am I out of disk space on my ESXi Host

Posted on 2009-04-20
Last Modified: 2012-05-06
Hi Experts
I have an IBM X3500 (standard spec server) with VMware ESXi hosting 2 VM's, Windows Server 2003 Small Business Server as the domain controller and Server 2008 Standard running as a terminal server.
The Guest OS's are freshly installed from about a week ago and we have migrated our production environment from an old HP server running SBS 03 to this ESXi host.

The SBS 03 VM on ESXi is crashing occasionally, usually under intense load with it asking a question as shown in the attached picture.

Looks to me like it's showing I'm out of space on one of the drives but I am not out of space that I can see.

We have 2x 73GB SAS drives in RAID 1 and 3x 146GB SAS drives in RAID 5.

On the 73GB mirror we have the following.
The ESXi hypervisor
a 50 GB Virtual HDD created to be C: for SBS 2003

On the 146GBx3 in RAID 5 array we have
a 150 GB Virtual HDD created to be D: for SBS 2003
a 50 GB Virtual HDD created to be C: for Server 2008

The 73GB Mirror has about 12GB free.
The RAID 5 array has about 70GB free.

I thought it might be running out of space on the mirror for the swapfile so i moved that to the RAID 5 array but it did not help.

The Virtual HDD's have very little data on them asides from their OS requirements.

How do I stop the SBS 03 VM crashing?


thanks guys
Question by:tlcservice

    Author Comment

    As stated the SBS 03 machine is configured with 2 VHDD's, 1 50 GB on the RAID 1 Mirror and 1 150 GB on the RAID 5 array.
    When I open the datastore I find the file referenced in the previous picture, 20_1-000001.vmdk.
    This is a 2.5GB file, looks to a VHDD (.vmdk extension).
    There is also another 10 GB VHDD file.
    I did not create either of these, does anyone know how any why they showed up?
    I am now about to delete the Server 2008 off the RAID 5 array and copy the SBS 03 VM on to the RAID 5 as we can easily rebuild the Server 2008 VM later and right now it seems we need some space.


    Author Comment

    Ok, successfully transferred the SBS 03 VM accross from the 73 GB Mirror to the RAID 5 array.
    The VM powers on and runs stable.
    I tried some intensive tasks to be sure.
    However the random HDD's that showed up out of nowhere, 1 of them has stayed the same at 10GB.
    The other 1 has grown from 2.5GB to 11GB.
    I get the feeling it's some sort of tempory (either SWAP or like a hibernation file).
    Either way, we can't have it chewing up disc space.
    How do I stop those VHDD's increasing in size?
    LVL 22

    Expert Comment

    Looks like the virtual disk, disk mode configuration needs to be reviewed.

    disk mode is
    A property of a virtual disk that defines its external behavior but is completely
    invisible to the guest operating system. There are four modes: persistent (changes
    to the disk are always preserved across sessions), nonpersistent (changes are never
    preserved), undoable (changes are preserved at the user¹s discretion), and append
    (similar to undoable, but the changes are preserved until a system administrator
    deletes the redolog file).

    There was mention of a redo disk, which is a temporary storage area that can be committed or discarded.
    LVL 8

    Expert Comment

    Am I missing something obvious or is your server in snapshot?
    Via the VI client right click on your SBS Server go to Snapshot->Snapshot manager.
    Do you have snapshots there..
    The screenshot attached does not have any snapshots.

    LVL 8

    Expert Comment

    You could assist us futher in diagnosing this by posting the contense of your .vmx config file.

    Author Comment

    There was 1 snapshot of the virtual machine but I deleted it awhile ago, my snapshot manager looks exactly like the one you posted markzz.
    If I copy the 11GB vmdk file into a different folder the SBS 03 VM refuses to boot stating it has a missing file. I see in the VMX file it is looking for the 11GB vmdk file, but why does it need it?
    Thanks for your replies.
    LVL 22

    Expert Comment

    How many drives are on the virtual server?
    LVL 8

    Expert Comment

    Lets see
    Hmm that vmx is a bit of a mess.
    Whay do you have 2 SCSI Controllers? Unless your doing something extrordinary it servers no benifit except it makes the guest more difficult to work with.
    Your SBS Server has an 11GB disk0 (likely C:\) and a 150GB Disk1

    Without trying to reconfigure your system via text based posts which obviously isn't possible I would suggest you perform an ofline clone.
    You are going to need about 170GB of disk to do this. If you don't have enough disk you'll need to buy, borrow or in some fassion get more disk.
    You could even setup another ESXi server and clone to it but your VMFS store must have 170GB free.
    Cone to your other ESXi server, Confirm the SBS guest can now poweron and does not have disk (vmdk files) with 00001 etc names.
    OK all confirmed.
    Take a copy of the .vmx file
    Now remove the Second SCSI controller.
    Powerup again
    All good!!
    Now shutdown the SBS guest and clone it back to the original ESXi host server.

    This is going to take some time..
    One point to consider here is virtualisation is available to us all, but unless implemented correctly you may well be creating a great deal of work for yourself rather than reducing your work load. Also what would happen if you lost the lot?
    The point again is
    Professional implementation. (including backup)
    This is not free.. It will take a substantial capital investment but can be cost justified over 3 years to show it's cost positive. What I mean is you can spend money and still show the boss you'll save him money.
    Do the figures.
    Show you expected Server growth and therefore cost.
    Cost it in a physical world.
    Cost it in a virtual world.
    LVL 8

    Expert Comment

    OH here's an example vmx file, open it with wordpad
    LVL 2

    Expert Comment

    The error indicates that you have snapshot, because snapshots use redo.log
    LVL 5

    Accepted Solution

    The issue you are having is that a snapshot was created and still exists.  ESXi has a bug in it, that when you delete a snapshot the changes are merged back into the original file. THE PROBLEMS IS that once the file gets way too big (in your case it has), then it does not merge back in correctly/at all but removes the link in the snapshot manager (as you have noticed).

    THIS FILE WILL KEEP GROWING as it is a differential change of the original.  You have only one solution (that i know of anyway), which i have used a few times before myself as this has happened to a few customers.

    VMWare Convert the current system to a new image, which will create a new virtual hard disk.  If you have ESX full, then you can use a command from the shell to do this, but in ESXi you can't, hence the only viable option.

    If you leave it too long, eventually the hard disk will fill up and you won't be able to boot any of the images...

    Author Closing Comment

    Thanks for your help guys.

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    In VMware vSphere 4 1 and 5.0, if you have two or more hosts, using shared storage, (SAN/NAS) you can create a VMware vSphere Cluster and enable High Availability mode (HA), if you have the correct VMware vSphere Licensing. VMware vSphere High Av…
    VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
    Teach the user how to install vSphere Update Manager  Console to Windows system:  Install vSphere Update Manager: Configure vSphere Update Manager plug-in in vSphere Client: Verify vSphere Update Manager settings in vSphere Client:
    Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now