Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Vmware consolidation failed, vm will not startup, no more disk space

Posted on 2013-11-28
Medium Priority
Last Modified: 2013-12-03

I have  a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..

Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.

The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.

I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)

For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)

In the data store I have multiple files:

Server2012.vmdk                     (336.703.500,00)
server2012-00001.vmdk       (345.753.156,00)
server2012-00002.vmdk       (272.156.700,00)
server2012-00003.vmdk         (20.924.420,00)
Vmmcores.gz                          (5.184.00)
a bunch of log files
server 2012.vmx   vmsd file, hlog file.
Question by:Benderama
  • 8
  • 6
  • 2
  • +1

Author Comment

ID: 39683010
Last night I tried the following: I moved the server2012-00001.vmdk to another computer,

since my virtual machine seems to be using the 00003 version, after safely backing up the file I deleted server2012-00001.vmdk from the datastore creating 336GB of space.

Then I consolidated the server and it was running again! BUT when i looked closer it was a old version of the virtual machine, it did not have the data I need to recover anymore.

Now my question is, what version is the correct one? can I safely delete (after backing it up) another vmdk file ? why are there multiple vmdk files?
LVL 125
ID: 39683172
Moving a virtual machine with snapshots can be dangerous.

I would recommend, you use VMware vCenter Converter Standalone 5.5

 HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5

Use Converter to create a V2V to a local workstation, or add more storage.

Author Comment

ID: 39683190

thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.

I got the error "The specified virtual disk needs repair." so I tried

vmware-vdiskmanager -R <path of the vmdk(virtual disk)>

It didn't give any errors but the machine still would not start.

Unfortunately, adding more storage is not an option.

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

LVL 16

Expert Comment

by:Joseph Nyaema
ID: 39683192
You have one large disk of 800GB which has been split into 4 disks...  The option to split disks into mylitiple disks was selected and not all space was allocated at time of creation.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now.  I believe you need all four VMDKs.

Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.

On the same tab, you can then try to copact and defragment the drive.  Compacting may gain you more space, that if the disk has free space internally that can be compacted.

YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.

Author Comment

ID: 39683212
Where can I compact and defragment the drivers in the Vsphere client?

I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
LVL 125
ID: 39683216
Compacting and De-Fragmentation is not supported.


1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.

you could try...

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE

Author Comment

ID: 39683225
When I access the Esxi trought the workstation installed on a client I do get the option to defragment and compact but its done in a few seconds and doesn't change anything.

I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.

I can access the Esxi throught SSH, should I try this: http://www.whitneytechnologies.com/?p=270  on one of the backupped files?
LVL 125
ID: 39683235
Can you screenshot your datastore, I believe the issue is the snapshot, you need to MERGE the snapshot. (also there could be corruption, because you have ran out of disk space).

normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.

Deleting the original VM, and then restoring or converting back.

Author Comment

ID: 39683276
This is the datastore, I just greyed out the company name:

Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.

By doing this I was able to start the virtual machine but with an old version.

I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is:  (354.051.231.744 bytes)
LVL 125

Accepted Solution

Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 39683296
By doing this you have now, altered the chain, which you will also need to repair before merging the contents.

by removing the snapshot, will result to all the changes being lost.

I would follow the following procedure.  

1. Copy the SERVER 2012 Standa-000001-delta.vmdk back to the server.
2. Free-up storage space, this ,may involve shutting down the terminal server, and moving off the server.
3. Merge the Snapshot, either using CLONE (vmkfstools) or VMware Converter.

Unless you just want to lose all the changes, and also delete the second and third snapshot.

The parent disk, and snapshots are all required to for a complete VM.

have a read of my article

HOW TO: VMware Snapshots :- Be Patient
LVL 16

Assisted Solution

by:Joseph Nyaema
Joseph Nyaema earned 1000 total points
ID: 39683380
Doubt you will be able to recover from this cause I notice the  0001.vmdk you moved out has been recreated.  Even if you, expect some corruption.

Probably better off rebuiling this VM and create two separate VMDKs, one disk for OS (64GB more than enough) and the other for Data (600GB) for your backps.  This way backups don't fill up your C drive and cause the VM to crush cause of lack of space.

Next time you VSphere Client to delete snapshot:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option

Also you need to be scheduling clean ups from withing the VM once started up to create/freeup  more disk space.
LVL 125
ID: 39683394
You will not be able to delete the Snapshot from Snapshot Manager, now the chain has been changed.

Author Comment

ID: 39683397
It seems I can't even upload the 0001 to the datastore again. It just stays on 0% and I can see a file being created on the datastore.

I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?

Author Comment

ID: 39683398
I deleted the snapshots (in the beginning) by using this option:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
LVL 125
ID: 39683408
Yes, it's likely it's failed to delete the snapshot, because you did not have the disk space to merge.

You cannot connect it directly to server via USB, it's not supported.

It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.

Do you have another ESXi server?

Can you add more storage space?

even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
LVL 20

Expert Comment

ID: 39691333
Wow this is a messy situation...

Here is a long shot.

1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only

Author Closing Comment

ID: 39691901
I ended up reinstalling the virtual Machine,

I managed to recover my data from a physical machine using getdataback for NTFS

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Microsoft Jet database engine errors can crop up out of nowhere to disrupt the working of the Exchange server. Decoding why a particular error occurs goes a long way in determining the right solution for it.
Herein one will find an aggregate of some of my experience building and deploying virtualization stacks both in standalone, clustered Hyper-V, clustered Hyper-V with a Scale-Out File Server (SOFS) backend, and Storage Spaces Direct (S2D).
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…

578 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question