Vmware consolidation failed, vm will not startup, no more disk space

Posted on 2013-11-28
Last Modified: 2013-12-03

I have  a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..

Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.

The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.

I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)

For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)

In the data store I have multiple files:

Server2012.vmdk                     (336.703.500,00)
server2012-00001.vmdk       (345.753.156,00)
server2012-00002.vmdk       (272.156.700,00)
server2012-00003.vmdk         (20.924.420,00)
Vmmcores.gz                          (5.184.00)
a bunch of log files
server 2012.vmx   vmsd file, hlog file.
Question by:Benderama
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 6
  • 2
  • +1

Author Comment

ID: 39683010
Last night I tried the following: I moved the server2012-00001.vmdk to another computer,

since my virtual machine seems to be using the 00003 version, after safely backing up the file I deleted server2012-00001.vmdk from the datastore creating 336GB of space.

Then I consolidated the server and it was running again! BUT when i looked closer it was a old version of the virtual machine, it did not have the data I need to recover anymore.

Now my question is, what version is the correct one? can I safely delete (after backing it up) another vmdk file ? why are there multiple vmdk files?
LVL 120
ID: 39683172
Moving a virtual machine with snapshots can be dangerous.

I would recommend, you use VMware vCenter Converter Standalone 5.5

 HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5

Use Converter to create a V2V to a local workstation, or add more storage.

Author Comment

ID: 39683190

thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.

I got the error "The specified virtual disk needs repair." so I tried

vmware-vdiskmanager -R <path of the vmdk(virtual disk)>

It didn't give any errors but the machine still would not start.

Unfortunately, adding more storage is not an option.
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

LVL 16

Expert Comment

ID: 39683192
You have one large disk of 800GB which has been split into 4 disks...  The option to split disks into mylitiple disks was selected and not all space was allocated at time of creation.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now.  I believe you need all four VMDKs.

Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.

On the same tab, you can then try to copact and defragment the drive.  Compacting may gain you more space, that if the disk has free space internally that can be compacted.

YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.

Author Comment

ID: 39683212
Where can I compact and defragment the drivers in the Vsphere client?

I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
LVL 120
ID: 39683216
Compacting and De-Fragmentation is not supported.


1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.

you could try...

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE

Author Comment

ID: 39683225
When I access the Esxi trought the workstation installed on a client I do get the option to defragment and compact but its done in a few seconds and doesn't change anything.

I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.

I can access the Esxi throught SSH, should I try this:  on one of the backupped files?
LVL 120
ID: 39683235
Can you screenshot your datastore, I believe the issue is the snapshot, you need to MERGE the snapshot. (also there could be corruption, because you have ran out of disk space).

normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.

Deleting the original VM, and then restoring or converting back.

Author Comment

ID: 39683276
This is the datastore, I just greyed out the company name:

Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.

By doing this I was able to start the virtual machine but with an old version.

I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is:  (354.051.231.744 bytes)
LVL 120

Accepted Solution

Andrew Hancock (VMware vExpert / EE MVE^2) earned 250 total points
ID: 39683296
By doing this you have now, altered the chain, which you will also need to repair before merging the contents.

by removing the snapshot, will result to all the changes being lost.

I would follow the following procedure.  

1. Copy the SERVER 2012 Standa-000001-delta.vmdk back to the server.
2. Free-up storage space, this ,may involve shutting down the terminal server, and moving off the server.
3. Merge the Snapshot, either using CLONE (vmkfstools) or VMware Converter.

Unless you just want to lose all the changes, and also delete the second and third snapshot.

The parent disk, and snapshots are all required to for a complete VM.

have a read of my article

HOW TO: VMware Snapshots :- Be Patient
LVL 16

Assisted Solution

Nyaema earned 250 total points
ID: 39683380
Doubt you will be able to recover from this cause I notice the  0001.vmdk you moved out has been recreated.  Even if you, expect some corruption.

Probably better off rebuiling this VM and create two separate VMDKs, one disk for OS (64GB more than enough) and the other for Data (600GB) for your backps.  This way backups don't fill up your C drive and cause the VM to crush cause of lack of space.

Next time you VSphere Client to delete snapshot:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option

Also you need to be scheduling clean ups from withing the VM once started up to create/freeup  more disk space.
LVL 120
ID: 39683394
You will not be able to delete the Snapshot from Snapshot Manager, now the chain has been changed.

Author Comment

ID: 39683397
It seems I can't even upload the 0001 to the datastore again. It just stays on 0% and I can see a file being created on the datastore.

I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?

Author Comment

ID: 39683398
I deleted the snapshots (in the beginning) by using this option:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
LVL 120
ID: 39683408
Yes, it's likely it's failed to delete the snapshot, because you did not have the disk space to merge.

You cannot connect it directly to server via USB, it's not supported.

It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.

Do you have another ESXi server?

Can you add more storage space?

even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
LVL 20

Expert Comment

ID: 39691333
Wow this is a messy situation...

Here is a long shot.

1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only

Author Closing Comment

ID: 39691901
I ended up reinstalling the virtual Machine,

I managed to recover my data from a physical machine using getdataback for NTFS

Featured Post

Free eBook: Backup on AWS

Everything you need to know about backup and disaster recovery with AWS, for FREE!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question