Solved

Vmware consolidation failed, vm will not startup, no more disk space

Posted on 2013-11-28
17
5,765 Views
Last Modified: 2013-12-03
Hello,

I have  a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..

Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.

The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.

I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)

For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)

In the data store I have multiple files:

Server2012.vmdk                     (336.703.500,00)
server2012-00001.vmdk       (345.753.156,00)
server2012-00002.vmdk       (272.156.700,00)
server2012-00003.vmdk         (20.924.420,00)
Vmmcores.gz                          (5.184.00)
a bunch of log files
server2012.nvram
server 2012.vmx   vmsd file, hlog file.
0
Comment
Question by:Benderama
  • 8
  • 6
  • 2
  • +1
17 Comments
 

Author Comment

by:Benderama
ID: 39683010
Last night I tried the following: I moved the server2012-00001.vmdk to another computer,

since my virtual machine seems to be using the 00003 version, after safely backing up the file I deleted server2012-00001.vmdk from the datastore creating 336GB of space.

Then I consolidated the server and it was running again! BUT when i looked closer it was a old version of the virtual machine, it did not have the data I need to recover anymore.

Now my question is, what version is the correct one? can I safely delete (after backing it up) another vmdk file ? why are there multiple vmdk files?
0
 
LVL 117
ID: 39683172
Moving a virtual machine with snapshots can be dangerous.

I would recommend, you use VMware vCenter Converter Standalone 5.5

 HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5

Use Converter to create a V2V to a local workstation, or add more storage.
0
 

Author Comment

by:Benderama
ID: 39683190
Hello,

thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.

I got the error "The specified virtual disk needs repair." so I tried

vmware-vdiskmanager -R <path of the vmdk(virtual disk)>

It didn't give any errors but the machine still would not start.


Unfortunately, adding more storage is not an option.
0
 
LVL 16

Expert Comment

by:Nyaema
ID: 39683192
You have one large disk of 800GB which has been split into 4 disks...  The option to split disks into mylitiple disks was selected and not all space was allocated at time of creation.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now.  I believe you need all four VMDKs.

Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.

On the same tab, you can then try to copact and defragment the drive.  Compacting may gain you more space, that if the disk has free space internally that can be compacted.

YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.
0
 

Author Comment

by:Benderama
ID: 39683212
Where can I compact and defragment the drivers in the Vsphere client?

I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
0
 
LVL 117
ID: 39683216
Compacting and De-Fragmentation is not supported.

Options:-

1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.

you could try...

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE
0
 

Author Comment

by:Benderama
ID: 39683225
When I access the Esxi trought the workstation installed on a client I do get the option to defragment and compact but its done in a few seconds and doesn't change anything.

I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.

I can access the Esxi throught SSH, should I try this: http://www.whitneytechnologies.com/?p=270  on one of the backupped files?
0
 
LVL 117
ID: 39683235
Can you screenshot your datastore, I believe the issue is the snapshot, you need to MERGE the snapshot. (also there could be corruption, because you have ran out of disk space).

normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.

Deleting the original VM, and then restoring or converting back.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 

Author Comment

by:Benderama
ID: 39683276
This is the datastore, I just greyed out the company name:

datastore
Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.

By doing this I was able to start the virtual machine but with an old version.

I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is:  (354.051.231.744 bytes)
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 39683296
By doing this you have now, altered the chain, which you will also need to repair before merging the contents.

by removing the snapshot, will result to all the changes being lost.

I would follow the following procedure.  

1. Copy the SERVER 2012 Standa-000001-delta.vmdk back to the server.
2. Free-up storage space, this ,may involve shutting down the terminal server, and moving off the server.
3. Merge the Snapshot, either using CLONE (vmkfstools) or VMware Converter.

Unless you just want to lose all the changes, and also delete the second and third snapshot.

The parent disk, and snapshots are all required to for a complete VM.

have a read of my article

HOW TO: VMware Snapshots :- Be Patient
0
 
LVL 16

Assisted Solution

by:Nyaema
Nyaema earned 250 total points
ID: 39683380
Doubt you will be able to recover from this cause I notice the  0001.vmdk you moved out has been recreated.  Even if you, expect some corruption.

Probably better off rebuiling this VM and create two separate VMDKs, one disk for OS (64GB more than enough) and the other for Data (600GB) for your backps.  This way backups don't fill up your C drive and cause the VM to crush cause of lack of space.

Next time you VSphere Client to delete snapshot:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option

Also you need to be scheduling clean ups from withing the VM once started up to create/freeup  more disk space.
0
 
LVL 117
ID: 39683394
You will not be able to delete the Snapshot from Snapshot Manager, now the chain has been changed.
0
 

Author Comment

by:Benderama
ID: 39683397
It seems I can't even upload the 0001 to the datastore again. It just stays on 0% and I can see a file being created on the datastore.

I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?
0
 

Author Comment

by:Benderama
ID: 39683398
I deleted the snapshots (in the beginning) by using this option:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
0
 
LVL 117
ID: 39683408
Yes, it's likely it's failed to delete the snapshot, because you did not have the disk space to merge.

You cannot connect it directly to server via USB, it's not supported.

It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.

Do you have another ESXi server?

Can you add more storage space?

even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39691333
Wow this is a messy situation...

Here is a long shot.

1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only
0
 

Author Closing Comment

by:Benderama
ID: 39691901
I ended up reinstalling the virtual Machine,


I managed to recover my data from a physical machine using getdataback for NTFS
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

#Citrix #Citrix Netscaler #HTTP Compression #Load Balance
It Is not possible to enable LLDP in vSwitch(at least is not supported by VMware), so in this article we will enable this, and also go trough how to enabled CDP and how to get this information in vSwitches and also in vDS.
In this Micro Tutorial viewers will learn how to use Boot Corrector from Paragon Rescue Kit Free to identify and fix the boot problems of Windows 7/8/2012R2 etc. As an example is used Windows 2012R2 which lost its active partition flag (often happen…
This tutorial will walk an individual through the steps necessary to configure their installation of BackupExec 2012 to use network shared disk space. Verify that the path to the shared storage is valid and that data can be written to that location:…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now