• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 7407
  • Last Modified:

Vmware consolidation failed, vm will not startup, no more disk space

Hello,

I have  a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..

Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.

The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.

I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)

For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)

In the data store I have multiple files:

Server2012.vmdk                     (336.703.500,00)
server2012-00001.vmdk       (345.753.156,00)
server2012-00002.vmdk       (272.156.700,00)
server2012-00003.vmdk         (20.924.420,00)
Vmmcores.gz                          (5.184.00)
a bunch of log files
server2012.nvram
server 2012.vmx   vmsd file, hlog file.
0
Benderama
Asked:
Benderama
  • 8
  • 6
  • 2
  • +1
2 Solutions
 
BenderamaAuthor Commented:
Last night I tried the following: I moved the server2012-00001.vmdk to another computer,

since my virtual machine seems to be using the 00003 version, after safely backing up the file I deleted server2012-00001.vmdk from the datastore creating 336GB of space.

Then I consolidated the server and it was running again! BUT when i looked closer it was a old version of the virtual machine, it did not have the data I need to recover anymore.

Now my question is, what version is the correct one? can I safely delete (after backing it up) another vmdk file ? why are there multiple vmdk files?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Moving a virtual machine with snapshots can be dangerous.

I would recommend, you use VMware vCenter Converter Standalone 5.5

 HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5

Use Converter to create a V2V to a local workstation, or add more storage.
0
 
BenderamaAuthor Commented:
Hello,

thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.

I got the error "The specified virtual disk needs repair." so I tried

vmware-vdiskmanager -R <path of the vmdk(virtual disk)>

It didn't give any errors but the machine still would not start.


Unfortunately, adding more storage is not an option.
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Joseph NyaemaIT ConsultantCommented:
You have one large disk of 800GB which has been split into 4 disks...  The option to split disks into mylitiple disks was selected and not all space was allocated at time of creation.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now.  I believe you need all four VMDKs.

Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.

On the same tab, you can then try to copact and defragment the drive.  Compacting may gain you more space, that if the disk has free space internally that can be compacted.

YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.
0
 
BenderamaAuthor Commented:
Where can I compact and defragment the drivers in the Vsphere client?

I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Compacting and De-Fragmentation is not supported.

Options:-

1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.

you could try...

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE
0
 
BenderamaAuthor Commented:
When I access the Esxi trought the workstation installed on a client I do get the option to defragment and compact but its done in a few seconds and doesn't change anything.

I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.

I can access the Esxi throught SSH, should I try this: http://www.whitneytechnologies.com/?p=270  on one of the backupped files?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Can you screenshot your datastore, I believe the issue is the snapshot, you need to MERGE the snapshot. (also there could be corruption, because you have ran out of disk space).

normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.

Deleting the original VM, and then restoring or converting back.
0
 
BenderamaAuthor Commented:
This is the datastore, I just greyed out the company name:

datastore
Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.

By doing this I was able to start the virtual machine but with an old version.

I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is:  (354.051.231.744 bytes)
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
By doing this you have now, altered the chain, which you will also need to repair before merging the contents.

by removing the snapshot, will result to all the changes being lost.

I would follow the following procedure.  

1. Copy the SERVER 2012 Standa-000001-delta.vmdk back to the server.
2. Free-up storage space, this ,may involve shutting down the terminal server, and moving off the server.
3. Merge the Snapshot, either using CLONE (vmkfstools) or VMware Converter.

Unless you just want to lose all the changes, and also delete the second and third snapshot.

The parent disk, and snapshots are all required to for a complete VM.

have a read of my article

HOW TO: VMware Snapshots :- Be Patient
0
 
Joseph NyaemaIT ConsultantCommented:
Doubt you will be able to recover from this cause I notice the  0001.vmdk you moved out has been recreated.  Even if you, expect some corruption.

Probably better off rebuiling this VM and create two separate VMDKs, one disk for OS (64GB more than enough) and the other for Data (600GB) for your backps.  This way backups don't fill up your C drive and cause the VM to crush cause of lack of space.

Next time you VSphere Client to delete snapshot:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option

Also you need to be scheduling clean ups from withing the VM once started up to create/freeup  more disk space.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You will not be able to delete the Snapshot from Snapshot Manager, now the chain has been changed.
0
 
BenderamaAuthor Commented:
It seems I can't even upload the 0001 to the datastore again. It just stays on 0% and I can see a file being created on the datastore.

I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?
0
 
BenderamaAuthor Commented:
I deleted the snapshots (in the beginning) by using this option:

Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, it's likely it's failed to delete the snapshot, because you did not have the disk space to merge.

You cannot connect it directly to server via USB, it's not supported.

It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.

Do you have another ESXi server?

Can you add more storage space?

even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
0
 
compdigit44Commented:
Wow this is a messy situation...

Here is a long shot.

1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only
0
 
BenderamaAuthor Commented:
I ended up reinstalling the virtual Machine,


I managed to recover my data from a physical machine using getdataback for NTFS
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

  • 8
  • 6
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now