Vmware consolidation failed, vm will not startup, no more disk space
Hello,
I have a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..
Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.
The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.
I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)
For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)
In the data store I have multiple files:
Server2012.vmdk (336.703.500,00)
server2012-00001.vmdk (345.753.156,00)
server2012-00002.vmdk (272.156.700,00)
server2012-00003.vmdk (20.924.420,00)
Vmmcores.gz (5.184.00)
a bunch of log files
server2012.nvram
server 2012.vmx vmsd file, hlog file.
I have a huge problem, we are running Vmware Esxi 5.0 on a server, there are 2 virtual machines running on this Esxi, and SBS 2011 and a server 2012 Standard for running Veeam.
Both have their own datastore..
Only the SBS server is being used by the company, the 2012 Standard was setup for running veeam to back-up the SBS Server (multiple revisions). A few days ago, I noticed that the Server 2012 was on hold due to lack of disk space. It was using 800GB while the OS is only using 200GB.
The reason seemed to be a failed Veeam backup. Since there was no space left on this datastore (or the other one) I foolishly deleted all the snapshots in the snapshot manager, in the hopes of creating disk space. I did this once before, after consolidating the snapshots I had enough diskspace to clone the Virtual machine and shrink it and put it back. This time however I do not have enough disk space to consolidate the disks. And now my server 2012 won’t start anymore.
I tried to move the failed virtual machine to a Vmware Workstation in the hopes of getting it running again but with no luck. I am certain that if I have more disk space and the consolidation didn’t fail, the Virtual machine would be restored and I could start it again (as I did before)
For the Server 2012 I only had 1 backup revision, and it failed due to the last failed backup (I know this was stupid but I was also lacking disk space for the back-up and the client can’t invest in more storage atm)
In the data store I have multiple files:
Server2012.vmdk (336.703.500,00)
server2012-00001.vmdk (345.753.156,00)
server2012-00002.vmdk (272.156.700,00)
server2012-00003.vmdk (20.924.420,00)
Vmmcores.gz (5.184.00)
a bunch of log files
server2012.nvram
server 2012.vmx vmsd file, hlog file.
Moving a virtual machine with snapshots can be dangerous.
I would recommend, you use VMware vCenter Converter Standalone 5.5
HOW TO: P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5
Use Converter to create a V2V to a local workstation, or add more storage.
I would recommend, you use VMware vCenter Converter Standalone 5.5
HOW TO: P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5
Use Converter to create a V2V to a local workstation, or add more storage.
ASKER
Hello,
thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.
I got the error "The specified virtual disk needs repair." so I tried
vmware-vdiskmanager -R <path of the vmdk(virtual disk)>
It didn't give any errors but the machine still would not start.
Unfortunately, adding more storage is not an option.
thanks for the reply, I tried that, while doing this I shrunk the 800GB virtual machine to 200GB, but on the workstation it would not start.
I got the error "The specified virtual disk needs repair." so I tried
vmware-vdiskmanager -R <path of the vmdk(virtual disk)>
It didn't give any errors but the machine still would not start.
Unfortunately, adding more storage is not an option.
You have one large disk of 800GB which has been split into 4 disks... The option to split disks into mylitiple disks was selected and not all space was allocated at time of creation.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now. I believe you need all four VMDKs.
Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.
On the same tab, you can then try to copact and defragment the drive. Compacting may gain you more space, that if the disk has free space internally that can be compacted.
YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.
I suspect that the disk has been growing overtime and claiming space, hence your predicament right now. I believe you need all four VMDKs.
Look at the proerties of the VM, select the disk and loot at its information properties to ascertain the above.
On the same tab, you can then try to copact and defragment the drive. Compacting may gain you more space, that if the disk has free space internally that can be compacted.
YOu can try moving the VM to a machine with enought space like 1TB free, trystart the VM, cleanup and compact the VMDK.
ASKER
Where can I compact and defragment the drivers in the Vsphere client?
I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
I can only use the client to connect to the Esxi, The Vcenter server was installed on the failed virtual machine.
Compacting and De-Fragmentation is not supported.
Options:-
1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.
you could try...
Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE
Options:-
1. Increase datastore.
2. Move VM off, via Conversion, sort our the snapshot issue, delete original VMs.
3. Copy the VM back.
you could try...
Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE
ASKER
When I access the Esxi trought the workstation installed on a client I do get the option to defragment and compact but its done in a few seconds and doesn't change anything.
I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.
I can access the Esxi throught SSH, should I try this: http://www.whitneytechnologies.com/?p=270 on one of the backupped files?
I still have about 17GB left on the SBS datastore but am afraid of changing anything there because the SBS is working fine.
I can access the Esxi throught SSH, should I try this: http://www.whitneytechnologies.com/?p=270 on one of the backupped files?
Can you screenshot your datastore, I believe the issue is the snapshot, you need to MERGE the snapshot. (also there could be corruption, because you have ran out of disk space).
normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.
Deleting the original VM, and then restoring or converting back.
normally, we could merge the snapshot if you had datastore space, by cloning the existing virtual disk, I'm afraid your options, are Converting the VM to a Workstation Disk. This will merge the snapshot.
Deleting the original VM, and then restoring or converting back.
ASKER
This is the datastore, I just greyed out the company name:
Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.
By doing this I was able to start the virtual machine but with an old version.
I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is: (354.051.231.744 bytes)
Note, as I said, last night I transferred the file SERVER 2012 Standa-000001-delta.vmdk to client and delete it in the datastore.
By doing this I was able to start the virtual machine but with an old version.
I still have the file SERVER 2012 Standa-000001-delta.vmdk on backup.
The file is: (354.051.231.744 bytes)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You will not be able to delete the Snapshot from Snapshot Manager, now the chain has been changed.
ASKER
It seems I can't even upload the 0001 to the datastore again. It just stays on 0% and I can see a file being created on the datastore.
I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?
I have the vmdk on a usb drive, is there a way to transfer it back to the datastore if I connect the usb drive directly to the server?
ASKER
I deleted the snapshots (in the beginning) by using this option:
Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
Select Inventory > Virtual Machine > Snapshot > Snapshot Manager.
In the Snapshot Manager, click a snapshot to select it.
Select a delete option
Yes, it's likely it's failed to delete the snapshot, because you did not have the disk space to merge.
You cannot connect it directly to server via USB, it's not supported.
It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.
Do you have another ESXi server?
Can you add more storage space?
even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
You cannot connect it directly to server via USB, it's not supported.
It's a shame you did not post earlier when you had the issue, there is a possibility we could have saved the Virtual Machine.
Do you have another ESXi server?
Can you add more storage space?
even via NFS or iSCSI attachment to NAS or Windows Server, Windows Workstation ?
Wow this is a messy situation...
Here is a long shot.
1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only
Here is a long shot.
1) Remove all snapshot files and any other reference to the snapshot
2) Open the VMX file and make sure it is point to the parent VMDK
3) Make sure the parent VMDK is set to read/write since a snapshot makes the parent read-only
ASKER
I ended up reinstalling the virtual Machine,
I managed to recover my data from a physical machine using getdataback for NTFS
I managed to recover my data from a physical machine using getdataback for NTFS
ASKER
since my virtual machine seems to be using the 00003 version, after safely backing up the file I deleted server2012-00001.vmdk from the datastore creating 336GB of space.
Then I consolidated the server and it was running again! BUT when i looked closer it was a old version of the virtual machine, it did not have the data I need to recover anymore.
Now my question is, what version is the correct one? can I safely delete (after backing it up) another vmdk file ? why are there multiple vmdk files?