Solved

VM is running on snapshot, and "configuration issues virtual machine consolidation is needed"

Posted on 2014-10-25
11
423 Views
Last Modified: 2016-02-25
Im running ESXi 5.5 with 2 datastore. Physical server is a DL380p GEN8 with P420i BBW controllor
DS1=raid5=140Gb space
DS2=raid1=700Gb space

On DS1 there's a SBS2011 server running with 2 virtual disks (see pics attached)
I used Veaam to backup the server, after that it wasn’t possible anymore to poweroff/poweron the machine due to a running task. Finally use esxtop to kill the server. That’s worked. But after that the message appears in vshpere: "configuration issues virtual machine consolidation is needed".

After that I read the Be patient post from Andrew Hancock (http://www.experts-exchange.com/Software/VMWare/A_10300-HOW-TO-VMware-Snapshots-Be-Patient.html) and noticed that this server I running on a snapshop for a while. But those snapshots aren't displayed in the snapshopmanager, but in the DS browser you can find them (see pic).

I a little scared after reading some post to use the "Consolidate" option in Vshpere for this server. Also I read that it's possible if the "Delete all" button is greyed out in the snapshotmanager, to make a new snapshot and after that click on the delete all button.  Because of the possibility that this can take hours or days, and the server cannot be powered off, this tasks need to be done online. Please advise me.

 I'm thinking about another solution. I'm using vcenter server and I have a second datastore with I can use to run this server on with enough space. Is it a good idea to clone the VM to this other DS without any snapshot so I can run it clean on DS2?

A few times i noticed that Veeam 6.1 is stuck on 99% of the backup on "processing server", probably thats the moment when veeam should remove the snapshot. What to do here?


Finally, is it better (performance) to run the SBS2011 server on the raid1 datastore?

THX!
0
Comment
Question by:sitpro
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
11 Comments
 
LVL 120
ID: 40404316
I wrote the article, so I known it's contents, so thanks firstly for reading it!

It's possible that the task that had hung, was the Snapshot Create or Remote task, this does happen from time to time (Veeam is not the only backup product that relies on the Storage API, e.g. and Snapshots to backup VMs, so do not think it's just Veeam, ALL vendors Backup products use the same Snapshot process!).

VMs running on a Snapshot is bad news

1. eventually the storage space will be full on your datastore, the VM will fail because there is no storage space, and possibly the disks can get corrupted.

2. When the VM is running on a Snapshot - Performance is POOR!

ACT NOW! DO not ignore Snapshots, waiting they get bigger, and longer to delete.

The process is as stated

1. Take a Snapshot
2. Wait 60 seconds
3. DELETE ALL
4. WAIT AND BE PATIENT!

How long to wait, depends on speed of datastore and how large the snapshot is..... it could take seconds, minutes, days or weeks....

You need to check ALL VMs after every backup.....for left over snapshots and ACT! (as above!).

If you cannot get emergency maintenance to take the server offline and power off, then you will have to do it ONLINE, but Snapshot Merge/DELETE ALL will be slower, and performance will get worse for your END Users!

So, you choice Offline or Online....

The issue is, you have three snapshots, and hence the reason why it possible hung....

DS1 RAID5, could be too slow to handle the snapshot deletion....

So options..

1. Delete Snapshot ONLINE

2. Delete Snapshot OFFLINE.

I actually favour a third option....

3. Power OFF the VM, and Right Click the VM, and click CLONE this VM to a new datastore, this will MERGE all the Snapshots, power up this new VM, and delete the OLD.
0
 

Author Comment

by:sitpro
ID: 40404666
thx Andrew,

i also prefer option3, but there's one issue. The server is a SBS2011 server which is being used almost daily. So i can't take the risk to bring it down and do the merge operation. Do you understand?

So the best way is to use vcenter to do the clone-operation to the new datastore? I must do that online. Do you have any tips or suggestions about this process? I saw that you can customise the process, or do i need to use the defaults?

Will the performance be very bad during this process for the endusers?

thx!!
0
 
LVL 120

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 40404691
i also prefer option3, but there's one issue. The server is a SBS2011 server which is being used almost daily. So i can't take the risk to bring it down and do the merge operation. Do you understand?

Yes, I do understand, that we work in a 24x7x365 environments, but sometimes, we must perform emergency maintenance, as you leave your server running on a snapshot, if yo do nothing, eventually it will FAIL, and BE OFF, and it will take you longer to recover causing more service issues for users, and there is also the danger of corruption, and loss of data.

You will need to access, the risks and impact to users.

So the best way is to use vcenter to do the clone-operation to the new datastore? I must do that online. Do you have any tips or suggestions about this process? I saw that you can customise the process, or do i need to use the defaults?

Because the issue you are experiencing could be caused by the first large snapshot, and you will have to delete/merge all three. CLONE is a method of getting out of SNAPSHOT HELL, because part of the CLONE function will merge all the snapshots into the parent, leaving a CLONE of the original VM, but without snapshots.

You do not need to use the customization specification, just ignore it.

BUT to AVOID corruption, you must do it with the VM OFF.

Also, You cannot do a clone, because by the time the clone has finished, the VM, will be in use and your CLONE will be out of date.

I'm sorry, Emergency Downtime, and Shutdown the VM, and CLONE.
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Author Comment

by:sitpro
ID: 40404694
Thats clear. We should bring it down and clone it to the other DS. But is there any indication of the process? It isnt possible to bring down the server for more then 1 day. And is it a solution to use veaam to image the server before the clone process, just for backup if the clone process corrupt the server.
0
 
LVL 120

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 40404711
CLONING will give you a new server, it will not touch the original server, so you will always have a server to rollback to.

At this point I would not use Veeam again, until the Snapshot situation has been cleared, and *ALWAYS* check after a backup there is no snapshot.

It does show the process, when you are cloning. Speed of clone does depend on speed of disks.
0
 

Author Comment

by:sitpro
ID: 40404713
Thats good to know. And if the clone operation take to long time, can you cancell it?

Is there any extra space needed on the DS where the server now is located?
Im i right that vcenter calculate of theres enough space on the destination DS?
0
 
LVL 120
ID: 40404840
Yes, you can cancel the CLONE operation. Right Click the Task, CANCEL.

No additional space is needed,  (it should perform better on the DS2 RAID1 datastore).

Yes, it will not perform the CLONE, unless there is adequate space.
0
 

Author Comment

by:sitpro
ID: 40404928
I shall plan the operation an let you know the results. Many thanks for now!
0
 

Author Comment

by:sitpro
ID: 40498814
Finally did the clone-action described by Andrey. The VM contains 2 virtual disk:

- one disk with 180Gb
- one disk with 300Gb

Clone the VM from a VMware Raid5 (HP P410 with sas disks) to the same server to a raid1 (2 900Gb sas 10k). The process takes aprox 2 hours. The overall performance of this VM  has become a lot faster after this snapshot cleanup and moving to the raid1 volume.

Also the Veeam backup procedure is a lot faster now. When backing up this VM (totaly  500Gb) to a NAS (Synology RS812) by network, it take 1.58 minutes (70mb/sec).
0
 

Author Closing Comment

by:sitpro
ID: 40498815
Many thanks to Andrew Hancock for his VMware knowledge. And for sure: do not trust VMware procesbar and WAIT AND BE PATIENT!
0

Featured Post

NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Unable to remove VSphere from Host Machine 5 78
Migrate VM to other host 12 101
Deploying DAG on Exchange 2013 Standard edition 10 95
VM replication scenario and caveats ? 6 48
When converting a physical machine to a virtual machine using VMware vCenter Converter Standalone or vCenter Converter Enterprise, if an adapter type is not selected during the initial customization the resulting virtual machine may contain an IDE d…
HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question