Solved

VM is running on snapshot, and "configuration issues virtual machine consolidation is needed"

Posted on 2014-10-25
11
386 Views
Last Modified: 2016-02-25
Im running ESXi 5.5 with 2 datastore. Physical server is a DL380p GEN8 with P420i BBW controllor
DS1=raid5=140Gb space
DS2=raid1=700Gb space

On DS1 there's a SBS2011 server running with 2 virtual disks (see pics attached)
I used Veaam to backup the server, after that it wasn’t possible anymore to poweroff/poweron the machine due to a running task. Finally use esxtop to kill the server. That’s worked. But after that the message appears in vshpere: "configuration issues virtual machine consolidation is needed".

After that I read the Be patient post from Andrew Hancock (http://www.experts-exchange.com/Software/VMWare/A_10300-HOW-TO-VMware-Snapshots-Be-Patient.html) and noticed that this server I running on a snapshop for a while. But those snapshots aren't displayed in the snapshopmanager, but in the DS browser you can find them (see pic).

I a little scared after reading some post to use the "Consolidate" option in Vshpere for this server. Also I read that it's possible if the "Delete all" button is greyed out in the snapshotmanager, to make a new snapshot and after that click on the delete all button.  Because of the possibility that this can take hours or days, and the server cannot be powered off, this tasks need to be done online. Please advise me.

 I'm thinking about another solution. I'm using vcenter server and I have a second datastore with I can use to run this server on with enough space. Is it a good idea to clone the VM to this other DS without any snapshot so I can run it clean on DS2?

A few times i noticed that Veeam 6.1 is stuck on 99% of the backup on "processing server", probably thats the moment when veeam should remove the snapshot. What to do here?


Finally, is it better (performance) to run the SBS2011 server on the raid1 datastore?

THX!
0
Comment
Question by:sitpro
  • 6
  • 4
11 Comments
 
LVL 117
ID: 40404316
I wrote the article, so I known it's contents, so thanks firstly for reading it!

It's possible that the task that had hung, was the Snapshot Create or Remote task, this does happen from time to time (Veeam is not the only backup product that relies on the Storage API, e.g. and Snapshots to backup VMs, so do not think it's just Veeam, ALL vendors Backup products use the same Snapshot process!).

VMs running on a Snapshot is bad news

1. eventually the storage space will be full on your datastore, the VM will fail because there is no storage space, and possibly the disks can get corrupted.

2. When the VM is running on a Snapshot - Performance is POOR!

ACT NOW! DO not ignore Snapshots, waiting they get bigger, and longer to delete.

The process is as stated

1. Take a Snapshot
2. Wait 60 seconds
3. DELETE ALL
4. WAIT AND BE PATIENT!

How long to wait, depends on speed of datastore and how large the snapshot is..... it could take seconds, minutes, days or weeks....

You need to check ALL VMs after every backup.....for left over snapshots and ACT! (as above!).

If you cannot get emergency maintenance to take the server offline and power off, then you will have to do it ONLINE, but Snapshot Merge/DELETE ALL will be slower, and performance will get worse for your END Users!

So, you choice Offline or Online....

The issue is, you have three snapshots, and hence the reason why it possible hung....

DS1 RAID5, could be too slow to handle the snapshot deletion....

So options..

1. Delete Snapshot ONLINE

2. Delete Snapshot OFFLINE.

I actually favour a third option....

3. Power OFF the VM, and Right Click the VM, and click CLONE this VM to a new datastore, this will MERGE all the Snapshots, power up this new VM, and delete the OLD.
0
 

Author Comment

by:sitpro
ID: 40404666
thx Andrew,

i also prefer option3, but there's one issue. The server is a SBS2011 server which is being used almost daily. So i can't take the risk to bring it down and do the merge operation. Do you understand?

So the best way is to use vcenter to do the clone-operation to the new datastore? I must do that online. Do you have any tips or suggestions about this process? I saw that you can customise the process, or do i need to use the defaults?

Will the performance be very bad during this process for the endusers?

thx!!
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40404691
i also prefer option3, but there's one issue. The server is a SBS2011 server which is being used almost daily. So i can't take the risk to bring it down and do the merge operation. Do you understand?

Yes, I do understand, that we work in a 24x7x365 environments, but sometimes, we must perform emergency maintenance, as you leave your server running on a snapshot, if yo do nothing, eventually it will FAIL, and BE OFF, and it will take you longer to recover causing more service issues for users, and there is also the danger of corruption, and loss of data.

You will need to access, the risks and impact to users.

So the best way is to use vcenter to do the clone-operation to the new datastore? I must do that online. Do you have any tips or suggestions about this process? I saw that you can customise the process, or do i need to use the defaults?

Because the issue you are experiencing could be caused by the first large snapshot, and you will have to delete/merge all three. CLONE is a method of getting out of SNAPSHOT HELL, because part of the CLONE function will merge all the snapshots into the parent, leaving a CLONE of the original VM, but without snapshots.

You do not need to use the customization specification, just ignore it.

BUT to AVOID corruption, you must do it with the VM OFF.

Also, You cannot do a clone, because by the time the clone has finished, the VM, will be in use and your CLONE will be out of date.

I'm sorry, Emergency Downtime, and Shutdown the VM, and CLONE.
0
 

Author Comment

by:sitpro
ID: 40404694
Thats clear. We should bring it down and clone it to the other DS. But is there any indication of the process? It isnt possible to bring down the server for more then 1 day. And is it a solution to use veaam to image the server before the clone process, just for backup if the clone process corrupt the server.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40404711
CLONING will give you a new server, it will not touch the original server, so you will always have a server to rollback to.

At this point I would not use Veeam again, until the Snapshot situation has been cleared, and *ALWAYS* check after a backup there is no snapshot.

It does show the process, when you are cloning. Speed of clone does depend on speed of disks.
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 

Author Comment

by:sitpro
ID: 40404713
Thats good to know. And if the clone operation take to long time, can you cancell it?

Is there any extra space needed on the DS where the server now is located?
Im i right that vcenter calculate of theres enough space on the destination DS?
0
 
LVL 117
ID: 40404840
Yes, you can cancel the CLONE operation. Right Click the Task, CANCEL.

No additional space is needed,  (it should perform better on the DS2 RAID1 datastore).

Yes, it will not perform the CLONE, unless there is adequate space.
0
 

Author Comment

by:sitpro
ID: 40404928
I shall plan the operation an let you know the results. Many thanks for now!
0
 

Author Comment

by:sitpro
ID: 40498814
Finally did the clone-action described by Andrey. The VM contains 2 virtual disk:

- one disk with 180Gb
- one disk with 300Gb

Clone the VM from a VMware Raid5 (HP P410 with sas disks) to the same server to a raid1 (2 900Gb sas 10k). The process takes aprox 2 hours. The overall performance of this VM  has become a lot faster after this snapshot cleanup and moving to the raid1 volume.

Also the Veeam backup procedure is a lot faster now. When backing up this VM (totaly  500Gb) to a NAS (Synology RS812) by network, it take 1.58 minutes (70mb/sec).
0
 

Author Closing Comment

by:sitpro
ID: 40498815
Many thanks to Andrew Hancock for his VMware knowledge. And for sure: do not trust VMware procesbar and WAIT AND BE PATIENT!
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Join & Write a Comment

Suggested Solutions

HOW TO: Install and Configure VMware vSphere Hypervisor 6.5 (ESXi 6.5), Step by Step Tutorial with screenshots. From Download, Checking Media, to Completed Installation.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
Teach the user how to rename, unmount, delete and upgrade VMFS datastores. Open vSphere Web Client: Rename VMFS and NFS datastores: Upgrade VMFS-3 volume to VMFS-5: Unmount VMFS datastore: Delete a VMFS datastore:
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now