?
Solved

snapshots stuck in datastore, getting big

Posted on 2011-05-12
24
Medium Priority
?
5,132 Views
Last Modified: 2012-06-27
Greetings,

bit if a mess here - at least from my perspective. really hoping this is a no-brainer for you experts. Exchange server is a VM. had a couple of issues with veeam 5 backups failing (various reasons). on the last failed run before I deleted out the job and ran a full, which completed successfully, there was an error starting it couldn't remove a snapshot. The Veeam snapshot was removed on the full that was successful but somehow a VM snapshot was created - I think. I have no delta files in the datastore but I do have a disk named BU-EXCH1-000002.vmdk that is almost 27 GB now. I believe this file is holding changes to the server since it was created. It is hard for me to believe that the exchange server has accumulated this much info over a few days, but it has. I have increased the size of my datastore to give the file room to grow until I can figure out how to get rid of it. I was planning on going through snapshot manager and doing a delete all (snapshot manager only shows the consolidated helper, no other snapshots) but was concerned this may fail or cripple the machines performance while it worked. I used converter last night to test a conversion and it worked fine. However, it rolled up the snapshot, as expected, and increased my vmdk file by 27 GB. anyway to avoid that or is that taken up disk space just a casualty of an errant snapshot?

Anyway, suggestions greatly appreciated. I need to be careful as this is my exchange server. I can take it offline / power off if needed but not until tomorrow night (Friday night).

Thanks a lot
0
Comment
Question by:rpliner
  • 13
  • 10
24 Comments
 
LVL 7

Author Comment

by:rpliner
ID: 35746882
here is a screenshot of the datastore
datastore
0
 
LVL 124
ID: 35746901
Okay, what you need to do, is Remove all the Snapshots from Snapshots Manager.

But first to check everything is okay, Create a Snapshot in Snapshot Manager, and wait 60 seconds, and then delete it. This will confirm if Snapshot are working correctly.

After the above check the datastore and see if you have any remaining snapshots.
0
 
LVL 124
ID: 35746909
Okay, you have one snapshot, check the above.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 7

Author Comment

by:rpliner
ID: 35746958
thanks for the quick responses hanccocka. the server is about 450 GB. will a test snapshot complete that quickly? I have about 24 GB free space in the datastore. is that sufficient for a test snapshot? I think it is but would like verification. then, if I do a delete all from snapshot manager, won't that greatly affect performance of the server? additionally, I have read that a snapshot of that size will take several hours to be removed. if so, I think I'll wait until tonight unless you see no reason to wait.
0
 
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 35747027
Creating a Snapshot on any busy production server can have risks.

Because the Snapshot process, can temporaily, freeze the machine, for a very short time.

The best thing would be to complete this operation at a quiet, time and arrange for a scheduled service outage.

Because later to delete the large snapshot, merging the 25GB back into a 450GB main vmdk virtual disk, will have some impact on the running service.


0
 
LVL 124
ID: 35747046
Depending upon how fast your datastore is, it can take several hours to write all the data from the delta 25GB snapshot file into the 450GB main vmdk file, once this data has been merged the delta 25GB will be removed.

How much space do you have on the datastore?

I'm sure you already know this....

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  
0
 
LVL 7

Author Comment

by:rpliner
ID: 35747057
I will schedule downtime this evening and perform these actions. I will post back as soon as it is completed. If I schedule downtime, and since I would like to increase the size of the two disk partitions on the server, can I just do another conversion with vConverter, power down the current exchange server, power up the converted one, and go from there? Since that'll roll-up / remove the snapshot, I may go that route unless you have a reasons to not do so.

 thanks again for your help

0
 
LVL 124

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 35747079
Yes, VMware Converter is one of the VMware recommended ways of getting out of Snapshot hell.

Download VMware vCenter Converter here

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vcenter_converter_standalone/4_0

VMware vCenter Converter Standalone 4.x Documentation

http://www.vmware.com/support/pubs/converter_pubs.html

VMware vCenter Converter Standalone 4.3 User Guide

http://www.vmware.com/pdf/convsa_43_guide.pdf

For the conversion steps, read fellow Expert Bestway's article.

http://www.experts-exchange.com/Software/VMWare/A_3639-VMware-vConverter-P2V-for-Windows-Servers.html

Best Practice Video Guide here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004588
0
 
LVL 124
ID: 35747084
If you are doing the VMware Conversion, there is no need to committ the snapshot.

Just complete a conversion to a fresh new extended disk.
0
 
LVL 124
ID: 35747135
Understanding Snaps - http://kb.vmware.com/kb/1015180
Snap Best Practices - http://kb.vmware.com/kb/1025279
0
 
LVL 7

Author Comment

by:rpliner
ID: 35747260
the datastore is 492 GB with about 24 GB free right now. Thanks for the snapshot info and helpful inks. However, I never create snapshots and no one else here has access to the vCenter or the servers. The only reason I noticed is after a Veeam backup that failed, I received an email stating so. When I went to check it out, I noticed my free space on the datastore was off about 10 GB. That's when I noticed the snapshot. No idea how / when it was created. I also noticed there is no delta file. just the vmdk for 00002.

Thanks again
0
 
LVL 124
ID: 35747294
Yes, did you know you can set up vCenter for Snapshots?

Just in case you didn't know, this so it can warn you if Veeam doesn't delete them!
0
 
LVL 124
ID: 35747298
That's an Alarm!
0
 
LVL 7

Author Comment

by:rpliner
ID: 35747480
Thing is Veeam states it's not there snapshot as the logs show Veeam snaps were deleted correctly.

You stated that's an alarm - meaning an alarm like something is wrong or an alarm meaning a configured alarm in vCenter? If the latter it must be built-in because I didn't set up one.

Is it odd there is no delta file? Also, only if you have time, can you briefly explain why the file is so big? The server doesn't pass that many emails.

0
 
LVL 124
ID: 35747587
Yes, you can set an alarm in vCenter now, so when a Snapshot is present, you get an Alert/Alarm,

Handy for snapshots which are hanging around. (that shouldn't be).

As for the size, all changed blocks in the main VMDK are written to the snapshot.

So there must have been this many changes on the server.

The larger the main virtual disk, the larger the snapshot.

Not be virus scanning the virtual server, defragmenting the server, the ctk file is the change block tracking file, which tracks changes between backups to make Incremental Backups faster, so there have been this many changes in the VM, as this is quirte large as well.

0
 
LVL 7

Author Comment

by:rpliner
ID: 35747609
That's so weird to me because I dont touch this server, it's just runs exchange and gets backed up by Veeam. Unreal to me how this many changes could occur. Anyway, thanks again. Will update.
0
 
LVL 124
ID: 35747638
Change's will alwauys occur in a running machine, just by powering it on and leaving and do nothing.

we've checked, power up a Windows file server, create a snapshot, and leave for 24 hours.

the snapshot will be 2GB. with no users, access or anything.

So if you have an Exchange Server, lots of disk read and writes are happening in the stores. Online defragmentation of the datastores.

How many Exchange Store log files are generated a day on the server?
0
 
LVL 7

Author Comment

by:rpliner
ID: 35747863
no idea about the log files but I guess enough to grow this file by several GBs over 5 days or so. I will check.

0
 
LVL 7

Author Comment

by:rpliner
ID: 35748443
sorry to impose with yet another question(s), but when running the conversion this evening (my sanity can't wait until friday night), should I run the stand-alone from the exchange server (that's what I did last night) or should I power down the server and run it from another machine? If the former, I should stop all of the exchange services, correct?

thx again
0
 
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 35748472
To complete an exchange conversion, I would recommend COLD CLONE, or HOT with Converter installed on the server, but make sure you shutdown all the services.
0
 
LVL 7

Author Comment

by:rpliner
ID: 35748495
Cold it is. just to note, I considered using Veeam to make a replica of the server, with the services stopped, but I noticed after running it that the snapshot was not rolled up into the replica machine as the vmdk size was the same as the original in the datastore while the test conversion I ran last night had added the snapshot to the new vmdk increasing it by 27 GB or so. anyway...
0
 
LVL 124
ID: 35748507
Good Luck.
0
 
LVL 3

Expert Comment

by:quankenyu
ID: 35749848
you may run the following powerCLI command to check snapshot age

Connect-VIserver vcenter
$SnapshotAge = 14
$Date = Get-Date
foreach-object {Get-VM | Get-Snapshot | Where {$_.Created -lt (($Date).AddDays(-$SnapshotAge))} }

Here vcenter is your vcenter server name
Snapshotage =#, the number of days you want to check
0
 
LVL 7

Author Closing Comment

by:rpliner
ID: 35754474
rather than committing the snapshot, I went with the cold conversion. worked great. actually edited the VM specs, including adding space to the drives. running fine.

thanks again hanccocka, always great directions and info.
0

Featured Post

Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

As much as Microsoft wants to kill off PST file support, just as they tried to do with public folders, there are still times when it is useful or downright necessary to export Exchange mailboxes to PST files. Thankfully, it is still possible to e…
Steps to fix error: “Couldn’t mount the database that you specified. Specified database: HU-DB; Error code: An Active Manager operation fail”
This Micro Tutorial steps you through the configuration steps to configure your ESXi host Management Network settings and test the management network, ensure the host is recognized by the DNS Server, configure a new password, and the troubleshooting…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question