snapshots stuck in datastore, getting big

Greetings,

bit if a mess here - at least from my perspective. really hoping this is a no-brainer for you experts. Exchange server is a VM. had a couple of issues with veeam 5 backups failing (various reasons). on the last failed run before I deleted out the job and ran a full, which completed successfully, there was an error starting it couldn't remove a snapshot. The Veeam snapshot was removed on the full that was successful but somehow a VM snapshot was created - I think. I have no delta files in the datastore but I do have a disk named BU-EXCH1-000002.vmdk that is almost 27 GB now. I believe this file is holding changes to the server since it was created. It is hard for me to believe that the exchange server has accumulated this much info over a few days, but it has. I have increased the size of my datastore to give the file room to grow until I can figure out how to get rid of it. I was planning on going through snapshot manager and doing a delete all (snapshot manager only shows the consolidated helper, no other snapshots) but was concerned this may fail or cripple the machines performance while it worked. I used converter last night to test a conversion and it worked fine. However, it rolled up the snapshot, as expected, and increased my vmdk file by 27 GB. anyway to avoid that or is that taken up disk space just a casualty of an errant snapshot?

Anyway, suggestions greatly appreciated. I need to be careful as this is my exchange server. I can take it offline / power off if needed but not until tomorrow night (Friday night).

Thanks a lot
LVL 7
king daddyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

king daddyAuthor Commented:
here is a screenshot of the datastore
datastore
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Okay, what you need to do, is Remove all the Snapshots from Snapshots Manager.

But first to check everything is okay, Create a Snapshot in Snapshot Manager, and wait 60 seconds, and then delete it. This will confirm if Snapshot are working correctly.

After the above check the datastore and see if you have any remaining snapshots.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Okay, you have one snapshot, check the above.
CEOs need to know what they should worry about

Nearly every week during the past few years has featured a headline about the latest data breach, malware attack, ransomware demand, or unrecoverable corporate data loss. Those stories are frequently followed by news that the CEOs at those companies were forced to resign.

king daddyAuthor Commented:
thanks for the quick responses hanccocka. the server is about 450 GB. will a test snapshot complete that quickly? I have about 24 GB free space in the datastore. is that sufficient for a test snapshot? I think it is but would like verification. then, if I do a delete all from snapshot manager, won't that greatly affect performance of the server? additionally, I have read that a snapshot of that size will take several hours to be removed. if so, I think I'll wait until tonight unless you see no reason to wait.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Creating a Snapshot on any busy production server can have risks.

Because the Snapshot process, can temporaily, freeze the machine, for a very short time.

The best thing would be to complete this operation at a quiet, time and arrange for a scheduled service outage.

Because later to delete the large snapshot, merging the 25GB back into a 450GB main vmdk virtual disk, will have some impact on the running service.


Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Depending upon how fast your datastore is, it can take several hours to write all the data from the delta 25GB snapshot file into the 450GB main vmdk file, once this data has been merged the delta 25GB will be removed.

How much space do you have on the datastore?

I'm sure you already know this....

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  
king daddyAuthor Commented:
I will schedule downtime this evening and perform these actions. I will post back as soon as it is completed. If I schedule downtime, and since I would like to increase the size of the two disk partitions on the server, can I just do another conversion with vConverter, power down the current exchange server, power up the converted one, and go from there? Since that'll roll-up / remove the snapshot, I may go that route unless you have a reasons to not do so.

 thanks again for your help

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes, VMware Converter is one of the VMware recommended ways of getting out of Snapshot hell.

Download VMware vCenter Converter here

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vcenter_converter_standalone/4_0

VMware vCenter Converter Standalone 4.x Documentation

http://www.vmware.com/support/pubs/converter_pubs.html

VMware vCenter Converter Standalone 4.3 User Guide

http://www.vmware.com/pdf/convsa_43_guide.pdf

For the conversion steps, read fellow Expert Bestway's article.

http://www.experts-exchange.com/Software/VMWare/A_3639-VMware-vConverter-P2V-for-Windows-Servers.html

Best Practice Video Guide here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004588

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
If you are doing the VMware Conversion, there is no need to committ the snapshot.

Just complete a conversion to a fresh new extended disk.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Understanding Snaps - http://kb.vmware.com/kb/1015180
Snap Best Practices - http://kb.vmware.com/kb/1025279
king daddyAuthor Commented:
the datastore is 492 GB with about 24 GB free right now. Thanks for the snapshot info and helpful inks. However, I never create snapshots and no one else here has access to the vCenter or the servers. The only reason I noticed is after a Veeam backup that failed, I received an email stating so. When I went to check it out, I noticed my free space on the datastore was off about 10 GB. That's when I noticed the snapshot. No idea how / when it was created. I also noticed there is no delta file. just the vmdk for 00002.

Thanks again
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes, did you know you can set up vCenter for Snapshots?

Just in case you didn't know, this so it can warn you if Veeam doesn't delete them!
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
That's an Alarm!
king daddyAuthor Commented:
Thing is Veeam states it's not there snapshot as the logs show Veeam snaps were deleted correctly.

You stated that's an alarm - meaning an alarm like something is wrong or an alarm meaning a configured alarm in vCenter? If the latter it must be built-in because I didn't set up one.

Is it odd there is no delta file? Also, only if you have time, can you briefly explain why the file is so big? The server doesn't pass that many emails.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes, you can set an alarm in vCenter now, so when a Snapshot is present, you get an Alert/Alarm,

Handy for snapshots which are hanging around. (that shouldn't be).

As for the size, all changed blocks in the main VMDK are written to the snapshot.

So there must have been this many changes on the server.

The larger the main virtual disk, the larger the snapshot.

Not be virus scanning the virtual server, defragmenting the server, the ctk file is the change block tracking file, which tracks changes between backups to make Incremental Backups faster, so there have been this many changes in the VM, as this is quirte large as well.

king daddyAuthor Commented:
That's so weird to me because I dont touch this server, it's just runs exchange and gets backed up by Veeam. Unreal to me how this many changes could occur. Anyway, thanks again. Will update.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Change's will alwauys occur in a running machine, just by powering it on and leaving and do nothing.

we've checked, power up a Windows file server, create a snapshot, and leave for 24 hours.

the snapshot will be 2GB. with no users, access or anything.

So if you have an Exchange Server, lots of disk read and writes are happening in the stores. Online defragmentation of the datastores.

How many Exchange Store log files are generated a day on the server?
king daddyAuthor Commented:
no idea about the log files but I guess enough to grow this file by several GBs over 5 days or so. I will check.

king daddyAuthor Commented:
sorry to impose with yet another question(s), but when running the conversion this evening (my sanity can't wait until friday night), should I run the stand-alone from the exchange server (that's what I did last night) or should I power down the server and run it from another machine? If the former, I should stop all of the exchange services, correct?

thx again
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
To complete an exchange conversion, I would recommend COLD CLONE, or HOT with Converter installed on the server, but make sure you shutdown all the services.
king daddyAuthor Commented:
Cold it is. just to note, I considered using Veeam to make a replica of the server, with the services stopped, but I noticed after running it that the snapshot was not rolled up into the replica machine as the vmdk size was the same as the original in the datastore while the test conversion I ran last night had added the snapshot to the new vmdk increasing it by 27 GB or so. anyway...
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Good Luck.
quankenyuCommented:
you may run the following powerCLI command to check snapshot age

Connect-VIserver vcenter
$SnapshotAge = 14
$Date = Get-Date
foreach-object {Get-VM | Get-Snapshot | Where {$_.Created -lt (($Date).AddDays(-$SnapshotAge))} }

Here vcenter is your vcenter server name
Snapshotage =#, the number of days you want to check
king daddyAuthor Commented:
rather than committing the snapshot, I went with the cold conversion. worked great. actually edited the VM specs, including adding space to the drives. running fine.

thanks again hanccocka, always great directions and info.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.