snapshots stuck in datastore, getting big
Greetings,
bit if a mess here - at least from my perspective. really hoping this is a no-brainer for you experts. Exchange server is a VM. had a couple of issues with veeam 5 backups failing (various reasons). on the last failed run before I deleted out the job and ran a full, which completed successfully, there was an error starting it couldn't remove a snapshot. The Veeam snapshot was removed on the full that was successful but somehow a VM snapshot was created - I think. I have no delta files in the datastore but I do have a disk named BU-EXCH1-000002.vmdk that is almost 27 GB now. I believe this file is holding changes to the server since it was created. It is hard for me to believe that the exchange server has accumulated this much info over a few days, but it has. I have increased the size of my datastore to give the file room to grow until I can figure out how to get rid of it. I was planning on going through snapshot manager and doing a delete all (snapshot manager only shows the consolidated helper, no other snapshots) but was concerned this may fail or cripple the machines performance while it worked. I used converter last night to test a conversion and it worked fine. However, it rolled up the snapshot, as expected, and increased my vmdk file by 27 GB. anyway to avoid that or is that taken up disk space just a casualty of an errant snapshot?
Anyway, suggestions greatly appreciated. I need to be careful as this is my exchange server. I can take it offline / power off if needed but not until tomorrow night (Friday night).
Thanks a lot
bit if a mess here - at least from my perspective. really hoping this is a no-brainer for you experts. Exchange server is a VM. had a couple of issues with veeam 5 backups failing (various reasons). on the last failed run before I deleted out the job and ran a full, which completed successfully, there was an error starting it couldn't remove a snapshot. The Veeam snapshot was removed on the full that was successful but somehow a VM snapshot was created - I think. I have no delta files in the datastore but I do have a disk named BU-EXCH1-000002.vmdk that is almost 27 GB now. I believe this file is holding changes to the server since it was created. It is hard for me to believe that the exchange server has accumulated this much info over a few days, but it has. I have increased the size of my datastore to give the file room to grow until I can figure out how to get rid of it. I was planning on going through snapshot manager and doing a delete all (snapshot manager only shows the consolidated helper, no other snapshots) but was concerned this may fail or cripple the machines performance while it worked. I used converter last night to test a conversion and it worked fine. However, it rolled up the snapshot, as expected, and increased my vmdk file by 27 GB. anyway to avoid that or is that taken up disk space just a casualty of an errant snapshot?
Anyway, suggestions greatly appreciated. I need to be careful as this is my exchange server. I can take it offline / power off if needed but not until tomorrow night (Friday night).
Thanks a lot
Okay, what you need to do, is Remove all the Snapshots from Snapshots Manager.
But first to check everything is okay, Create a Snapshot in Snapshot Manager, and wait 60 seconds, and then delete it. This will confirm if Snapshot are working correctly.
After the above check the datastore and see if you have any remaining snapshots.
But first to check everything is okay, Create a Snapshot in Snapshot Manager, and wait 60 seconds, and then delete it. This will confirm if Snapshot are working correctly.
After the above check the datastore and see if you have any remaining snapshots.
Okay, you have one snapshot, check the above.
ASKER
thanks for the quick responses hanccocka. the server is about 450 GB. will a test snapshot complete that quickly? I have about 24 GB free space in the datastore. is that sufficient for a test snapshot? I think it is but would like verification. then, if I do a delete all from snapshot manager, won't that greatly affect performance of the server? additionally, I have read that a snapshot of that size will take several hours to be removed. if so, I think I'll wait until tonight unless you see no reason to wait.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Depending upon how fast your datastore is, it can take several hours to write all the data from the delta 25GB snapshot file into the 450GB main vmdk file, once this data has been merged the delta 25GB will be removed.
How much space do you have on the datastore?
I'm sure you already know this....
A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes. When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file. There is an active link between the original VMDK file and the new delta file. Anything that gets written to the VM actually gets written to the delta file. The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM. If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.
How much space do you have on the datastore?
I'm sure you already know this....
A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes. When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file. There is an active link between the original VMDK file and the new delta file. Anything that gets written to the VM actually gets written to the delta file. The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM. If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.
ASKER
I will schedule downtime this evening and perform these actions. I will post back as soon as it is completed. If I schedule downtime, and since I would like to increase the size of the two disk partitions on the server, can I just do another conversion with vConverter, power down the current exchange server, power up the converted one, and go from there? Since that'll roll-up / remove the snapshot, I may go that route unless you have a reasons to not do so.
thanks again for your help
thanks again for your help
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
If you are doing the VMware Conversion, there is no need to committ the snapshot.
Just complete a conversion to a fresh new extended disk.
Just complete a conversion to a fresh new extended disk.
Understanding Snaps - http://kb.vmware.com/kb/1015180
Snap Best Practices - http://kb.vmware.com/kb/1025279
Snap Best Practices - http://kb.vmware.com/kb/1025279
ASKER
the datastore is 492 GB with about 24 GB free right now. Thanks for the snapshot info and helpful inks. However, I never create snapshots and no one else here has access to the vCenter or the servers. The only reason I noticed is after a Veeam backup that failed, I received an email stating so. When I went to check it out, I noticed my free space on the datastore was off about 10 GB. That's when I noticed the snapshot. No idea how / when it was created. I also noticed there is no delta file. just the vmdk for 00002.
Thanks again
Thanks again
Yes, did you know you can set up vCenter for Snapshots?
Just in case you didn't know, this so it can warn you if Veeam doesn't delete them!
Just in case you didn't know, this so it can warn you if Veeam doesn't delete them!
That's an Alarm!
ASKER
Thing is Veeam states it's not there snapshot as the logs show Veeam snaps were deleted correctly.
You stated that's an alarm - meaning an alarm like something is wrong or an alarm meaning a configured alarm in vCenter? If the latter it must be built-in because I didn't set up one.
Is it odd there is no delta file? Also, only if you have time, can you briefly explain why the file is so big? The server doesn't pass that many emails.
You stated that's an alarm - meaning an alarm like something is wrong or an alarm meaning a configured alarm in vCenter? If the latter it must be built-in because I didn't set up one.
Is it odd there is no delta file? Also, only if you have time, can you briefly explain why the file is so big? The server doesn't pass that many emails.
Yes, you can set an alarm in vCenter now, so when a Snapshot is present, you get an Alert/Alarm,
Handy for snapshots which are hanging around. (that shouldn't be).
As for the size, all changed blocks in the main VMDK are written to the snapshot.
So there must have been this many changes on the server.
The larger the main virtual disk, the larger the snapshot.
Not be virus scanning the virtual server, defragmenting the server, the ctk file is the change block tracking file, which tracks changes between backups to make Incremental Backups faster, so there have been this many changes in the VM, as this is quirte large as well.
Handy for snapshots which are hanging around. (that shouldn't be).
As for the size, all changed blocks in the main VMDK are written to the snapshot.
So there must have been this many changes on the server.
The larger the main virtual disk, the larger the snapshot.
Not be virus scanning the virtual server, defragmenting the server, the ctk file is the change block tracking file, which tracks changes between backups to make Incremental Backups faster, so there have been this many changes in the VM, as this is quirte large as well.
ASKER
That's so weird to me because I dont touch this server, it's just runs exchange and gets backed up by Veeam. Unreal to me how this many changes could occur. Anyway, thanks again. Will update.
Change's will alwauys occur in a running machine, just by powering it on and leaving and do nothing.
we've checked, power up a Windows file server, create a snapshot, and leave for 24 hours.
the snapshot will be 2GB. with no users, access or anything.
So if you have an Exchange Server, lots of disk read and writes are happening in the stores. Online defragmentation of the datastores.
How many Exchange Store log files are generated a day on the server?
we've checked, power up a Windows file server, create a snapshot, and leave for 24 hours.
the snapshot will be 2GB. with no users, access or anything.
So if you have an Exchange Server, lots of disk read and writes are happening in the stores. Online defragmentation of the datastores.
How many Exchange Store log files are generated a day on the server?
ASKER
no idea about the log files but I guess enough to grow this file by several GBs over 5 days or so. I will check.
ASKER
sorry to impose with yet another question(s), but when running the conversion this evening (my sanity can't wait until friday night), should I run the stand-alone from the exchange server (that's what I did last night) or should I power down the server and run it from another machine? If the former, I should stop all of the exchange services, correct?
thx again
thx again
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Cold it is. just to note, I considered using Veeam to make a replica of the server, with the services stopped, but I noticed after running it that the snapshot was not rolled up into the replica machine as the vmdk size was the same as the original in the datastore while the test conversion I ran last night had added the snapshot to the new vmdk increasing it by 27 GB or so. anyway...
you may run the following powerCLI command to check snapshot age
Connect-VIserver vcenter
$SnapshotAge = 14
$Date = Get-Date
foreach-object {Get-VM | Get-Snapshot | Where {$_.Created -lt (($Date).AddDays(-$Snapsho
Here vcenter is your vcenter server name
Snapshotage =#, the number of days you want to check
Connect-VIserver vcenter
$SnapshotAge = 14
$Date = Get-Date
foreach-object {Get-VM | Get-Snapshot | Where {$_.Created -lt (($Date).AddDays(-$Snapsho
Here vcenter is your vcenter server name
Snapshotage =#, the number of days you want to check
ASKER
rather than committing the snapshot, I went with the cold conversion. worked great. actually edited the VM specs, including adding space to the drives. running fine.
thanks again hanccocka, always great directions and info.
thanks again hanccocka, always great directions and info.
ASKER