Solved

NetApp Snapshot Volume is over 100%

Posted on 2013-06-18
10
1,830 Views
Last Modified: 2013-07-12
We received an alert from LogicMonitor.com at 5:50AM, with the below message. About 2 hours prior to that (at 3:37AM), Veeam ran a backup of our Exchange server where vCenter triggered a "VM with snapshots" alert as well (below), but the vCenter alert was cleared at 5:36AM.

I understand Veeam creates snapshots to backup a VM, but then it deletes it. I checked the Snapshot manager of our Exchange VM and it is showing there are no snapshots right now (good). So why is LogicMonitor still giving us the alert.

vCenter Alert at 3:37 AM
Target: i1whlmail01
Previous Status: Green
New Status: Red
 
Alarm Definition:
([Yellow Metric Is above 2,097,152KB; Red Metric Is above 3,145,728KB])
 
Current values for metric/state:
 Metric Overhead due to delta disk backings = 50,261,657KB
 
Description:
Alarm 'VM with snapshots' on i1whlmail01 changed from Green to Red


vCenter Alert at 5:36 AM
Target: i1whlmail01
Previous Status: Red
New Status: Green
 
Alarm Definition:
([Yellow Metric Is above 2,097,152KB; Red Metric Is above 3,145,728KB])
 
Current values for metric/state:
 Metric Overhead due to delta disk backings = 0KB
 
Description:
Alarm 'VM with snapshots' on i1whlmail01 changed from Red to Green


LogicMonitor Alert at 5:50 AM
The NetApp i1whlnetapp01, volume /vol/Napp01_vol2/.snapshot , is now at 166 percent of space capacity, putting it in a state of warn.
Thresholds are > 100.
This state has existed since 2013-06-18 05:37:09 PDT - or for 0h 12m.
When the snapshot volume is over 100%, it means that deleting a file will not free up space in the file system until any referencing snapshot is deleted. This makes managing disk space harder.
Recommendation: check to ensure that the snapshot policy in effect is appropriate for the kind of data being stored on this filer. If so, increase the snapshot reserve.
Otherwise, change the snapshot policy or delete snapshots.
0
Comment
Question by:pzozulka
  • 4
  • 3
  • 2
  • +1
10 Comments
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
Comment Utility
Firstly, sometimes the VMware Snapshot Alert is "faulty!".

Check my EE Article to check your VM is not on a snapshot!

HOW TO: VMware Snapshots :- Be Patient

The Snapshot Alert you have received is because of Snapshots on your SAN for the volume/LUN.

Do you use NetApp Snapshots or SnapMirror?
0
 
LVL 8

Author Comment

by:pzozulka
Comment Utility
I've read your HOW TO article in the past and it has saved me many times before. Very well written. To answer you question, we use both snapshots and SnapMirror. This volume is snapmirrored every 3 hours at the 15 min mark.

I believe the vCenter alert about snapshots, and the storage device snapshot alert are related.

snapshotsvolume2 snapshot
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
If you use SnapMirror, it creates a SAN Snapshot.

Is your VM running on a snapshot?
0
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 250 total points
Comment Utility
The snapshots that are busy are the snapmirror snapshots and can only be deleted if/when the snapmirror is finished or you can queisce/break snapmirror, delete snapshot and review if you have a common snapshot on both sides you can continue to replicate from.

Snapshot reserve is just for accounting so that if you have space in the volume it will not do anything to your data, on the other hand if your snapshots start to encroash into your data your volume may go offline as you'll run out of space.
0
 
LVL 8

Author Comment

by:pzozulka
Comment Utility
I'm just trying to figure out what is causing this to occur:

I was under the impression that snapshots start out at zero, and then grow based on changes in the data compared to the previous snapshot. Having said that, based on my previous post, what does 93.91GB Cumalitve Total Size mean? Does that mean that there has been 93.91GB of changes since the last snapshot?

And lastly, what's the difference between the two columns, Total Size vs. Cumalitive Total Size?
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
sizes of snapshots will be based on the "delta" changes between each snapshot.
0
 
LVL 8

Author Comment

by:pzozulka
Comment Utility
Correct, so does that mean that on 6/18/2013 03:15:03 (from above image), there was 92.65GB of changes?

I'm not sure how to read the above image and Total Size vs. Cumalitive Total Size.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
total size is the size of the snapshot, (delta change, e.g. changes between snapshots)

cumulative all the snapshots.
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
The Netapp snapshots work a bit different than your copy on write snapshots in that they're pointer to the data.  So let's say you've added a VM, no snapshot space is being used but if you have a snapshot with a VM in place and you delete it then the snapshot will be holding the data and the snapshot will grow.  It can also be changes inside of the VMs, let's say a database that will look as different data if it's encrypted or binary where there is no way to dedupe it on the WAFL.
0
 
LVL 19

Expert Comment

by:compdigit44
Comment Utility
Have you checked your datastore or orphan snapshots? Sometime not all snapshots are removed completed and are not listed in snapshot manager.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
When we have a dead host and we lose all connections to the ESXi, and we need to find a way to move all VMs from that dead ESXi host.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now