Solved

VMware - Monitor Disk Activity of VMs?

Posted on 2011-03-23
13
1,261 Views
Last Modified: 2012-06-21
Perhaps a silly question but is there a way to monitor disk activity of all my VM's?  I'm interested in finding out if a VM is writing excessively to it's disk.  Our VM's hard disk are located on a NetApp filer.  We mirror the NetApp volumes out of state to our disastor recovery site.  We are seeing large volumes of data getting mirrored occasionally to the disaster site.  A theory is that a VM may be writing large quantities of data to it's disk resulting in the large amounts of data getting mirrored.

We are running ESX 4.1.  Can I monitor using Virtual Center?  Is there a script that I can use?

Thanks in advance for your help,
Terry
0
Comment
Question by:wahlster
  • 4
  • 4
  • 2
  • +3
13 Comments
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 250 total points
ID: 35204269
Perhaps I can offer some insight.  

Are you using deduplication on the Netapp?  If so are you defragging the OS inside the VM or re-indexing the a SQL Database that jumbles (undedupes) the blocks causing a lot of data to appear as if it has changed.  What I would check is the total volume size on the Netapp (if using CIFS it makes it easy) as well checking to see if you're setup for thin provision (NFS is by default but LUNs may or may not be).

How large are your snapshots on the Netapp vs total volume size?  If you aren't compressing your snapmirrors you should configure compression as well..it's a little more overhead on the CPU but can cut down snapmirror size by 60%
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 35204294
Above I meant if using NFS to check total volume size..not CIFS..
0
 
LVL 16

Expert Comment

by:danm66
ID: 35204489
are you using vCenter server?  If so, you can review the various charts and measurements from the past to see what's been happening.  

Once you find the data you need, then you can see if there are appropriate alarms for them on the alarms tab (you can view/set the alarms at the vm/host/cluster/datacenter/VC levels, but only edit the default ones at the DC and VC levels).  On the alarms tab there are buttons in the upper left corner, click on the definitions to see what ones are in place and right-click in the white space to add a new one and you can see the different ones that are available.
0
 
LVL 4

Accepted Solution

by:
VMwareGuy earned 250 total points
ID: 35206301
vCenter 4.1 has new perf counters, here is what you want to do:

1)  select the VM in question in the inventory
2)  select the performance tab in the right pane
3) select the advanced button to bring up the advanced performance view
4)  click the link that says chart options
5)  select the DISK performance object
6)  examine the writes to disk, get the average and peaks, and you can also look at latency #s too

Now - when you say mirrored, are you sure it is being mirrored and not replicated via asynchronous replication?   I'm not certain how netapp does this because it is not SCSI based like iSCSI or FC storage, but I'm thinking it must have a way of only replicating changes to the data blocks as opposed to replicating full files...  but even still, any time you deploy a new VM on one of these replicated volumes the entire VM will get replicated the first time you lay it down on the disk and this can be expensive.    

 
0
 
LVL 14

Expert Comment

by:robincm
ID: 35206415
What VMware guy said. You can also get live stats out of vSphere client, but it seems to somewhat hammer the vCenter server.
Another way to get live raw numbers is to run esxtop on the service console of the ESX server, see http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1008205 which is entitled "using esxtop to identify storage performance issues".
0
 

Author Comment

by:wahlster
ID: 35207604
Thanks for your replies.  We are using Virtual Center.  As far as using deduplication on the Netapp I'm not sure but we are not compressing the snapshots.  LUNS are getting mirrored not replicated to our disaster site.  I looked at the Disk charts as suggested by VMwareGuy but nothing stands out.  We are going to give NetApp support a call to see if they can find something.
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 42

Expert Comment

by:paulsolov
ID: 35207972
Connect to the filer using netapp system manager. If you don't have it download from the now site. Check the volumes since lins can be setup for dedupe and thin provision. By replication I meant snapmirror
0
 
LVL 4

Expert Comment

by:VMwareGuy
ID: 35207973
Look at Disk \ write requests.. that should stand out.  It is recorded in a basic summation, which is confusing, because you have to look at the sample period, which is 20 seconds by default.   you can also look at the write rate peaks, recorded in Kbps.  the peak will incidate the the max KBps writes taking place.  you can compare these #s to other VMs on the same storage to see if it is indeed what is causing the excessive data writes.
0
 
LVL 2

Expert Comment

by:berkcomjonathan
ID: 35208849
If you have each VM's disk in a separate volume, from the filer you can do:

stats show -i 5 volume

That will give you a 5 second idea of volume statistics.  Then look for volumes that have high write_data (b/s). You can increase or decrease the interval (-i) to fit your needs.  I had a situation pretty similar to this.  I cloned a VM and failed to increase the memory on the clone.  It was swapping like a mofo, which meant heavy NFS writes to it's volume.
0
 

Author Comment

by:wahlster
ID: 35219279
Sorry about the delay.  Issue was a result of disk write as a result of an upgrade of Symantec's Enterprise Vault.  
Thanks for your help!
Terry
0
 

Author Comment

by:wahlster
ID: 35219300
I did not want to accept my answer as a solution.  
0
 

Author Comment

by:wahlster
ID: 35219309
Unselect my post as an answer.
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 35220004
If you're running EV that makes sense and also depends whether you write to a server that uses the Netapp or directly to the Netapp (allowing it do the compression and dedupe).  Also ensure that you are archiving and before snapmirror takes place as well as enable compression on the snapmirror.

Since EV archives and puts the data in as flat files your snapmirror shouldn't contain much more data then delta of your archive for the day unless you're ingesting your initial archive into the vault
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Teach the user how to delpoy the vCenter Server Appliance and how to configure its network settings Deploy OVF: Open VM console and configure networking:
Teach the user how to install ESXi 5.5 and configure the management network System Requirements: ESXi Installation:  Management Network Configuration: Management Network Testing:

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now