• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1447
  • Last Modified:

VMware - Monitor Disk Activity of VMs?

Perhaps a silly question but is there a way to monitor disk activity of all my VM's?  I'm interested in finding out if a VM is writing excessively to it's disk.  Our VM's hard disk are located on a NetApp filer.  We mirror the NetApp volumes out of state to our disastor recovery site.  We are seeing large volumes of data getting mirrored occasionally to the disaster site.  A theory is that a VM may be writing large quantities of data to it's disk resulting in the large amounts of data getting mirrored.

We are running ESX 4.1.  Can I monitor using Virtual Center?  Is there a script that I can use?

Thanks in advance for your help,
Terry
0
wahlster
Asked:
wahlster
  • 4
  • 4
  • 2
  • +3
2 Solutions
 
Paul SolovyovskySenior IT AdvisorCommented:
Perhaps I can offer some insight.  

Are you using deduplication on the Netapp?  If so are you defragging the OS inside the VM or re-indexing the a SQL Database that jumbles (undedupes) the blocks causing a lot of data to appear as if it has changed.  What I would check is the total volume size on the Netapp (if using CIFS it makes it easy) as well checking to see if you're setup for thin provision (NFS is by default but LUNs may or may not be).

How large are your snapshots on the Netapp vs total volume size?  If you aren't compressing your snapmirrors you should configure compression as well..it's a little more overhead on the CPU but can cut down snapmirror size by 60%
0
 
Paul SolovyovskySenior IT AdvisorCommented:
Above I meant if using NFS to check total volume size..not CIFS..
0
 
Danny McDanielClinical Systems AnalystCommented:
are you using vCenter server?  If so, you can review the various charts and measurements from the past to see what's been happening.  

Once you find the data you need, then you can see if there are appropriate alarms for them on the alarms tab (you can view/set the alarms at the vm/host/cluster/datacenter/VC levels, but only edit the default ones at the DC and VC levels).  On the alarms tab there are buttons in the upper left corner, click on the definitions to see what ones are in place and right-click in the white space to add a new one and you can see the different ones that are available.
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

 
VMwareGuyCommented:
vCenter 4.1 has new perf counters, here is what you want to do:

1)  select the VM in question in the inventory
2)  select the performance tab in the right pane
3) select the advanced button to bring up the advanced performance view
4)  click the link that says chart options
5)  select the DISK performance object
6)  examine the writes to disk, get the average and peaks, and you can also look at latency #s too

Now - when you say mirrored, are you sure it is being mirrored and not replicated via asynchronous replication?   I'm not certain how netapp does this because it is not SCSI based like iSCSI or FC storage, but I'm thinking it must have a way of only replicating changes to the data blocks as opposed to replicating full files...  but even still, any time you deploy a new VM on one of these replicated volumes the entire VM will get replicated the first time you lay it down on the disk and this can be expensive.    

 
0
 
Robin CMSenior Security and Infrastructure EngineerCommented:
What VMware guy said. You can also get live stats out of vSphere client, but it seems to somewhat hammer the vCenter server.
Another way to get live raw numbers is to run esxtop on the service console of the ESX server, see http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1008205 which is entitled "using esxtop to identify storage performance issues".
0
 
wahlsterAuthor Commented:
Thanks for your replies.  We are using Virtual Center.  As far as using deduplication on the Netapp I'm not sure but we are not compressing the snapshots.  LUNS are getting mirrored not replicated to our disaster site.  I looked at the Disk charts as suggested by VMwareGuy but nothing stands out.  We are going to give NetApp support a call to see if they can find something.
0
 
Paul SolovyovskySenior IT AdvisorCommented:
Connect to the filer using netapp system manager. If you don't have it download from the now site. Check the volumes since lins can be setup for dedupe and thin provision. By replication I meant snapmirror
0
 
VMwareGuyCommented:
Look at Disk \ write requests.. that should stand out.  It is recorded in a basic summation, which is confusing, because you have to look at the sample period, which is 20 seconds by default.   you can also look at the write rate peaks, recorded in Kbps.  the peak will incidate the the max KBps writes taking place.  you can compare these #s to other VMs on the same storage to see if it is indeed what is causing the excessive data writes.
0
 
berkcomjonathanCommented:
If you have each VM's disk in a separate volume, from the filer you can do:

stats show -i 5 volume

That will give you a 5 second idea of volume statistics.  Then look for volumes that have high write_data (b/s). You can increase or decrease the interval (-i) to fit your needs.  I had a situation pretty similar to this.  I cloned a VM and failed to increase the memory on the clone.  It was swapping like a mofo, which meant heavy NFS writes to it's volume.
0
 
wahlsterAuthor Commented:
Sorry about the delay.  Issue was a result of disk write as a result of an upgrade of Symantec's Enterprise Vault.  
Thanks for your help!
Terry
0
 
wahlsterAuthor Commented:
I did not want to accept my answer as a solution.  
0
 
wahlsterAuthor Commented:
Unselect my post as an answer.
0
 
Paul SolovyovskySenior IT AdvisorCommented:
If you're running EV that makes sense and also depends whether you write to a server that uses the Netapp or directly to the Netapp (allowing it do the compression and dedupe).  Also ensure that you are archiving and before snapmirror takes place as well as enable compression on the snapmirror.

Since EV archives and puts the data in as flat files your snapmirror shouldn't contain much more data then delta of your archive for the day unless you're ingesting your initial archive into the vault
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

  • 4
  • 4
  • 2
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now