A PowerShell script to measure VM data change rates using Changed Block Tracking (CBT)

Sometimes you want to know how fast data on the virtual disks of your VM is changing. Estimates of data change rates are needed in many situations such like planning backup device capacity, or estimating network bandwidth to replicate disk for disaster recovery, or considering the use of deduplication in primary storage. All too often, I’ve had to do that kind of planning based on blind guesses. But now I’ve found a fairly easy way to get real measurements of data change rates using a Powershell script that exploits the VMware “Changed Block Tracking”, or CBT, feature of VMware ESX since version 4.1.

Edited 17-May-2016 to correct a serious error in the original script. Earlier version of the script gave incorrect (low) results if more there were more than 2000 disk segments changed.

VMware CBT was developed to enable differential and incremental backups of virtual disks and is used by most backup tools that use the VMware API for Data Protection (VADP). If you are backing up VMs using a modern backup tool designed to work with VMWare without backup agents in your guest VMs, then CBT is probably already enabled on all your VMs and you can use that feature to measure disk change rates without any effect on your backups. If you are not using VADP for backups, then you may have to take an extra step to enable CBT before you can get your measurements.

Here is a high-level outline of the steps you need to take:

Verify CBT is enabled, or enable it for the VMs you need to measure
Run the PowerShell script provided in this article to establish a baseline
Run the PowerShell script again to measure changes since the baseline, as many times and at whatever interval you need to get useful measurements.

Before you run the script, you will need to edit it according to your environment, to set correct values for:

The path to a folder where data (baseline and data files) will be stored.
The default of C:\Temp may be good enough for you.
The hostname of the vCenter instance to connect to.
The default of ‘vcenter.domain.tld’ probably won’t work!
The path to VMWare PowerCLI on the computer where the script is run.
The default is "C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI”

You may also want remove the “Get-Credential” statement that prompts for a username and password for the vCenter connection. If you are using Windows integrated authentication, either interactively or for a scheduled task, then you can simply comment out the “$Creds = Get-Credential…” line of the script.

There are a few things you need to know about what happens when you run the script:

The script will get a list of all VMs in vCenter to measure
Every time the script is run, it creates a snapshot on every VM.
The snapshot is immediately deleted (removed).
When first run, it creates a file named “baseline.csv”
If the baseline file already exists, it creates or appends to a file named “data.csv”

Now you may be thinking “Whoa, I don’t want to create a snapshot on every VM in my environment!” Keep in mind that only one snapshot is created at a time, and each snapshot is immediately removed before moving onto the next VM. This makes the script a bit slow, but ensures it has minimal impact. I found it took an average of 12 seconds per VM, or about an hour for 300 VMs, in my environment. This does not affect the usefulness of the results because a separate timestamp is recorded for the baseline and each measurement for each VM.

Please keep in mind that I have run this script in only one environment so I can’t promise you it will work perfectly for you the first time. Furthermore, I modified the script to remove all details specific to my environment and those changes are untested. On the other hand, I have tried to document the script well with comments so that you can easily adapt to your own needs, even if you aren’t a PowerShell guru.

If you are not using a backup tool that has already enabled CBT for your VMs, or you are not sure, then you should read the following VMware KB articles.

And finally, here is the script.

#
                      # This scripts measures the amount of disk change on VMs each time it is run.
                      # It measures all VM virtual disks for which CBT has been enabled.
                      #
                      # The first time it is run, it creatse a file containing baseline data (CBT change IDs and times).
                      # Each subsequent run measures changes since the baseline was set.
                      # It supports multiple virtual disks per VM, but not addition of new virtual disks after the baseline is established.
                      # Note that every run creates a short-lived snapshot on every VM that has CBT enabed.
                      #
                      # Time to run will vary in each environment.  In my case, it took about 20 seconds per VM for the 1st run,
                      # and about half that long for subsequent runs.
                      #
                      # To reset the baseline, just delete (or move, or rename) the baseline file and re-run.
                      #
                      #
                      # # Inspired by the VMGuru.com blog posting of August 2011 -- "CBT Tracker Powershell Script - now with more zombie"
                      # # Re-imagined completely by Carlo.G, February 2016
                      # # Corrected to call QueryChangedDiskArea repeatedly, May 2016
                      #
                      
                      
                      #
                      # Important literals used in this script
                      #
                      $Folder = 'C:\Temp'
                      $Basefile = 'Baselines.csv'
                      $Datafile = 'Data.csv'
                      $vCenter = 'vcenter.domain.tld'
                      $DTformat = 'yyyy-MM-dd HH:mm:ss' # Chosen to import correctly into Excel
                      $Creds = Get-Credential  'domain\userid' -Message 'Provide userid\password with permissions on VCenter'
                      #
                      # Initialize PowerCLI and connect to vCenter
                      #
                      Set-Location "C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts"
                      . .\Initialize-PowerCLIEnvironment.ps1 $true
                      Connect-VIServer $vcenter  -Credential $creds
                       
                      
                      #
                      # Get list of VMs with Change Block Tracking (CBT) enabled
                      #
                      $Vms = Get-VM | ?{$_.PowerState -eq 'PoweredOn'} | ?{(Get-View $_).Config.ChangeTrackingEnabled}
                      # Override with a list of specific VM names for testing
                      # $Vms = @('TEST_VM_1','TEST_VM_2','TEST_VM_3') | % {Get-VM $_}
                      
                      
                      #
                      # Exclude  VMs that are sensitive to snapshots
                      #
                      $Vms = $VMs | ?{$_.name -notmatch 'HATES_SNAPSHOTS' }
                       
                      #
                      # Retrieve or generate baseline change ids (one for each disks) for each VM in list to be measured
                      # Generating a new baseline requires creating a snapshot (which is removed immediately)
                      #
                      # Note that there may be more than one disk per VM.
                      # and in some cases only some disks have CBT enabled.
                      #
                      $n = $VMs.count
                      $i = 0
                      Write-Host "Getting baselines for $n VMs"
                      $baselines = @()
                      Try   {$baselines = Import-CSV (Join-Path $folder $basefile)} 
                      Catch {Write-Host -f cyan "No $(Join-Path $folder $basefile) file found, so new one will be created"  }
                      $more = $false
                      ForEach ($vm in $Vms) {
                          ++$i
                          $b = $baselines | ?{$vm.name -eq $_.VMname}  | select -first 1
                          If ($b -eq $null) {
                              Write-Host "Creating baseline for $($vm.name) ($i of $n)"
                              $more = $true
                              $TimeStamp = Get-Date -format $DTformat
                              $snapshot  = New-Snapshot -VM $vm -Name 'Temp for CBT baseline - Delete immediately ' -Description "for Carlo's change block tracking script, $TimeStamp"
                              $snapview  = Get-View $snapshot
                              $snapdisks = $snapview.Config.Hardware.Device | where {($_.GetType()).Name -eq "VirtualDisk"}   
                              ForEach ($d in $snapdisks) {
                                  [array]$baselines += New-Object 'PSObject' | select `
                                      @{N='VmName';E={$vm.name}},
                                      @{N='VmGBProv';E={$vm.ProvisionedSpaceGB}},
                                      @{N='VmGBUsed';E={$vm.UsedSpaceGB}},
                                      @{N='DiskSummary';E={$d.deviceinfo.summary}},
                                      @{N='DiskName';E={$d.deviceinfo.label}},
                                      @{N='DiskKey';E={$d.key}},
                                      @{N='TimeStamp';E={$TimeStamp}},
                                      @{N='ChangeId';E={$d.Backing.ChangeId}}
                                      }
                              Remove-Snapshot $snapshot -Confirm:$false  
                              }
                          } 
                      If ($more) {$Baselines | Export-CSV (Join-Path $folder $basefile) -NoTypeInformation -Force}
                      
                      
                      #
                      # For each item in Baselines list, measure changes since baseline was set
                      # A snapshot (which is removed immediately) is needed for each VM measured
                      # Append results to existing file (if it exists)
                      #
                      # Note that there may be more than one disk per VM.
                      # and in some cases only some disks have CBT enabled.
                      #
                      $lastVMname = $null
                      $snapshot = $null
                      $data = @()
                      $Baselines = $BaseLines | Sort VMname, DiskKey | ? {$_.ChangeID -ne '' -and $_.ChangeId -ne $null}
                      $n = $Baselines.count
                      $i = 0
                      ForEach ($b in $Baselines) {
                          ++$i
                          If ($b.VmName -ne $lastVMname) {
                              If ($snapshot -ne $null) {Remove-Snapshot $snapshot -Confirm:$false}
                              $LastVmName = $b.VmName
                              $vm = Get-VM $b.VmName
                              $vmview    = Get-View $vm
                              $TimeStamp = Get-Date   -format $DTformat
                              $Interval = (New-Timespan $b.TimeStamp $TimeStamp)
                              $snapshot  = New-Snapshot -VM $vm -Name 'Temp for CBT polling - Delete immediately ' -Description "for Carlo's change block tracking script, $TimeStamp"
                              $snapview  = Get-View $snapshot
                              $snapdisks = $snapview.Config.Hardware.Device | where {($_.GetType()).Name -eq "VirtualDisk"} | sort key
                              }  
                          $disk = $snapdisks | ?{$_.key -eq $b.DiskKey}
                          Try {
                              $Offset = 0
                              $GBChanged = 0
                              Do {
                                  $changes = $VmView.QueryChangedDiskAreas($SnapView.MoRef,$Disk.key,$Offset,$b.ChangeId)
                                  $GBchanged += ($changes.ChangedArea | % {$_.length} | measure-object -sum).sum/1024/1024/1024
                                  $LastChange = $changes.changedarea | Sort Start | select -last 1
                                  $Offset = $LastChange.start + $LastChange.Length
                                  }
                              While ($Disk.CapacityInBytes -gt $Offset -and $Changes.ChangeArea.Count -gt 0) 
                              }
                          Catch {$GBchanged = 'error'}
                          Write-Host "$($b.Vmname) $($b.DiskName) $GBchanged GB changed since $($b.TimeStamp)  ($i of $n)"
                          $data += New-Object 'PSObject' | select `
                                      @{N='VmName';E={$vm.name}},
                                      @{N='DiskName';E={$disk.deviceinfo.label}},
                                      @{N='DiskSummary';E={$disk.deviceinfo.summary}},
                                      @{N='BaseTime';E={$B.TimeStamp}},
                                      @{N='ThisTime';E={$TimeStamp}},
                                      @{N='Interval';E={$Interval}},
                                      @{N='GBChanged';E={$GBChanged}},
                                      @{N='VmGBProv';E={$vm.ProvisionedSpaceGB}},
                                      @{N='VmGBUsed';E={$vm.UsedSpaceGB}}
                          }
                      Remove-Snapshot $snapshot -Confirm:$false -EA 0
                      $Data | Export-CSV (Join-Path $folder $datafile) -NoTypeInformation -Append
                      
                                        
                      #
                      # Check for left-over CBT-related snapshots
                      #
                      Write-Host "Checking for leftover snapshots..."
                      $Snapshots = Get-VM  | Get-Snapshot | ?{$_.name -match '\sCBT\s'}
                      If ($snapshots) {
                          Write-Host -f red "$(Snapshots.count) left-over CBT snapshots????"
                          $Snapshots | ft vm,created,name
                          }

Open in new window

A PowerShell script to measure VM data change rates using Changed Block Tracking (CBT)

Comments (1)