We help IT Professionals succeed at work.

Removing multiple and newest duplicate items in Calendar

594 Views
Last Modified: 2017-04-06
Hi Guys,

Currently the script I found can remove 1 newest duplicate items in Calendar and I wish to remove multiple instead of 1. Meaning if I got 3 duplicates, I want to remove the newest 2. Where can I edit so that it can remove multiple items?

SCRIPT
<#
    .SYNOPSIS
    Remove-DuplicateItems
   
    Michel de Rooij
    michel@eightwone.com
      
    THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE
    ENTIRE RISK OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS
    WITH THE USER.
      
    Version 1.63, February 17th, 2017
   
    .DESCRIPTION
    This script will scan each folder of a given primary mailbox and personal archive (when
    configured, Exchange 2010 and later) and removes duplicate items per folder. You can specify
    how the items should be deleted and what items to process, e.g. mail items or appointments.
    Sample scenarios are misbehaving 3rd party synchronization tool creates duplicate items or
    (accidental) import of PST file with duplicate items. Script will process
    mailbox and archive if configured, unless MailboxOnly or ArchiveOnly is specified. For
    Exchange 2007, you need to specify -MailboxOnly.
     
    Note that usage of the Verbose, Confirm and WhatIf parameters is supported.
    When using Confirm, you will be prompted per batch.
      
    .LINK
    http://eightwone.com
   
    .NOTES
    Microsoft Exchange Web Services (EWS) Managed API 1.2 or up is required.

    Revision History
    --------------------------------------------------------------------------------
    1.0     Initial release
    1.1     Fixed issue with PowerShell v3 (System.Collections.Generic.List`1)
            Specified mailbox will also match object using mail attribute
    1.2     Added requested Retain option (default Newest, was undetermined)
    1.21    Switched Retain from using DateTimeReceived to LastModifiedTime
    1.3     Changed parameter Mailbox, you can now use an e-mail address as well
            Added parameter Credentials
            Added item class and size for certain  duplication checks
            Changed item removal process. Remove items after, not while processing
            folder. Avoids asynchronous deletion issues.
            Works against Office 365
    1.4     Added personal archive support
    1.41    Fixed typo preventing script from working on Ex2007
    1.5     Prevents sending cancellation notices when removing calendar items
    1.6     Added IncludeFolder parameter
            Added ExcludeFolder parameter
            Added MD5 hashing of keys to lower memory usage
            Added MailboxWide switch (CAUTION)
    1.61    Fixed impersonation logic issue
    1.62    Fixed using 2+ Exclude folders
    1.63    Identity parameter replaces Mailbox
            Made "can't access information store" more verbose.
            Fixed bug in non-wildcard matching

    .PARAMETER Identity
    Identity of the Mailbox. Can be CN/SAMAccountName (for on-premises) or e-mail format (on-prem & Office 365)
 
    .PARAMETER Server
    Exchange Client Access Server to use for Exchange Web Services. When ommited,
    script will attempt to use Autodiscover.

    .PARAMETER Credentials
    Specify credentials to use. When not specified, current credentials are used.
    Credentials can be set using $Credentials= Get-Credential
       
    .PARAMETER Impersonation
    When specified, uses impersonation for mailbox access, otherwise current
    logged on user is used. For details on how to configure impersonation
    access for Exchange 2010 using RBAC, see this article:
    http://msdn.microsoft.com/en-us/library/exchange/bb204095(v=exchg.140).aspx
    For details on how to configure impersonation for Exchange 2007, see KB article:
    http://msdn.microsoft.com/en-us/library/exchange/bb204095%28v=exchg.80%29.aspx
   
    .PARAMETER Retain
    Determines which items are not discarded (based on Last Modification Time):
    - Oldest:             Oldest received item is kept, newest item(s) are deleted
    - Newest:             Newest received item is kept, oldest item(s) are deleted (default)

    .PARAMETER DeleteMode
    Determines how to remove messages. Options are:
    - HardDelete:         Items will be permanently deleted.
    - SoftDelete:         Items will be moved to the dumpster (default).
    - MoveToDeletedItems: Items will be moved to the Deleted Items folder.

    When using MoveToDeletedItems, the Deleted Items folder will not be processed.
   
    .PARAMETER Type
    Determines what kind of folders to check for duplicates.
    Options: Mail, Calendar, Contacts, Tasks, Notes or All (Default).

    .PARAMETER Mode
    Determines how items are matched. Options are:
    - Quick: Removes duplicate items with matching PidTagSearchKey
    attribute; This is the default mode.
    - Full: Removes duplicate items with predefined matching criteria,
    depending on item class:
    - Contacts: File As, First Name, Last Name, Company Name,
    Business Phone, Mobile Phone, Home Phone, Size
    - Distribution List: FileAs, Number of Members, Size
    - Calendar: Subject, Location, Start & End Date, Size
    - Task: Subject, Start Date, Due Date, Status, Size
    - Note: Contents, Color, Size
    - Mail: Subject, Internet Message ID, DateTimeSent,
    DateTimeReceived, Sender, Size
    - Other: Subject, DateTimeReceived, Size

    Note that when Quick mode is used and PidTagSearchKey is missing or
    inaccessible, search will fall back to Full mode. For more info on
    PidTagSearchKey: http://msdn.microsoft.com/en-us/library/cc815908.aspx

    .PARAMETER MailboxOnly
    Only process primary mailbox of specified users. You als need to use this parameter when
    running against mailboxes on Exchange Server 2007.

    .PARAMETER ArchiveOnly
    Only process personal archives of specified users.

    .PARAMETER IncludeFolder
    Specify one or more names of folder(s) to include, e.g. 'Projects'. You can use wildcards
    around or at the end to include folders containing or starting with this string, e.g.
    'Projects*' or '*Project*'. Matching is always case-insensitive. You can also well-known
    folders, by using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are
    Calendar, Contacts, Inbox, Notes, SentItems, and Tasks.

    .PARAMETER ExcludeFolder
    Specify one or more folder(s) to exclude. Usage of wildcards and well-known folders
    identical to IncludeFolder.

    .PARAMETER Force
    Force removal of items without prompting.

    .PARAMETER MailboxWide
    Performs duplicate cleanup against whole mailbox, instead of per folder.
    Caution: The first unique item encountered will be retained. Since there is no way to order mailbox folders,
    when an item is found in Folder A and in Folder B, it is undetermined which item will be kept.

    .EXAMPLE
    .\Remove-DuplicateItems.ps1 -Mailbox Francis -Type All -Impersonation -DeleteMode SoftDelete -Mode Quick -Verbose

    Check Francis' mailbox for duplicate items in each folder, soft deleting
    duplicates, matching on PidTagSearchKey and using impersonation.

    .EXAMPLE
    .\Remove-DuplicateItems.ps1 -Mailbox Philip -Retain Oldest -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

    Check Philip's mailbox for duplicate task items in each folder and moves
    duplicates to the Deleted Items folder, using preset matching criteria
    and impersonation. When duplicates are found, the oldest is retained.

    .EXAMPLE
    $Credentials= Get-Credential
    .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Credentials $Credentials

    Sets $Credentials variable. Then, check olrik@office365tenant.com's mailbox for duplicate items in each folder, using
    Credentials provided earlier.

    .EXAMPLE
    $Credentials= Get-Credential
    .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '*In*','Archive*' -ExcludeFolders '#Inbox#'

    Remove duplicate items from specified mailbox in Office365 using fixed FQDN - bypassing AutoDiscover, limiting operation against folders
    containing 'In' or starting with 'Archive', but don't remove duplicates from the Well-Known Folder 'Inbox'.
#>

[cmdletbinding(
    SupportsShouldProcess=$true,
    ConfirmImpact="High"
)]
param(
  [parameter( Position=0, Mandatory=$true, ValueFromPipelineByPropertyName=$true, ParameterSetName="All")]
  [alias('Mailbox')]
  [string]$Identity,
  [parameter( Position=1, Mandatory=$false, ParameterSetName="All")]
  [ValidateSet("Mail","Calendar","Contacts","Tasks","Notes","Groups","All")]
  [string]$Type="All",
  [parameter( Mandatory=$false, ParameterSetName="All")]
  [ValidateSet("Oldest", "Newest")]
  [string]$Retain="Newest",
  [parameter( Mandatory=$false, ParameterSetName="All")]
  [string]$Server,
  [parameter( Mandatory=$false, ParameterSetName="All")]
  [switch]$Impersonation,
  [parameter( Mandatory=$false, ParameterSetName="All")]
  [ValidateSet("HardDelete","SoftDelete","MoveToDeletedItems")]
  [string]$DeleteMode='SoftDelete',
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [System.Management.Automation.PsCredential]$Credentials,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [ValidateSet("Quick","Full")]
  [string]$Mode='Quick',
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [parameter( Mandatory= $false, ParameterSetName="MailboxOnly")]
  [switch]$MailboxOnly,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [parameter( Mandatory= $false, ParameterSetName="ArchiveOnly")]
  [switch]$ArchiveOnly,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [string[]$IncludeFolders,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [string[]$ExcludeFolders,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [switch]$MailboxWide,
  [parameter( Mandatory= $false, ParameterSetName="All")]
  [switch]$Force
)

process {

  # Process folders these batches
  $MaxFolderBatchSize= 100
  # Process items in these page sizes
  $MaxItemBatchSize= 100
  # Max of concurrent item deletes
  $MaxDeleteBatchSize= 100
   
  # Errors
  $ERR_EWSDLLNOTFOUND                      = 1000
  $ERR_EWSLOADING                          = 1001
  $ERR_MAILBOXNOTFOUND                     = 1002
  $ERR_AUTODISCOVERFAILED                  = 1003
  $ERR_CANTACCESSMAILBOXSTORE              = 1004
  $ERR_PROCESSINGMAILBOX                   = 1005
  $ERR_PROCESSINGARCHIVE                   = 1006
  $ERR_INVALIDCREDENTIALS                  = 1007
   
  Function Get-EmailAddress( $Identity) {
    $address= [regex]::Match([string]$Identity, ".*@.*\..*", "IgnoreCase")
    if( $address.Success ) {
      return $address.value.ToString()
    }
    Else {
      # Use local AD to look up e-mail address using $Identity as SamAccountName
      $ADSearch= New-Object DirectoryServices.DirectorySearcher( [ADSI]"")
      $ADSearch.Filter= "(|(cn=$Identity)(samAccountName=$Identity)(mail=$Identity))"
      $Result= $ADSearch.FindOne()
      If( $Result) {
        $objUser= $Result.getDirectoryEntry()
        return $objUser.mail.toString()
      }
      else {
        return $null
      }
    }
  }

  Function Load-EWSManagedAPIDLL {
    $EWSDLL= "Microsoft.Exchange.WebServices.dll"
    If( Test-Path "$pwd\$EWSDLL") {
      $EWSDLLPath= "$pwd"
    }
    Else {
      $EWSDLLPath = (($(Get-ItemProperty -ErrorAction SilentlyContinue -Path Registry::$(Get-ChildItem -ErrorAction SilentlyContinue -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Web Services'|Sort-Object Name -Descending| Select-Object -First 1 -ExpandProperty Name)).'Install Directory'))
      if (!( Test-Path "$EWSDLLPath\$EWSDLL")) {
        Write-Error "This script requires EWS Managed API 1.2 or later to be installed, or the Microsoft.Exchange.WebServices.DLL in the current folder."
        Write-Error "You can download and install EWS Managed API from http://go.microsoft.com/fwlink/?LinkId=255472"
        Exit $ERR_EWSDLLNOTFOUND
      }
    }

    Write-Verbose "Loading $EWSDLLPath\$EWSDLL"
    try {
      # EX2010
      If(!( Get-Module Microsoft.Exchange.WebServices)) {
        Import-Module "$EWSDLLPATH\$EWSDLL"
      }
    }
    catch {
      #<= EX2010
      [void][Reflection.Assembly]::LoadFile( "$EWSDLLPath\$EWSDLL")
    }
    try {
      $Temp= [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1
    }
    catch {
      Write-Error "Problem loading $EWSDLL"
      Exit $ERR_EWSLOADING
    }
       
  }

  # After calling this any SSL Warning issues caused by Self Signed Certificates will be ignored
  # Source: http://poshcode.org/624
  Function set-TrustAllWeb() {
    Write-Verbose "Set to trust all certificates"
    $Provider=New-Object Microsoft.CSharp.CSharpCodeProvider  
    $Compiler=$Provider.CreateCompiler()  
    $Params=New-Object System.CodeDom.Compiler.CompilerParameters  
    $Params.GenerateExecutable=$False  
    $Params.GenerateInMemory=$True  
    $Params.IncludeDebugInformation=$False  
    $Params.ReferencedAssemblies.Add("System.DLL") | Out-Null  
 
    $TASource= @'
            namespace Local.ToolkitExtensions.Net.CertificatePolicy {
                public class TrustAll : System.Net.ICertificatePolicy {
                    public TrustAll() {  
                    }
                    public bool CheckValidationResult(System.Net.ServicePoint sp, System.Security.Cryptography.X509Certificates.X509Certificate cert,   System.Net.WebRequest req, int problem) {
                        return true;
                    }
                }
            }
'@

    $TAResults=$Provider.CompileAssemblyFromSource($Params, $TASource)  
    $TAAssembly=$TAResults.CompiledAssembly  
    $TrustAll=$TAAssembly.CreateInstance("Local.ToolkitExtensions.Net.CertificatePolicy.TrustAll")  
    [System.Net.ServicePointManager]::CertificatePolicy=$TrustAll  
  }
   
  Function Construct-SearchFilter {
    param(
      [string[]$Folders,
      [bool]$Negate
    )
    $FolderSearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection( ( [Microsoft.Exchange.WebServices.Data.LogicalOperator]::Or))
    ForEach( $Folder in $Folders) {
      $FolderName= Search-ReplaceWellKnownFolderNames $Folder
      If($FolderName -match '^\*(?<substring>.*?)\*$') {
        $SearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+ContainsSubstring([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,
        $matches['substring'], [Microsoft.Exchange.WebServices.Data.ContainmentMode]::Substring, [Microsoft.Exchange.WebServices.Data.ComparisonMode]::IgnoreCase)
      }
      Else {
        If($FolderName -match '^(?<prefix>.*?)\*$') {
          $SearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+ContainsSubstring([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,
          $matches['prefix'], [Microsoft.Exchange.WebServices.Data.ContainmentMode]::Prefixed, [Microsoft.Exchange.WebServices.Data.ComparisonMode]::IgnoreCase)
        }
        Else {
          $SearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+ContainsSubstring([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,
          $FolderName, [Microsoft.Exchange.WebServices.Data.ComparisonMode]::IgnoreCase)
        }
      }
      Write-Debug ('Adding filter: {0} {1} in {2} (Negate:{3})' -f $SearchFilter.ContainmentMode, $SearchFilter.Value, $SearchFilter.PropertyDefinition, $Negate)
      $FolderSearchFilter.Add( $SearchFilter)
    }
    If( $Negate) {
      $FolderSearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+Not( $FolderSearchFilter)
    }
    Return $FolderSearchFilter
  }

  Function Search-ReplaceWellKnownFolderNames {
    param(
      [string]$criteria=''
    )
    $criteria= $criteria -replace '#Inbox#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::Inbox
    $criteria= $criteria -replace '#Calendar#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::Calendar
    $criteria= $criteria -replace '#Contacts#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::Contacts
    $criteria= $criteria -replace '#Notes#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::Notes
    $criteria= $criteria -replace '#SentItems#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::SentItems
    $criteria= $criteria -replace '#Tasks#', [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::Tasks
    return $criteria
  }

  Function Get-Hash {
    param(
      [string]$string
    )
    $md5 = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
    $data = ([system.Text.Encoding]::UTF8).GetBytes( $string)
    return ([System.BitConverter]::ToString( $md5.ComputeHash( $data)) -replace '-', '')
  }

  Function Process-Mailbox {
    param(
      $Folder,
      $IncludeSearchCollection,
      $ExcludeSearchCollection
    )

    $ProcessingOK= $True
    $ThisMailboxMode= $Mode
    $temp= $null
    $GrandTotal= 0
    $TotalRemoved= 0
    $DeletedItemsFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::DeletedItems)
    $PidTagSearchKey = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x300B, [Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)  

    $FolderView= New-Object Microsoft.Exchange.WebServices.Data.FolderView( $MaxFolderBatchSize)
    $FolderView.Traversal= [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Deep
    $FolderView.PropertySet= New-Object Microsoft.Exchange.WebServices.Data.PropertySet(
      [Microsoft.Exchange.WebServices.Data.BasePropertySet]::IdOnly,
      [Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,
    [Microsoft.Exchange.WebServices.Data.FolderSchema]::FolderClass)
    $FolderSearchCollection= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection( [Microsoft.Exchange.WebServices.Data.LogicalOperator]::And)
    If( $Type -ne "All") {
      Write-Verbose "Searching for folder class $FolderSearchClass"
      $FolderSearchClass= (@{"Mail"="IPF.Note";"Calendar"="IPF.Appointment";"Contacts"="IPF.Contact";"Tasks"="IPF.Task";"Notes"="IPF.StickyNotes"})[$Type]
      $FolderSearchFilter= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo( [Microsoft.Exchange.WebServices.Data.FolderSchema]::FolderClass, $FolderSearchClass)
      $FolderSearchCollection.Add( $FolderSearchFilter)
    }
    If( $IncludeSearchCollection) {
      $FolderSearchCollection.Add( $IncludeSearchCollection)
    }
    If( $ExcludeSearchCollection) {
      $FolderSearchCollection.Add( $ExcludeSearchCollection)
    }
    Do {
      If( $FolderSearchCollection.Count -ge 1) {
        $FolderSearchResults= $EwsService.FindFolders( $Folder.Id, $FolderSearchCollection, $FolderView)
      }
      Else {
        $FolderSearchResults= $EwsService.FindFolders( $Folder.Id, $FolderView)
      }

      # MailboxWide, track global MD5 hashes
      If( $MailboxWide) {
        $UniqueList= [System.Collections.ArrayList]@()
      }
      ForEach( $SubFolder in $FolderSearchResults) {
        If( ! ( $DeleteMode -eq "MoveToDeletedItems" -and $SubFolder.Id -eq $DeletedItemsFolder.Id)) {
          Write-Verbose "Processing folder $($SubFolder.DisplayName)"
          $ItemView= New-Object Microsoft.Exchange.WebServices.Data.ItemView( $MaxItemBatchSize)
          #$ItemView.Traversal= [Microsoft.Exchange.WebServices.Data.ItemTraversal]::Shallow
          If( $Retain -eq "Oldest") {
            $ItemView.OrderBy.Add( [Microsoft.Exchange.WebServices.Data.ItemSchema]::LastModifiedTime, [Microsoft.Exchange.WebServices.Data.SortDirection]::Ascending)
          }
          Else {
            $ItemView.OrderBy.Add( [Microsoft.Exchange.WebServices.Data.ItemSchema]::LastModifiedTime, [Microsoft.Exchange.WebServices.Data.SortDirection]::Descending)
          }
          $ItemView.PropertySet= New-Object Microsoft.Exchange.WebServices.Data.PropertySet( [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)
          $ItemView.PropertySet.Add( $PidTagSearchKey)

          # Not MailboxWide (per folder), track MD5 hashes per folder
          If( -not $MailboxWide) {
            $UniqueList= [System.Collections.ArrayList]@()
          }
          $DuplicateList= [System.Collections.ArrayList]@()
          If( $psversiontable.psversion.major -lt 3) {
            $ItemIds= [activator]::createinstance(([type]'System.Collections.Generic.List`1').makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
          }
          Else {
            $type=("System.Collections.Generic.List"+'`'+"1")-as"Type"
            $type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
            $ItemIds = [Activator]::CreateInstance($type)
          }
          Do {
            $SendCancellationsMode= $null
            $AffectedTaskOccurrences= [Microsoft.Exchange.WebServices.Data.AffectedTaskOccurrence]::AllOccurrences
            $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, $ItemView)
            Write-Debug "Checking $($ItemSearchResults.Items.Count) items in $($SubFolder.DisplayName)"
            If( $ItemSearchResults.Items.Count -gt 0) {
              ForEach( $Item in $ItemSearchResults.Items) {
                Write-Debug "Inspecting item $($Item.Subject) of $($Item.DateTimeReceived), modified $($Item.LastModifiedTime)"
                Write-Debug "ItemView Offset:$($ItemView.Offset) FoundTotal:$($ItemSearchResults.TotalCount) FoundBatch:$($ItemSearchResults.Items.Count) UniqueSet:$($UniqueList.Count)"
                $GrandTotal++
                if ($ThisMailboxMode -eq "Quick") {
                  # Use PidTagSearchKey for matching duplicates
                  $PropVal =  $null
                  if( $Item.TryGetProperty( $PidTagSearchKey, [ref]$PropVal)){
                    $key= [System.BitConverter]::ToString($PropVal).Replace("-","")
                  }
                  Else {
                    Write-Debug "Can't access or missing PidTagSearchKey property, falling back to property mode"
                    $ThisMailboxMode= "Full"
                  }
                }
                If( $ThisMailboxMode -eq "Full") {
                  # Use predefined criteria for matching duplicates depending on ItemClass
                  $key= $Item.ItemClass
                  switch ($Item.ItemClass) {
                    "IPM.Note" {
                      if ($Item.DateTimeReceived)     { $key+= $Item.DateTimeReceived.ToString()}
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.InternetMessageId)    { $key+= $Item.InternetMessageId}
                      if ($Item.DateTimeSent)         { $key+= $Item.DateTimeSent.ToString()}
                      if ($Item.Sender)               { $key+= $Item.Sender}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                    "IPM.Appointment" {
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.Location)             { $key+= $Item.Location}
                      if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      if ($Item.End)                  { $key+= $Item.End.ToString()}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                    "IPM.Contact" {
                      if ($Item.FileAs)               { $key+= $Item.FileAs}
                      if ($Item.GivenName)            { $key+= $Item.GivenName}
                      if ($Item.Surname)              { $key+= $Item.Surname}
                      if ($Item.CompanyName)          { $key+= $Item.CompanyName}
                      if ($Item.PhoneNUmbers.TryGetValue("BusinessPhone", [ref]$temp))          { $key+= $temp}
                      if ($Item.PhoneNUmbers.TryGetValue("HomePhone", [ref]$temp))          { $key+= $temp}
                      if ($Item.PhoneNUmbers.TryGetValue("MobilePhone", [ref]$temp))          { $key+= $temp}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                    "IPM.DistList" {
                      if ($Item.FileAs)               { $key+= $Item.FileAs}
                      if ($Item.Members)              { $key+= $Item.Members.Count.ToString()}
                    }
                    "IPM.Task" {
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.StartDate)            { $key+= $Item.StartDate.ToString()}
                      if ($Item.DueDate)              { $key+= $Item.DueDate.ToString()}
                      if ($Item.Status)               { $key+= $Item.Status}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                    "IPM.Post" {
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                    Default {
                      if ($Item.DateTimeReceived)     { $key+= $Item.DateTimeReceived.ToString()}
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }
                  }
                }
                If( $key -ne $null) {
                  $hash= Get-Hash $key
                  If( $UniqueList.contains( $hash)){
                    Write-Debug "Duplicate: $hash ($key)"
                    $tmp= $DuplicateList.Add( $Item.Id)
                  }
                  Else {
                    Write-Debug "Unique: $($Item.id), $hash ($key)"
                    $tmp= $UniqueList.Add( $hash)
                  }
                }
                Else {
                  # Couldn't determine key, skip
                }
              }
              $ItemView.Offset+= $ItemSearchResults.Items.Count
            }
            Else {
              # No items found
                           
            }
          } While( $ItemSearchResults.MoreAvailable -and $ProcessingOK)
        }
        Else {
          Write-Debug "Skipping DeletedItems folder"
        }
        If( ($DuplicateList.Count -gt 0) -and ($Force -or $PSCmdlet.ShouldProcess( "Remove $($DuplicateList.Count) items from $($SubFolder.DisplayName)"))) {
          try {
            Write-Verbose "Removing $($DuplicateList.Count) items from $($SubFolder.DisplayName)"

            $SendCancellationsMode= [Microsoft.Exchange.WebServices.Data.SendCancellationsMode]::SendToNone
            $AffectedTaskOccurrences= [Microsoft.Exchange.WebServices.Data.AffectedTaskOccurrence]::SpecifiedOccurrenceOnly

            # Remove ItemIDs in batches
            ForEach( $ItemID in $DuplicateList) {
              $ItemIds.Add( $ItemID)
              If( $ItemIds.Count -eq $MaxDeleteBatchSize) {
                $res= $EwsService.DeleteItems( $ItemIds,
                [Microsoft.Exchange.WebServices.Data.DeleteMode]::$DeleteMode, $SendCancellationsMode, $AffectedTaskOccurrences)
                $ItemIds.Clear()
              }
            }
            # .. also remove last ItemIDs
            If( $ItemIds.Count -gt 0) {
              $res= $EwsService.DeleteItems( $ItemIds,
              [Microsoft.Exchange.WebServices.Data.DeleteMode]::$DeleteMode, $SendCancellationsMode, $AffectedTaskOccurrences)
              $ItemIds.Clear()
            }
            $TotalRemoved+= $DuplicateList.Count
          }
          catch{
            Write-Error "Problem removing items: $($error[0])"
            $ProcessingOK= $False
          }
        }
        Else {
          Write-Debug "No duplicates found in this folder"
        }
      } # ForEach SubFolder
      $FolderView.Offset+= $FolderSearchResults.Folders.Count
    } While ($FolderSearchResults.MoreAvailable)
    If( $ProcessingOK) {
      Write-Verbose ('Total number of items processed {0}, removed {1}' -f $GrandTotal, $TotalRemoved)
    }
    Return $ProcessingOK
  }

  ##################################################
  # Main
  ##################################################

  #Requires -Version 1.0

  Load-EWSManagedAPIDLL

  If( $Identity -is [array]) {
    # When multiple mailboxes are specified, call script for each mailbox
    [Void]$PSBoundParameters.Remove("Mailbox")
    $Identity | ForEach-Object { Remove-DuplicateItems -Identity $_ @PSBoundParameters }
  }
  else {
    $EmailAddress= get-EmailAddress $Identity
    If( !$EmailAddress) {
      Write-Error "Specified mailbox $Identity not found"
      Exit $ERR_MAILBOXNOTFOUND
    }
    Write-Host "Processing mailbox $Identity ($EmailAddress)"

    set-TrustAllWeb

    If( $MailboxOnly) {
      $ExchangeVersion= [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1
    }
    Else {
      $ExchangeVersion= [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP2
    }

    $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
    If( $Credentials) {
      try {
        Write-Verbose "Using credentials $($Credentials.UserName)"
        $EwsService.Credentials= New-Object System.Net.NetworkCredential( $Credentials.UserName, [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR( $Credentials.Password )))
      }
      catch {
        Write-Error "Invalid credentials provided " $error[0]
        Exit $ERR_INVALIDCREDENTIALS
      }
    }
    Else {
      $EwsService.UseDefaultCredentials= $true
    }

    Write-Verbose "DeleteMode is $DeleteMode"
    Write-Verbose "Processing $Type items"

    # Construct search filters
    Write-Verbose 'Constructing folder search filters'
    If( $IncludeFolders) {
      $FolderSearchFilter= Construct-SearchFilter $IncludeFolders $False
      $IncludeSearchCollection= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection( [Microsoft.Exchange.WebServices.Data.LogicalOperator]::Or, $FolderSearchFilter)
    }
    Else {
      $IncludeSearchCollection= $null
    }
    If( $ExcludeFolders) {
      $FolderSearchFilter= Construct-SearchFilter $ExcludeFolders $True
      $ExcludeSearchCollection= New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection( [Microsoft.Exchange.WebServices.Data.LogicalOperator]::Or, $FolderSearchFilter)
    }
    Else {
      $ExcludeSearchCollection= $null
    }

    If( $Impersonation) {
      Write-Verbose "Using $EmailAddress for impersonation"
      $EwsService.ImpersonatedUserId= New-Object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress, $EmailAddress)
    }
    Write-Host "Processing mailbox $Identity ($EmailAddress)"
       
    If ($Server) {
      $EwsUrl= "https://$Server/EWS/Exchange.asmx"
      Write-Verbose "Using Exchange Web Services URL $EwsUrl"
      $EwsService.Url= "$EwsUrl"
    }
    Else {
      Write-Verbose "Looking up EWS URL using Autodiscover for $EmailAddress"
      try {
        # Set script to terminate on all errors (autodiscover failure isn't) to make try/catch work
        $ErrorActionPreference= "Stop"
        $EwsService.autodiscoverUrl( $EmailAddress, {$true})
      }
      catch {
        Write-Error "Autodiscover failed: $($_.Exception.Message)"
        Exit $ERR_AUTODISCOVERFAILED
      }
      $ErrorActionPreference= "Continue"
      Write-Verbose "Using EWS on CAS $($EwsService.Url)"
    }
       
    If( -not $ArchiveOnly.IsPresent) {
      try {
        $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
        Write-Verbose "Processing primary mailbox $Identity"
        If(! ( Process-Mailbox $RootFolder $IncludeSearchCollection $ExcludeSearchCollection)) {
          Write-Error "Problem processing primary mailbox of $Identity ($EmailAddress)"
          Exit $ERR_PROCESSINGMAILBOX
        }
      }
      catch {
        Write-Error "Can't access mailbox information store: $($Error[0])"
        Exit $ERR_CANTACCESSMAILBOXSTORE
      }
    }

    If( -not $MailboxOnly.IsPresent) {
      try {
        $ArchiveRootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::ArchiveMsgFolderRoot)
        Write-Verbose "Processing archive mailbox $Identity"
        If(! ( Process-Mailbox $ArchiveRootFolder $IncludeSearchCollection $ExcludeSearchCollection)) {
          Write-Warning "Problem processing archive mailbox of $Identity ($EmailAddress)"
          Exit $ERR_PROCESSINGARCHIVE
        }
      }
      catch {
        Write-Debug "No archive configured or can't access archive"
      }
    }        
    Write-Verbose "Processing $Identity finished"
  }  
}  


THANKS IN ADVANCE!
Comment
Watch Question

Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
Set debug, then run the script.
$DebugPreference = 'continue'

Open in new window

The script does not limit itself to a single duplicate. By extension it must consider the items you're considering to be unique.

For appointment items it determines whether or not something is unique by analyzing Subject, Location, Start, End, and Size.

Author

Commented:
Thank you Chris Dent, I am currently looking at all the appointments.
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
If you find the debug trace is highlighting the items you're interested in removing as unique perhaps the value that differs can be identified.

The criteria the script uses can be changed by tweaking these lines:
                   "IPM.Appointment" {
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      if ($Item.Location)             { $key+= $Item.Location}
                      if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      if ($Item.End)                  { $key+= $Item.End.ToString()}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }

Open in new window

However, depending on what is breaking the comparison there's an element of risk in doing so.

It's a shame it's not written such that you can pick out the item searcher without rewriting large chunks, finding and deleting itself is a simple operation. That would have let you pick your own criteria without changing a thing.

Author

Commented:
Hi Chris Dent,

I realised that the first item was different from the 2 duplicate. below is the screenshot
Untitled.png
PowerShell Developer
CERTIFIED EXPERT
Top Expert 2010
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Author

Commented:
Nice! It helps. But I dont understand why the size differ as all the appointment is the same.

Author

Commented:
Hi Chris Dent,

I would like to know if I am able to remove selected duplicate entries based on subject filter?
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
Yes, it needs you to modify the keys being used.
                   "IPM.Appointment" {
                      if ($Item.Subject)              { $key+= $Item.Subject}
                      # if ($Item.Location)             { $key+= $Item.Location}
                      # if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      # if ($Item.End)                  { $key+= $Item.End.ToString()}
                      # if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }

Open in new window

Author

Commented:
Hi Chris,

How can I set?

"IPM.Appointment" {
                      if ($Item.Subject -eq 'This subject')              { $key+= $Item.Subject}
                      # if ($Item.Location)             { $key+= $Item.Location}
                      # if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      # if ($Item.End)                  { $key+= $Item.End.ToString()}
                      # if ($Item.Size)                 { $key+= $Item.Size.ToString()}
}

is it this way?
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
Essentially you're commenting out all of the criteria except subject for appointment items. At that point it only has subject to consider when evaluating duplicates.

Author

Commented:
I wont uncomment those when executing, I want to know if it is the way to compare the subject?
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
That's how you make it compare based on subject only. It will ignore location, start, end, and size (as they're never added to the comparison set, commented out).

Author

Commented:
"IPM.Appointment" {
                      if ($Item.Subject -eq 'This subject')              { $key+= $Item.Subject}
                      if ($Item.Location)             { $key+= $Item.Location}
                      if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      if ($Item.End)                  { $key+= $Item.End.ToString()}
                      # if ($Item.Size)                 { $key+= $Item.Size.ToString()}
}

if I based on the above code, Am I able to remove selected appointment?
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
Yes, I believe so. Run it with Whatif / Debug to verify, of course.

Author

Commented:
Ok. Thank very much Chris, will seek your help if there is still issue. Your help contributed a lot.

Author

Commented:
"IPM.Appointment" {
                      if ($Item.Subject -eq "Testing")              { $key+= $Item.Subject}
                      if ($Item.Location -eq "meeting room")             { $key+= $Item.Location}
                      if ($Item.Start)                { $key+= $Item.Start.ToString()}
                      if ($Item.End)                  { $key+= $Item.End.ToString()}
                      if ($Item.Size)                 { $key+= $Item.Size.ToString()}
                    }

Above code still remove all duplicate. instead of this appointment only.
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
Ohhhh sorry, you want to target just this one item... Apologies, I thought you wanted something a little more broad.

This goes back to the aspect of this script which bugs me. It's not written in a way you can trivially tease out detailed control like this. If I had written it I'd start with generic functions to search for things, get them, delete them, and so on. Anything doing duplicate detection would consume those utility functions, leaving you with things that could also do exactly what you ask.

Unfortunately there are no utility functions, so item targeting is not practical. It's very frustrating.

Author

Commented:
Is there anyway I can change the script to make it target specific items before deleting them?

Below is the code for removing duplicate right?

If( ($DuplicateList.Count -gt 0) -and ($Force -or $PSCmdlet.ShouldProcess( "Remove $($DuplicateList.Count) items from $($SubFolder.DisplayName)"))) {
          try {
            Write-Verbose "Removing $($DuplicateList.Count) items from $($SubFolder.DisplayName)"

            $SendCancellationsMode= [Microsoft.Exchange.WebServices.Data.SendCancellationsMode]::SendToNone
            $AffectedTaskOccurrences= [Microsoft.Exchange.WebServices.Data.AffectedTaskOccurrence]::SpecifiedOccurrenceOnly

            # Remove ItemIDs in batches
            ForEach( $ItemID in $DuplicateList) {
              $ItemIds.Add( $ItemID)
              If( $ItemIds.Count -eq $MaxDeleteBatchSize) {
                $res= $EwsService.DeleteItems( $ItemIds,
                [Microsoft.Exchange.WebServices.Data.DeleteMode]::$DeleteMode, $SendCancellationsMode, $AffectedTaskOccurrences)
                $ItemIds.Clear()
              }
            }
            # .. also remove last ItemIDs
            If( $ItemIds.Count -gt 0) {
              $res= $EwsService.DeleteItems( $ItemIds,
              [Microsoft.Exchange.WebServices.Data.DeleteMode]::$DeleteMode, $SendCancellationsMode, $AffectedTaskOccurrences)
              $ItemIds.Clear()
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
What you do mean by target specific items? As in you want to be able to provide things like a subject and decide an action?

If so, no, not really. The script needs completely redeveloping as a series of commands to grant that kind of flexibility.

Author

Commented:
Is there a way I can customize it to check for subject in this script? Basically I found another script that can search and delete based on the subject filter but it will delete all appointment including the original appointment. Hope you get what I mean.
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
In the broadest scope. It would need injecting after:
              ForEach( $Item in $ItemSearchResults.Items) {

Open in new window

For example:
              ForEach( $Item in $ItemSearchResults.Items) {
                  if ($Item.Subject -eq 'The subject') {    

Open in new window

The if statement needs to close, that replaces this section:
              $ItemView.Offset+= $ItemSearchResults.Items.Count
            }
            Else {
              # No items found

Open in new window

With:
              $ItemView.Offset+= $ItemSearchResults.Items.Count
              }
            }
            Else {
              # No items found

Open in new window

The only addition is a single } here.

Author

Commented:
oh man! you are my god. Hats off to you!

Author

Commented:
is there a way i can check for the resource email address instead of the name?

currently, $Item.Location is the name of the resource room.
Chris DentPowerShell Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
I'm not sure is the answer to that I'm afraid.

I don't have access to a copy of Exchange to fully explore the properties you get back so while I suspect the answer to be "no", I don't categorically know that to be so.

Author

Commented:
Because I realised that I had 2 similar subject with the same start and end time but different location. And this 2 had duplicates, meaning I have 4 items now (2 originals, 2 duplicates)

When I run this script, it detect as 3 duplicates.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.