Link to home
Start Free TrialLog in
Avatar of Doug
DougFlag for United States of America

asked on

Delete Duplicates In SharePoint List

I have a list which has tens of thousands of records and I need them to be unique.  I found a PowerShell script online that will work if I only have 1 unique column.  However, I have to group by 2 columns and I can't figure out how to make it work.

cls
if((Get-PSSnapin | Where {$_.Name -eq "Microsoft.SharePoint.PowerShell"}) -eq $null) {
     Add-PSSnapin Microsoft.SharePoint.PowerShell;
}

$ListName = "DuplicateTest"

$web = Get-SPWeb -identity "http://MyVM/sites/TestSite"
$list = $web.Lists[$ListName]

$AllDuplicates = $list.Items.GetDataTable() | Group-Object Title | where {$_.count -gt 1}
$count = 1
$max = $AllDuplicates.Count
foreach($duplicate in $AllDuplicates) 
{ 
	$duplicate.group | Select-Object -Skip 1 | % {$list.Items.DeleteItemById($_.ID)} 
	Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
	$count++ 
} 

Open in new window


To clarify, if I have this data in a SP list, only if both columns are duplicates should the items be removed.
Title     Carrier
1          Carrier1
1          Carrier1
12        Carrier1
12        Carrier2
100      Carrier1
100      Carrier1
100      Carrier2
ASKER CERTIFIED SOLUTION
Avatar of Justin Smith
Justin Smith
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Doug

ASKER

Simple but effective and works perfectly.  Thanks ACH1LLES!