Delete Duplicates In SharePoint List

I have a list which has tens of thousands of records and I need them to be unique.  I found a PowerShell script online that will work if I only have 1 unique column.  However, I have to group by 2 columns and I can't figure out how to make it work.

cls
if((Get-PSSnapin | Where {$_.Name -eq "Microsoft.SharePoint.PowerShell"}) -eq $null) {
     Add-PSSnapin Microsoft.SharePoint.PowerShell;
}

$ListName = "DuplicateTest"

$web = Get-SPWeb -identity "http://MyVM/sites/TestSite"
$list = $web.Lists[$ListName]

$AllDuplicates = $list.Items.GetDataTable() | Group-Object Title | where {$_.count -gt 1}
$count = 1
$max = $AllDuplicates.Count
foreach($duplicate in $AllDuplicates) 
{ 
	$duplicate.group | Select-Object -Skip 1 | % {$list.Items.DeleteItemById($_.ID)} 
	Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
	$count++ 
} 

Open in new window


To clarify, if I have this data in a SP list, only if both columns are duplicates should the items be removed.
Title     Carrier
1          Carrier1
1          Carrier1
12        Carrier1
12        Carrier2
100      Carrier1
100      Carrier1
100      Carrier2
LVL 9
DougApplication Development ManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Justin SmithSr. System EngineerCommented:
Group-Object title,carrier


Obviously test first.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
DougApplication Development ManagerAuthor Commented:
Simple but effective and works perfectly.  Thanks ACH1LLES!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SharePoint

From novice to tech pro — start learning today.