Solved

Delete Duplicates In SharePoint List

Posted on 2014-01-24
2
1,020 Views
Last Modified: 2014-01-24
I have a list which has tens of thousands of records and I need them to be unique.  I found a PowerShell script online that will work if I only have 1 unique column.  However, I have to group by 2 columns and I can't figure out how to make it work.

cls
if((Get-PSSnapin | Where {$_.Name -eq "Microsoft.SharePoint.PowerShell"}) -eq $null) {
     Add-PSSnapin Microsoft.SharePoint.PowerShell;
}

$ListName = "DuplicateTest"

$web = Get-SPWeb -identity "http://MyVM/sites/TestSite"
$list = $web.Lists[$ListName]

$AllDuplicates = $list.Items.GetDataTable() | Group-Object Title | where {$_.count -gt 1}
$count = 1
$max = $AllDuplicates.Count
foreach($duplicate in $AllDuplicates) 
{ 
	$duplicate.group | Select-Object -Skip 1 | % {$list.Items.DeleteItemById($_.ID)} 
	Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
	$count++ 
} 

Open in new window


To clarify, if I have this data in a SP list, only if both columns are duplicates should the items be removed.
Title     Carrier
1          Carrier1
1          Carrier1
12        Carrier1
12        Carrier2
100      Carrier1
100      Carrier1
100      Carrier2
0
Comment
Question by:skipper68
2 Comments
 
LVL 38

Accepted Solution

by:
Justin Smith earned 500 total points
ID: 39806865
Group-Object title,carrier


Obviously test first.
0
 
LVL 9

Author Closing Comment

by:skipper68
ID: 39806908
Simple but effective and works perfectly.  Thanks ACH1LLES!
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Join & Write a Comment

Hi all.   The other day I had to change the passwords for a bunch of users on the fly. Because they were so many, I decided to do it in an automated way and I would like to share it with you all.   If you are not doing it directly in a Domain Co…
I thought I'd write this up for anyone who has a request to create an anonymous whistle-blower-type submission form created using SharePoint 2010 (this would probably work the same for 2013). It's not 100% fool-proof but it's as close as you can get…
This video discusses moving either the default database or any database to a new volume.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now