[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Delete Duplicates In SharePoint List

Posted on 2014-01-24
2
Medium Priority
?
1,238 Views
Last Modified: 2014-01-24
I have a list which has tens of thousands of records and I need them to be unique.  I found a PowerShell script online that will work if I only have 1 unique column.  However, I have to group by 2 columns and I can't figure out how to make it work.

cls
if((Get-PSSnapin | Where {$_.Name -eq "Microsoft.SharePoint.PowerShell"}) -eq $null) {
     Add-PSSnapin Microsoft.SharePoint.PowerShell;
}

$ListName = "DuplicateTest"

$web = Get-SPWeb -identity "http://MyVM/sites/TestSite"
$list = $web.Lists[$ListName]

$AllDuplicates = $list.Items.GetDataTable() | Group-Object Title | where {$_.count -gt 1}
$count = 1
$max = $AllDuplicates.Count
foreach($duplicate in $AllDuplicates) 
{ 
	$duplicate.group | Select-Object -Skip 1 | % {$list.Items.DeleteItemById($_.ID)} 
	Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
	$count++ 
} 

Open in new window


To clarify, if I have this data in a SP list, only if both columns are duplicates should the items be removed.
Title     Carrier
1          Carrier1
1          Carrier1
12        Carrier1
12        Carrier2
100      Carrier1
100      Carrier1
100      Carrier2
0
Comment
Question by:skipper68
2 Comments
 
LVL 38

Accepted Solution

by:
Justin Smith earned 2000 total points
ID: 39806865
Group-Object title,carrier


Obviously test first.
0
 
LVL 9

Author Closing Comment

by:skipper68
ID: 39806908
Simple but effective and works perfectly.  Thanks ACH1LLES!
0

Featured Post

NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A recent project that involved parsing Tableau Desktop and Server log files to extract reusable user queries for use in other systems. I chose to use PowerShell to gather the data, and SharePoint to present it...
Measuring Server's processing rate with a simple powershell command. The differences in processing rate also was recorded in different use-cases, when a server in free and busy states.
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Screencast - Getting to Know the Pipeline

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question