Solved

Delete Duplicates In SharePoint List

Posted on 2014-01-24
2
1,160 Views
Last Modified: 2014-01-24
I have a list which has tens of thousands of records and I need them to be unique.  I found a PowerShell script online that will work if I only have 1 unique column.  However, I have to group by 2 columns and I can't figure out how to make it work.

cls
if((Get-PSSnapin | Where {$_.Name -eq "Microsoft.SharePoint.PowerShell"}) -eq $null) {
     Add-PSSnapin Microsoft.SharePoint.PowerShell;
}

$ListName = "DuplicateTest"

$web = Get-SPWeb -identity "http://MyVM/sites/TestSite"
$list = $web.Lists[$ListName]

$AllDuplicates = $list.Items.GetDataTable() | Group-Object Title | where {$_.count -gt 1}
$count = 1
$max = $AllDuplicates.Count
foreach($duplicate in $AllDuplicates) 
{ 
	$duplicate.group | Select-Object -Skip 1 | % {$list.Items.DeleteItemById($_.ID)} 
	Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
	$count++ 
} 

Open in new window


To clarify, if I have this data in a SP list, only if both columns are duplicates should the items be removed.
Title     Carrier
1          Carrier1
1          Carrier1
12        Carrier1
12        Carrier2
100      Carrier1
100      Carrier1
100      Carrier2
0
Comment
Question by:skipper68
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 38

Accepted Solution

by:
Justin Smith earned 500 total points
ID: 39806865
Group-Object title,carrier


Obviously test first.
0
 
LVL 9

Author Closing Comment

by:skipper68
ID: 39806908
Simple but effective and works perfectly.  Thanks ACH1LLES!
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A procedure for exporting installed hotfix details of remote computers using powershell
The following article is intended as a guide to using PowerShell as a more versatile and reliable form of application detection in SCCM.
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the adminiā€¦
This is my first video review of Microsoft Bookings, I will be doing a part two with a bit more information, but wanted to get this out to you folks.

717 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question