Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

SQL Remove duplicate records based on two fields

Posted on 2014-11-09
5
Medium Priority
?
3,096 Views
Last Modified: 2014-11-14
Hi

I have a table that I am using for testing purposes. There are two columns called "ValA" and "Article". I
want to use these two together to create a primary key, but first want to delete any extra records
that are duplicates based on thecombination of these two columns. How do I do this
0
Comment
Question by:Murray Brown
5 Comments
 
LVL 66

Accepted Solution

by:
Jim Horn earned 668 total points
ID: 40431572
A couple of ways to pull this off, but below is the code to create a subquery to pick off all of the ValA / Article combinations that have more than one row, then the main query deletes everything in the subquery.
DELETE 
FROM YourTable yt
  JOIN (
      SELECT ValA, Article
      FROM YourTable
      GROUP BY ValA, Article
      HAVING COUNT(Article) > 1) dups ON yt.ValA = dups.ValA AND yt.Article = dups.Article

Open in new window

0
 
LVL 15

Assisted Solution

by:Haris Djulic
Haris Djulic earned 668 total points
ID: 40431580
This is the code that will find you the duplicates :

select rtrim(ValA)+rtrim(Article),count( *)
from your_table_name
group by rtrim(ValA)+rtrim(Article)
having count(*)>1

Open in new window


and I pressume you have some kind of additional ID field in your table you can use it to delete those with minimum IDs..

delete from your_table_name
where YOUR_ID_FIELD in (
  select min(YOUR_ID_FIELD ) from your_table_name where rtrim(ValA)+rtrim(Article) in  
  (select value from (
    
    select rtrim(ValA)+rtrim(Article) value,count( *) num
from your_table_name
group by rtrim(ValA)+rtrim(Article)
having count(*)>1)a
    )
  )

Open in new window

0
 
LVL 25

Assisted Solution

by:chaau
chaau earned 664 total points
ID: 40431628
This code will remove duplicates leaving at least one record from the duplicate records intact:
with cte as (
select *, ROW_NUMBER() OVER (PARTITION BY ValA, Article) AS rn
from yourTable)
DELETE FROM cte WHERE rn > 1

Open in new window

If you have a column that you can use as a sort order to determine which record is more preferred to be kept than you can use it. E.g. imagine you have a dateAdded column that tells when the record was added. If you wish to keep the most recent records you would use this query:
with cte as (
select *, ROW_NUMBER() OVER (PARTITION BY ValA, Article ORDER BY dateAdded DESC) AS rn
from yourTable)
DELETE FROM cte WHERE rn > 1

Open in new window

0
 

Author Closing Comment

by:Murray Brown
ID: 40442665
Thank you all
0
 
LVL 66

Expert Comment

by:Jim Horn
ID: 40442690
Thanks for the split.  Good luck with your project.  -Jim
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
When trying to connect from SSMS v17.x to a SQL Server Integration Services 2016 instance or previous version, you get the error “Connecting to the Integration Services service on the computer failed with the following error: 'The specified service …
Familiarize people with the process of utilizing SQL Server functions from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Microsoft Ac…
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

824 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question