Solved

SQL Remove duplicate records based on two fields

Posted on 2014-11-09
5
1,855 Views
Last Modified: 2014-11-14
Hi

I have a table that I am using for testing purposes. There are two columns called "ValA" and "Article". I
want to use these two together to create a primary key, but first want to delete any extra records
that are duplicates based on thecombination of these two columns. How do I do this
0
Comment
Question by:murbro
5 Comments
 
LVL 65

Accepted Solution

by:
Jim Horn earned 167 total points
ID: 40431572
A couple of ways to pull this off, but below is the code to create a subquery to pick off all of the ValA / Article combinations that have more than one row, then the main query deletes everything in the subquery.
DELETE 
FROM YourTable yt
  JOIN (
      SELECT ValA, Article
      FROM YourTable
      GROUP BY ValA, Article
      HAVING COUNT(Article) > 1) dups ON yt.ValA = dups.ValA AND yt.Article = dups.Article

Open in new window

0
 
LVL 15

Assisted Solution

by:Haris Djulic
Haris Djulic earned 167 total points
ID: 40431580
This is the code that will find you the duplicates :

select rtrim(ValA)+rtrim(Article),count( *)
from your_table_name
group by rtrim(ValA)+rtrim(Article)
having count(*)>1

Open in new window


and I pressume you have some kind of additional ID field in your table you can use it to delete those with minimum IDs..

delete from your_table_name
where YOUR_ID_FIELD in (
  select min(YOUR_ID_FIELD ) from your_table_name where rtrim(ValA)+rtrim(Article) in  
  (select value from (
    
    select rtrim(ValA)+rtrim(Article) value,count( *) num
from your_table_name
group by rtrim(ValA)+rtrim(Article)
having count(*)>1)a
    )
  )

Open in new window

0
 
LVL 24

Assisted Solution

by:chaau
chaau earned 166 total points
ID: 40431628
This code will remove duplicates leaving at least one record from the duplicate records intact:
with cte as (
select *, ROW_NUMBER() OVER (PARTITION BY ValA, Article) AS rn
from yourTable)
DELETE FROM cte WHERE rn > 1

Open in new window

If you have a column that you can use as a sort order to determine which record is more preferred to be kept than you can use it. E.g. imagine you have a dateAdded column that tells when the record was added. If you wish to keep the most recent records you would use this query:
with cte as (
select *, ROW_NUMBER() OVER (PARTITION BY ValA, Article ORDER BY dateAdded DESC) AS rn
from yourTable)
DELETE FROM cte WHERE rn > 1

Open in new window

0
 

Author Closing Comment

by:murbro
ID: 40442665
Thank you all
0
 
LVL 65

Expert Comment

by:Jim Horn
ID: 40442690
Thanks for the split.  Good luck with your project.  -Jim
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

When you hear the word proxy, you may become apprehensive. This article will help you to understand Proxy and when it is useful. Let's talk Proxy for SQL Server. (Not in terms of Internet access.) Typically, you'll run into this type of problem w…
This article shows gives you an overview on SQL Server 2016 row level security. You will also get to know the usages of row-level-security and how it works
Familiarize people with the process of retrieving data from SQL Server using an Access pass-thru query. Microsoft Access is a very powerful client/server development tool. One of the ways that you can retrieve data from a SQL Server is by using a pa…
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now