Solved

SQL query to detect and delete duplicate records

Posted on 2004-08-05
5
4,606 Views
Last Modified: 2012-08-13
I need a SQL query to detect and delete duplicate records, that is records where firstname and lastname are identical (if the case is different, would still be duplicate).  Detection and deletion could be in separate steps.
Thanks
0
Comment
Question by:MichaelMullin
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
5 Comments
 
LVL 50

Accepted Solution

by:
Lowfatspread earned 125 total points
ID: 11730240
you need the Primary key of the row as well

select 'matched to ', t.pk, d.*
from  Table as D
inner Join Table as T
on D.Pk < T.PK
and D.Firstname=T.FirstName
and D.Lastname=T.lastName

you need criteria to decide which one to delete  

Delete from Table
Where Exists (Select T.pk from table as T
                      Where T.pk > Table.pk
                         and T.firstname=Table.firstname
                        and t.lastname = table.lastname)


I hope this isn't HOMEWORK ?    
0
 
LVL 17

Expert Comment

by:BillAn1
ID: 11730281
If no primary key, try something like this :

SELECT DISTINCT firstname, lastname
INTO #temp_table
FROM source_table

DELETE FROM source_table

INSERT INTO source_table
SELECT * FROM #temp_table

0
 
LVL 50

Expert Comment

by:Lowfatspread
ID: 11730356
do you have a case sensistivity problem ?

if so convert both names to upper case and then do the test....

0
 
LVL 34

Expert Comment

by:arbert
ID: 11730487
Agree with lowfat--if you have something you can use for a key, you're better off to use that method.  If not, you need to use a method like BillAn1 suggested (I would just truncate the table instead of deleting the old rows--also, if you have a lot of data, this can be very slow) or a cursor.
0
 
LVL 69

Expert Comment

by:Scott Pletcher
ID: 11730947
Here is a sample using a cursor but that does not require a separate table or a dump/reload:



DECLARE dupsCsr CURSOR READ_ONLY FOR
SELECT [firstName], [lastName], COUNT(*) AS numDups
FROM yourTable
GROUP BY [firstName], [lastName]
HAVING COUNT(*) > 1
DECLARE @firstName VARCHAR(30) --Change to match datatype on your table
DECLARE @lastName VARCHAR(30) --Change to match datatype on your table
DECLARE @numDups INT

OPEN dupsCsr
FETCH NEXT FROM dupsCsr INTO @firstName, @lastName, @numDups
WHILE @@FETCH_STATUS = 0
BEGIN
      SET @numDups = @numDups - 1 --delete all but 1 of the duplicates
      SET ROWCOUNT @numDups
      DELETE FROM yourTable
      WHERE [firstName] = @firstName AND [lastName] = @lastName
      FETCH NEXT FROM dupsCsr INTO @firstName, @lastName, @numDups
END --WHILE
CLOSE dupsCsr
DEALLOCATE dupsCsr

SET ROWCOUNT 0 --restore default
0

Featured Post

Database Solutions Engineer FAQs

In this series, we will discuss common questions received as a database Solutions Engineer at Percona. In this role, we speak with a wide array of MySQL and MongoDB users responsible for both extremely large and complex environments to smaller single-server environments.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article shows gives you an overview on SQL Server 2016 row level security. You will also get to know the usages of row-level-security and how it works
Recently we ran in to an issue while running some SQL jobs where we were trying to process the cubes.  We got an error saying failure stating 'NT SERVICE\SQLSERVERAGENT does not have access to Analysis Services. So this is a way to automate that wit…
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

635 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question