Solved

Need to delete duplicate records

Posted on 2016-08-20
7
46 Views
Last Modified: 2016-08-24
I was having problems with a table and I realized that there are some duplicate records that need to be deleted.
How do I do this if they are exact copies?

STOREKEY          |STORENUM|BNMB|STRT
NY1S05000010 |1001             |10       |Spring Street
NY1S05000010 |1001             |10       |Spring Street

I need to get rid of the 2nd one.  The dupes are not all over the file.  In other words, It only happens occasionally.
How it happened is beyond me.

Is there a quick script that I could write to get rid of these?
0
Comment
Question by:breeze351
7 Comments
 
LVL 9

Expert Comment

by:bas2754
ID: 41763851
I ran into this once and followed the guidance of this article to resolve:
-----
https://support.microsoft.com/en-us/kb/139444

I would advise testing on a test table first and ensure the results are what you expect.  Another expert may be able to provide exact commands for your situation, but this should get you headed in the right direction.
1
 
LVL 65

Expert Comment

by:Jim Horn
ID: 41763865
Let me know how this grabs ya:  SQL Server Delete Duplicate Rows Solutions.

Applied to your example and tested on my SQL 2012 box...
IF OBJECT_ID('tempdb..#tmp') IS NOT NULL
	DROP TABLE #tmp
GO

CREATE TABLE #tmp (STOREKEY varchar(20), STORENUM int, BNMB int, STRT varchar(20))
GO

INSERT INTO #tmp (STOREKEY, STORENUM, BNMB, STRT) 
VALUES 
   ('NY1S05000010', 1001, 10, 'Spring Street'), 
   ('NY1S05000010', 1001, 10, 'Spring Street'), -- a duplicate row
   ('TX1S05000010', 5963, 10, 'Main Street')  -- a row I just made up

-- Before removing the duplicates
SELECT * FROM #tmp

-- Delete the duplicates
;with cte as (
SELECT 
	STOREKEY, STORENUM, BNMB, STRT, 
	row_number() OVER (PARTITION BY STOREKEY, STORENUM, BNMB, STRT ORDER BY (SELECT NULL)) as row_number 
FROM #tmp) 
DELETE 
FROM cte
WHERE row_number > 1

-- After removing the duplicates
SELECT * FROM #tmp

 

Open in new window

3
 

Accepted Solution

by:
breeze351 earned 0 total points
ID: 41763880
Ok, I thought there might an easier way to do it.
I just write a record to a new table, if the key is there, I don't write.  When I'm done I no longer have dupes.
Thanks
Glenn
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:breeze351
ID: 41763901
I've requested that this question be closed as follows:

Accepted answer: 0 points for breeze351's comment #a41763880

for the following reason:

I had already thought of it.  Thought there might be an easier way
0
 
LVL 65

Expert Comment

by:Jim Horn
ID: 41763882
>Ok, I thought there might an easier way to do it.
Once the records are in the table, that's as easy as it gets.

>I just write a record to a new table, if the key is there, I don't write.
To prevent this from happening before the write you can create a unique index on the four columns, but there are other considerations in play such as memory, speed of insert/update, graceful way to handle violations, etc. that may not make this practical.

Another possibility is a trigger on the table to check inserts and updates, but if you're not comfortable with these then I'd recommend against them as it's added overhead.
0
 
LVL 51

Expert Comment

by:Mark Wills
ID: 41764353
>> How it happened is beyond me.

A couple of possibilities :
1) There is a unique identifier as a key
2) There is no unique index
3) unlikely, but possible that an ALTER TABLE with NOCHECK has happened at some stage to disable constraints

Looking at the data, it would appear that storekey should be enough to make that the unique / primary key.

So, once you clean up the duplicates, it would be worthwhile to then create a unique index :
create unique index PK_STOREKEY on <your table name> (STOREKEY)

Open in new window

or if STOREKEY is not nullable (if it is, the above code works), create a primary key
alter table <your table name> with nocheck add constraint PK_STOREKEY PRIMARY KEY (STOREKEY)

Open in new window


Hope that helps a bit more with ways to avoid it happening again.
1

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Nowadays, some of developer are too much worried about data. Who is using data, who is updating it etc. etc. Because, data is more costlier in term of money and information. So security of data is focusing concern in days. Lets' understand the Au…
Everyone has problem when going to load data into Data warehouse (EDW). They all need to confirm that data quality is good but they don't no how to proceed. Microsoft has provided new task within SSIS 2008 called "Data Profiler Task". It solve th…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now