Solved

How to delete thousands of duplicate records in access?

Posted on 2014-01-24
7
815 Views
Last Modified: 2014-01-25
Dear All,

There has to be an easier way to do this aside from doing it manually.

See, I have a table with like 150k records in it and there are tonnes of duplicates that will take days to delete and filter manually.

Wondering if someone can come up with a re-useable routine that will allow me to delete large quantities of duplicate records in varying tables. I am comfortable with copying and pasting vba and modifying variables to suit a certain situation if this helps.

I've enclosed a sample of the problem in the db attachment.

Any help appreciated.

Thanks.
Database2.accdb
0
Comment
Question by:discogs
7 Comments
 
LVL 47

Expert Comment

by:Dale Fye (Access MVP)
ID: 39807732
cannot download your database here, but a couple of questions.

1.  how do you determine duplicates (1 field, 2 fields, ...)?

2.  Once you have identified duplicates, how do you determine which one to keep?

One option is to use this criteria to Select the records you want to keep and push them to another, temporary, table.  Then delete all of the records from that table and insert the ones you saved into the temporary table back into your table.
0
 
LVL 7

Accepted Solution

by:
Christopher Martinez earned 500 total points
ID: 39807745
Hey bud! What i would suggest is using the "Find Duplicate" wizard and then using the Append Query wizard!

There is a great article on this over at Techrepublic.com that i wont try to beat with my explination but i believe this "how to" is what you need...even has nifty screenshots! Ive used this many times to clean up my duplicates.

http://www.techrepublic.com/article/eliminate-duplicate-records-with-this-built-in-access-query/
0
 

Author Comment

by:discogs
ID: 39807761
Thanks for the answers guys. @ fyed, since you were the first to respond, I have the following answers for your question.

1. Fields - pkCountryCode, pkLocation, pkPortCode;
2. It does not matter which record is kept as the records are identical across all fields.

Hope this helps.

Ta
0
Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

 

Author Comment

by:discogs
ID: 39807762
Would it help if I cut down the file size?
0
 

Author Closing Comment

by:discogs
ID: 39807885
Cheers.
0
 
LVL 75
ID: 39808268
Seems that fyed answered the question first.
0
 
LVL 47

Expert Comment

by:Dale Fye (Access MVP)
ID: 39808540
Thanks for the vote of confidence, Joe.  

The technique described in the article was not  quite the same technique as what I was recommending.  It was a great article with very clear instructions and visuals.  

My problem with duplicates is that they are generally not exact duplicates (as in the article), they may have the same street address, but don't have the same name, or phone number, or ...  and the challenge is determining which "duplicate" to get rid of.

In instances like this, the "keeper" record for each duplicate may be the one with the most recent [date modified], or simply the one with the largest autonumber ID (counter).  Those are generally easy to handle, you simply create a query that selects the earliest [data modified] or smallest [ID] from among each of the "duplicates".  I didn't have time to write this yesterday, so I will now for the sake of those that follow.

Assume myTable contains duplicate records based upon the Bolo_ID and docDate fields, and that the records you want to keep are the ones with the largest ID (primary key, counter/autonumber) field in the table.  Start out with the query to identify the unique records (based on Bolo_ID and docDate) within the dataset, and the maximum ID (counter/autonumber) value within each group.

SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate

Then, you simply join this query back to the original table and copy the matching records into a temporary table.

SELECT myTable.* INTO myTable_Temp
FROM myTable
INNER JOIN (
SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate) as KeepThese
ON myTable.ID = KeepThese.KeepThisID

If only it were this easy all of the time!
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

In the article entitled Working with Objects – Part 1 (http://www.experts-exchange.com/Microsoft/Development/MS_Access/A_4942-Working-with-Objects-Part-1.html), you learned the basics of working with objects, properties, methods, and events. In Work…
QuickBooks® has a great invoice interface that we were happy with for a while but that changed in 2001 through no fault of Intuit®. Our industry's unit names are dictated by RUS: the Rural Utilities Services division of USDA. Contracts contain un…
Familiarize people with the process of retrieving data from SQL Server using an Access pass-thru query. Microsoft Access is a very powerful client/server development tool. One of the ways that you can retrieve data from a SQL Server is by using a pa…
In Microsoft Access, learn how to use Dlookup and other domain aggregate functions and one method of specifying a string value within a string. Specify the first argument, which is the expression to be returned: Specify the second argument, which …

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now