How to delete thousands of duplicate records in access?

Posted on 2014-01-24
Medium Priority
Last Modified: 2014-01-25
Dear All,

There has to be an easier way to do this aside from doing it manually.

See, I have a table with like 150k records in it and there are tonnes of duplicates that will take days to delete and filter manually.

Wondering if someone can come up with a re-useable routine that will allow me to delete large quantities of duplicate records in varying tables. I am comfortable with copying and pasting vba and modifying variables to suit a certain situation if this helps.

I've enclosed a sample of the problem in the db attachment.

Any help appreciated.

Question by:discogs
LVL 50

Expert Comment

by:Dale Fye
ID: 39807732
cannot download your database here, but a couple of questions.

1.  how do you determine duplicates (1 field, 2 fields, ...)?

2.  Once you have identified duplicates, how do you determine which one to keep?

One option is to use this criteria to Select the records you want to keep and push them to another, temporary, table.  Then delete all of the records from that table and insert the ones you saved into the temporary table back into your table.

Accepted Solution

Christopher Martinez earned 2000 total points
ID: 39807745
Hey bud! What i would suggest is using the "Find Duplicate" wizard and then using the Append Query wizard!

There is a great article on this over at Techrepublic.com that i wont try to beat with my explination but i believe this "how to" is what you need...even has nifty screenshots! Ive used this many times to clean up my duplicates.


Author Comment

ID: 39807761
Thanks for the answers guys. @ fyed, since you were the first to respond, I have the following answers for your question.

1. Fields - pkCountryCode, pkLocation, pkPortCode;
2. It does not matter which record is kept as the records are identical across all fields.

Hope this helps.

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.


Author Comment

ID: 39807762
Would it help if I cut down the file size?

Author Closing Comment

ID: 39807885
LVL 75
ID: 39808268
Seems that fyed answered the question first.
LVL 50

Expert Comment

by:Dale Fye
ID: 39808540
Thanks for the vote of confidence, Joe.  

The technique described in the article was not  quite the same technique as what I was recommending.  It was a great article with very clear instructions and visuals.  

My problem with duplicates is that they are generally not exact duplicates (as in the article), they may have the same street address, but don't have the same name, or phone number, or ...  and the challenge is determining which "duplicate" to get rid of.

In instances like this, the "keeper" record for each duplicate may be the one with the most recent [date modified], or simply the one with the largest autonumber ID (counter).  Those are generally easy to handle, you simply create a query that selects the earliest [data modified] or smallest [ID] from among each of the "duplicates".  I didn't have time to write this yesterday, so I will now for the sake of those that follow.

Assume myTable contains duplicate records based upon the Bolo_ID and docDate fields, and that the records you want to keep are the ones with the largest ID (primary key, counter/autonumber) field in the table.  Start out with the query to identify the unique records (based on Bolo_ID and docDate) within the dataset, and the maximum ID (counter/autonumber) value within each group.

SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate

Then, you simply join this query back to the original table and copy the matching records into a temporary table.

SELECT myTable.* INTO myTable_Temp
FROM myTable
SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate) as KeepThese
ON myTable.ID = KeepThese.KeepThisID

If only it were this easy all of the time!

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

If you need a simple but flexible process for maintaining an audit trail of who created, edited, or deleted data from a table, or multiple tables, and you can do all of your work from within a form, this simple Audit Log will work for you.
Beware when using the ListIndex and the Column() properties of a listbox in Access 2007.  A bug has been identified in the Access 2007 listbox code which can cause the .ListIndex property to return a -1, and the .Columns(#) property to return a NULL…
In Microsoft Access, when working with VBA, learn some techniques for writing readable and easily maintained code.
Hi, this video explains a free download that you can incorporate into your Access databases, or use stand-alone for contact management. Contacts -- Names, Addresses, Phone Numbers, eMail Addresses, Websites, Lists, Projects, Notes, Attachments…

597 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question