Solved

How to delete thousands of duplicate records in access?

Posted on 2014-01-24
7
881 Views
Last Modified: 2014-01-25
Dear All,

There has to be an easier way to do this aside from doing it manually.

See, I have a table with like 150k records in it and there are tonnes of duplicates that will take days to delete and filter manually.

Wondering if someone can come up with a re-useable routine that will allow me to delete large quantities of duplicate records in varying tables. I am comfortable with copying and pasting vba and modifying variables to suit a certain situation if this helps.

I've enclosed a sample of the problem in the db attachment.

Any help appreciated.

Thanks.
Database2.accdb
0
Comment
Question by:discogs
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
7 Comments
 
LVL 47

Expert Comment

by:Dale Fye (Access MVP)
ID: 39807732
cannot download your database here, but a couple of questions.

1.  how do you determine duplicates (1 field, 2 fields, ...)?

2.  Once you have identified duplicates, how do you determine which one to keep?

One option is to use this criteria to Select the records you want to keep and push them to another, temporary, table.  Then delete all of the records from that table and insert the ones you saved into the temporary table back into your table.
0
 
LVL 7

Accepted Solution

by:
Christopher Martinez earned 500 total points
ID: 39807745
Hey bud! What i would suggest is using the "Find Duplicate" wizard and then using the Append Query wizard!

There is a great article on this over at Techrepublic.com that i wont try to beat with my explination but i believe this "how to" is what you need...even has nifty screenshots! Ive used this many times to clean up my duplicates.

http://www.techrepublic.com/article/eliminate-duplicate-records-with-this-built-in-access-query/
0
 

Author Comment

by:discogs
ID: 39807761
Thanks for the answers guys. @ fyed, since you were the first to respond, I have the following answers for your question.

1. Fields - pkCountryCode, pkLocation, pkPortCode;
2. It does not matter which record is kept as the records are identical across all fields.

Hope this helps.

Ta
0
PeopleSoft Has Never Been Easier

PeopleSoft Adoption Made Smooth & Simple!

On-The-Job Training Is made Intuitive & Easy With WalkMe's On-Screen Guidance Tool.  Claim Your Free WalkMe Account Now

 

Author Comment

by:discogs
ID: 39807762
Would it help if I cut down the file size?
0
 

Author Closing Comment

by:discogs
ID: 39807885
Cheers.
0
 
LVL 75
ID: 39808268
Seems that fyed answered the question first.
0
 
LVL 47

Expert Comment

by:Dale Fye (Access MVP)
ID: 39808540
Thanks for the vote of confidence, Joe.  

The technique described in the article was not  quite the same technique as what I was recommending.  It was a great article with very clear instructions and visuals.  

My problem with duplicates is that they are generally not exact duplicates (as in the article), they may have the same street address, but don't have the same name, or phone number, or ...  and the challenge is determining which "duplicate" to get rid of.

In instances like this, the "keeper" record for each duplicate may be the one with the most recent [date modified], or simply the one with the largest autonumber ID (counter).  Those are generally easy to handle, you simply create a query that selects the earliest [data modified] or smallest [ID] from among each of the "duplicates".  I didn't have time to write this yesterday, so I will now for the sake of those that follow.

Assume myTable contains duplicate records based upon the Bolo_ID and docDate fields, and that the records you want to keep are the ones with the largest ID (primary key, counter/autonumber) field in the table.  Start out with the query to identify the unique records (based on Bolo_ID and docDate) within the dataset, and the maximum ID (counter/autonumber) value within each group.

SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate

Then, you simply join this query back to the original table and copy the matching records into a temporary table.

SELECT myTable.* INTO myTable_Temp
FROM myTable
INNER JOIN (
SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate) as KeepThese
ON myTable.ID = KeepThese.KeepThisID

If only it were this easy all of the time!
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction The Visual Basic for Applications (VBA) language is at the heart of every application that you write. It is your key to taking Access beyond the world of wizards into a world where anything is possible. This article introduces you to…
AutoNumbers should increment automatically, without duplicates.  But sometimes something goes wrong, and the next AutoNumber value is a duplicate.  This article shows how to recover from this problem.
Familiarize people with the process of utilizing SQL Server stored procedures from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Micr…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question