Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1006
  • Last Modified:

How to delete thousands of duplicate records in access?

Dear All,

There has to be an easier way to do this aside from doing it manually.

See, I have a table with like 150k records in it and there are tonnes of duplicates that will take days to delete and filter manually.

Wondering if someone can come up with a re-useable routine that will allow me to delete large quantities of duplicate records in varying tables. I am comfortable with copying and pasting vba and modifying variables to suit a certain situation if this helps.

I've enclosed a sample of the problem in the db attachment.

Any help appreciated.

Thanks.
Database2.accdb
0
discogs
Asked:
discogs
1 Solution
 
Dale FyeCommented:
cannot download your database here, but a couple of questions.

1.  how do you determine duplicates (1 field, 2 fields, ...)?

2.  Once you have identified duplicates, how do you determine which one to keep?

One option is to use this criteria to Select the records you want to keep and push them to another, temporary, table.  Then delete all of the records from that table and insert the ones you saved into the temporary table back into your table.
0
 
Christopher MartinezCommented:
Hey bud! What i would suggest is using the "Find Duplicate" wizard and then using the Append Query wizard!

There is a great article on this over at Techrepublic.com that i wont try to beat with my explination but i believe this "how to" is what you need...even has nifty screenshots! Ive used this many times to clean up my duplicates.

http://www.techrepublic.com/article/eliminate-duplicate-records-with-this-built-in-access-query/
0
 
discogsAuthor Commented:
Thanks for the answers guys. @ fyed, since you were the first to respond, I have the following answers for your question.

1. Fields - pkCountryCode, pkLocation, pkPortCode;
2. It does not matter which record is kept as the records are identical across all fields.

Hope this helps.

Ta
0
Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

 
discogsAuthor Commented:
Would it help if I cut down the file size?
0
 
discogsAuthor Commented:
Cheers.
0
 
DatabaseMX (Joe Anderson - Microsoft MVP, Access and Data Platform)Commented:
Seems that fyed answered the question first.
0
 
Dale FyeCommented:
Thanks for the vote of confidence, Joe.  

The technique described in the article was not  quite the same technique as what I was recommending.  It was a great article with very clear instructions and visuals.  

My problem with duplicates is that they are generally not exact duplicates (as in the article), they may have the same street address, but don't have the same name, or phone number, or ...  and the challenge is determining which "duplicate" to get rid of.

In instances like this, the "keeper" record for each duplicate may be the one with the most recent [date modified], or simply the one with the largest autonumber ID (counter).  Those are generally easy to handle, you simply create a query that selects the earliest [data modified] or smallest [ID] from among each of the "duplicates".  I didn't have time to write this yesterday, so I will now for the sake of those that follow.

Assume myTable contains duplicate records based upon the Bolo_ID and docDate fields, and that the records you want to keep are the ones with the largest ID (primary key, counter/autonumber) field in the table.  Start out with the query to identify the unique records (based on Bolo_ID and docDate) within the dataset, and the maximum ID (counter/autonumber) value within each group.

SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate

Then, you simply join this query back to the original table and copy the matching records into a temporary table.

SELECT myTable.* INTO myTable_Temp
FROM myTable
INNER JOIN (
SELECT Max([ID]) as KeepThisID
FROM myTable
GROUP By Bolo_ID, docDate) as KeepThese
ON myTable.ID = KeepThese.KeepThisID

If only it were this easy all of the time!
0

Featured Post

Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

Tackle projects and never again get stuck behind a technical roadblock.
Join Now