[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 270
  • Last Modified:

Import Database into existing database, without the duplicates!

My working database contains nearly 300,000 records and one of my clients gave me their database of 40,000 records, but I know that there are many, many duplicates. I got great help the last time I did this and was able to fix a duplicate problem AFTER I imported the data. This time, I would like to get rid of the duplicates before I import them. First Name, Last Name and Company could be a good way to compare the records in both databases, but many of these people have left their companies, or I have the old information. For now, I would just like to see which names in my clients database DO NOT exist in my database and import them. Of course, this will not help with the Mikes and Michaels, or the Susans and Sues, or the misspelled names, but it's a start...

Any suggestions?

  • 2
1 Solution
create a calculated field in both data bases called "check" for instance.
check = name&firstname&company (and any other if you'd like to improve dup detection.

make a link in between the bases using this field

create a calculated field, say 'link_true' in the small database (the 40k lines one) = if (database1::check ; "ok" ; "")
search all records in small database where link_true="ok"
go to main database, import the found set you just did from the small database.
johnmoedAuthor Commented:
I tried a couple different relationships and was able to delete thousands of exact duplicates. E-mail address is a really good one to use for this and I also used direct dial number.

Thanks again!
the more fields you use, the more accurate the dups will be!
the real problem is to take into account trailing spaces, phone nbs with different formats, so it is also a good idea to remove all double, leading and trailing spaces from all text fields and keep the deleted ones in a corner for a while just in case.
till next, thanks for the points, at least 3 weeks I had not got anything, and bill is chasing me!

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now