Best performance duplication prevention a collection of objects?

Posted on 2005-04-20
Last Modified: 2010-03-31
I've created an application which scans through a log file line by line and creates an object representing each line... I.e the object will contain date strings line number etc...

 I wish to prevent duplicates of a certain type. i.e duplicate objects would have the same date but different string and line numbers....

I've been using a ArrayList of objects and each time before I add an object I check if the object is in the arraylist already.. I've overloaded the equals() function and I'm using the arraylist.contains(object) to check for the duplicate...

My problem is that the arraylist can have up to 60,000 elements, so I have to check for duplicates each time before I add an element..

Anyone know of a more efficient way of preventing duplicates because this is realy inefficient.
Question by:conorocallaghan
    LVL 86

    Expert Comment

    You can use a Set. The Set cannot contain duplicates
    LVL 15

    Expert Comment

    Yes, as mentioned above, HashSet would be your best bet.
    HashSet set = new HashSet();

    if (set.contains(key))
    // skip this
    continue logic...
    LVL 86

    Accepted Solution

    >>if (set.contains(key))

    Is not necessary to prevent duplicates - they won't occur anyway. It would only be necessary if you wanted to know 'key' is going to be skipped
    LVL 15

    Expert Comment

    I assume that set is not pre populated but get populated as it goes thru the log file.
    And in that case an action might be taken when line already exists... (and not necessarily override the old line with the new one).
    LVL 86

    Expert Comment

    :-) but why a 'C'?

    Featured Post

    Top 6 Sources for Identifying Threat Actor TTPs

    Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    sameEnds challenge 25 57
    scores100 challenge 3 67
    countHi2 challenge 7 30
    groovy example issue 10 33
    By the end of 1980s, object oriented programming using languages like C++, Simula69 and ObjectPascal gained momentum. It looked like programmers finally found the perfect language. C++ successfully combined the object oriented principles of Simula w…
    Introduction This article is the first of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article explains our test automation goals. Then rationale is given for the tools we use to a…
    Video by: Michael
    Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
    Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now