One liner---need to remove dups

Posted on 2005-04-28
Last Modified: 2008-03-06
guess i would have serveral option however what is a good one liner with perl that will remove
duplicate lines in a file.

I have a file with a few thousand lines in it and it has duplicate lines that was created because
of the "case"



how can i change all text to "lower case" then remove and line that is duplicated.


Question by:bt707

    Author Comment

    I found how to do it using    "tr -d" and "tr -s"  
    still curious if there is a simple way to do it with a
    perl command.
    LVL 16

    Accepted Solution

    dunno about a simple Perl command, but

    my %hash=() ;
    @arr = grep{!$hash{lc $_}++} <> ;
    print @arr ;


    Author Comment

    Thanks manav,

    sure glad to see your still around.

    I found a simple way to do what I needed with
    cat testfile777 | tr -s '[:upper:]' '[:lower:]' | sort | uniq > tr-s

    was curious how you would do it with perl, as usual there never seem
    to be anything you don't have a answer of solution to.

    LVL 84

    Expert Comment

    perldoc -q "How can I remove duplicate elements from a list or array?"

    perl -ne 'print unless $hash{lc}++' testfile777

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    What Is Threat Intelligence?

    Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

    Suggested Solutions

    I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
    There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
    Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
    In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor ( If you're interested in additional methods for monitoring bandwidt…

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    13 Experts available now in Live!

    Get 1:1 Help Now