Avatar of cantthinkofone
cantthinkofoneFlag for United Kingdom of Great Britain and Northern Ireland asked on

Perl : read a file and output only distinct lines (eliminate all duplicates)

I have a file which has content something like the following:

01 A line of text
02 Another line of text
03 Yet another one
01 A line of text
04 More text
02 Another line of text
01 A line of text


How do I read this file and output, to another file, only the "unique" rows, ie:

01 A line of text
02 Another line of text
03 Yet another one
04 More text

is this easy to do in a Perl script?

As always, any help much apprfeciated!
Perl

Avatar of undefined
Last Comment
cantthinkofone

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
ozo

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
funtaff

I'm sorry, but that was just beautiful.
funtaff

Based on what ozo said, you want something like this.  Make sure to give ozo the credit.


#!/usr/bin/perl
 
$in_file = "input.txt";
$out_file = "output.txt";
 
open(INFILE, $in_file) or die "Cannot open input file";
open(OUTFILE, ">$out_file") or die "Cannot open output file";
 
while( <INFILE> )
{
    print OUTFILE unless $seen{$_}++;
}
 
close(INFILE);
close(OUTFILE);

Open in new window

ASKER
cantthinkofone

Brilliant - thanks!  I'm always surprised when something I have thought to be tricky (for me) turns out to be so few lines of code :)  

Thanks for the extra bit too funtaff!
Your help has saved me hundreds of hours of internet surfing.
fblack61