Removing duplication in comma separated log file.

Dear All:

I have a file containing comma separated entries e.g.

test, test1, test2 ..etc
test, test1, test2 ..etc

You can see these two entries are same. I want a perl program that look at this entry and more, in the log file I have there could be thousand of entries that could be simmilar. I want a  program which for hundred's of common entries just would writes one entry only.

Best Regards

sunnybrad
sunnybradAsked:
Who is Participating?
 
ozoConnect With a Mentor Commented:
my %h;
while( <> ){
   print unless $h{$_}++;
}
0
 
TintinConnect With a Mentor Commented:
ozo, of course, has given a nice compact solution.

If you are on a Unix system, you can do:

uniq file >newfile

or if the duplicate lines are spread out and you don't mind it being sorted, you can do:

sort -u file >newfile
0
 
ozoCommented:
see also
perldoc -q duplicate
0
 
davorgCommented:
A command line version of ozo's solution

perl -i.bak -ne 'print unless $h{$_}++' your_file_goes_here


Dave...
0
All Courses

From novice to tech pro — start learning today.