Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 134
  • Last Modified:

Removing duplication in comma separated log file.

Dear All:

I have a file containing comma separated entries e.g.

test, test1, test2 ..etc
test, test1, test2 ..etc

You can see these two entries are same. I want a perl program that look at this entry and more, in the log file I have there could be thousand of entries that could be simmilar. I want a  program which for hundred's of common entries just would writes one entry only.

Best Regards

sunnybrad
0
sunnybrad
Asked:
sunnybrad
  • 2
2 Solutions
 
ozoCommented:
my %h;
while( <> ){
   print unless $h{$_}++;
}
0
 
TintinCommented:
ozo, of course, has given a nice compact solution.

If you are on a Unix system, you can do:

uniq file >newfile

or if the duplicate lines are spread out and you don't mind it being sorted, you can do:

sort -u file >newfile
0
 
ozoCommented:
see also
perldoc -q duplicate
0
 
davorgCommented:
A command line version of ozo's solution

perl -i.bak -ne 'print unless $h{$_}++' your_file_goes_here


Dave...
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now