Solved

Removing duplication in comma separated log file.

Posted on 2004-09-22
6
128 Views
Last Modified: 2010-03-05
Dear All:

I have a file containing comma separated entries e.g.

test, test1, test2 ..etc
test, test1, test2 ..etc

You can see these two entries are same. I want a perl program that look at this entry and more, in the log file I have there could be thousand of entries that could be simmilar. I want a  program which for hundred's of common entries just would writes one entry only.

Best Regards

sunnybrad
0
Comment
Question by:sunnybrad
  • 2
6 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 250 total points
ID: 12128754
my %h;
while( <> ){
   print unless $h{$_}++;
}
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 250 total points
ID: 12128931
ozo, of course, has given a nice compact solution.

If you are on a Unix system, you can do:

uniq file >newfile

or if the duplicate lines are spread out and you don't mind it being sorted, you can do:

sort -u file >newfile
0
 
LVL 84

Expert Comment

by:ozo
ID: 12129566
see also
perldoc -q duplicate
0
 
LVL 8

Expert Comment

by:davorg
ID: 12131148
A command line version of ozo's solution

perl -i.bak -ne 'print unless $h{$_}++' your_file_goes_here


Dave...
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
delete query using perl dbi 3 113
pattern matching in perl 2 111
Removing file extension within a file. 4 102
To run CGI/perl on tomcat and connect to oracle database. 5 146
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question