Solved

Removing duplication in comma separated log file.

Posted on 2004-09-22
6
130 Views
Last Modified: 2010-03-05
Dear All:

I have a file containing comma separated entries e.g.

test, test1, test2 ..etc
test, test1, test2 ..etc

You can see these two entries are same. I want a perl program that look at this entry and more, in the log file I have there could be thousand of entries that could be simmilar. I want a  program which for hundred's of common entries just would writes one entry only.

Best Regards

sunnybrad
0
Comment
Question by:sunnybrad
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
6 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 250 total points
ID: 12128754
my %h;
while( <> ){
   print unless $h{$_}++;
}
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 250 total points
ID: 12128931
ozo, of course, has given a nice compact solution.

If you are on a Unix system, you can do:

uniq file >newfile

or if the duplicate lines are spread out and you don't mind it being sorted, you can do:

sort -u file >newfile
0
 
LVL 84

Expert Comment

by:ozo
ID: 12129566
see also
perldoc -q duplicate
0
 
LVL 8

Expert Comment

by:davorg
ID: 12131148
A command line version of ozo's solution

perl -i.bak -ne 'print unless $h{$_}++' your_file_goes_here


Dave...
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Perl Script - Remove row of data based on column value 3 78
Problem with one substitution modifying another 3 84
Perl modules on linux ec2 3 131
Perl tar error 8 75
I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

710 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question