Solved

Removing duplication in comma separated log file.

Posted on 2004-09-22
6
125 Views
Last Modified: 2010-03-05
Dear All:

I have a file containing comma separated entries e.g.

test, test1, test2 ..etc
test, test1, test2 ..etc

You can see these two entries are same. I want a perl program that look at this entry and more, in the log file I have there could be thousand of entries that could be simmilar. I want a  program which for hundred's of common entries just would writes one entry only.

Best Regards

sunnybrad
0
Comment
Question by:sunnybrad
  • 2
6 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 250 total points
ID: 12128754
my %h;
while( <> ){
   print unless $h{$_}++;
}
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 250 total points
ID: 12128931
ozo, of course, has given a nice compact solution.

If you are on a Unix system, you can do:

uniq file >newfile

or if the duplicate lines are spread out and you don't mind it being sorted, you can do:

sort -u file >newfile
0
 
LVL 84

Expert Comment

by:ozo
ID: 12129566
see also
perldoc -q duplicate
0
 
LVL 8

Expert Comment

by:davorg
ID: 12131148
A command line version of ozo's solution

perl -i.bak -ne 'print unless $h{$_}++' your_file_goes_here


Dave...
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Perl Regular expression 9 199
Perl efficient DB Call 8 82
How to search multiple patterms in a file with perl? 4 79
Perl script to delete older files 6 73
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
In this video I am going to show you how to back up and restore Office 365 mailboxes using CodeTwo Backup for Office 365. Learn more about the tool used in this video here: http://www.codetwo.com/backup-for-office-365/ (http://www.codetwo.com/ba…

861 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now