?
Solved

List of Unique values

Posted on 2008-06-24
7
Medium Priority
?
753 Views
Last Modified: 2010-04-21
I need (preferably) a Perl script that

- reads input from stdin ... Input is a huge volume of records ... Fields/Columns in each record are tab separated ... Number of fields is not known in advance but all records will have same number of fields.
- accepts column numbers as command line arguments
- outputs all unique values seen in the input for the specified columns

e.g.
input file
A       22      78      rest
E       22      90      best
A       32      55      lest

./myscript.pl 1 4
ie, output all unique values in column 1 and column 4 ... output would look something like

COLUMN 1
A
E

COLUMN 4
rest
best
lest

While in most cases number of unique values must fit in the memory, there are some cases where they may be too big to fit in ... If such cases can be handled - well and good ... in case such cases cannot be handled, it would be good enough if a message saying "too many values in column n" is displayed.
0
Comment
Question by:sunnycoder
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 21862697
perl -alne 'BEGIN{@c=splice @ARGV}$c{$_}{$F[$_-1]}++for@c;END{print join"\n","COLUMN $_",keys %{$c{$_}},""for @c}' 1 4 < input
0
 
LVL 45

Author Closing Comment

by:sunnycoder
ID: 31470458
Perfect again ... thanks!!
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862729
Sorry about my complete unfamiliarity with perl ... how do I convert this above command into a script that accepts arguments.
0
Congratulations! You’re Certified – Now What?

Starting a new career can be overwhelming. Becoming certified in your field of expertise is a great start, but where do you go from here?  Here are some tips to help you on your career journey.

 
LVL 84

Expert Comment

by:ozo
ID: 21862763
this only keeps one column in memory at a time

perl -alne 'BEGIN{@c=@ARGV; @ARGV=(pop @c)x@c}print"COLUMN ",$c=shift @c and %s=() if 1..1; $s{$_}++||print for $F[$c-1];close ARGV&&print""if eof' 1 4  input
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862782
Its okay if it keeps all column in memory at the same time ... What I want is to be able to put it in a script

./myscript.pl 1 4

instead of

perl -alne .....
0
 
LVL 84

Expert Comment

by:ozo
ID: 21862824
#!/usr/bin/perl
while( <STDIN> ){
   $c=0;
   $c[$c++]{$_}++ for ('',split)[@ARGV];
}
$\=$/;
for( @ARGV ){
   print "COLUMN $_";
   print for keys %{shift @c};
}
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862831
perfect ... thanks a ton
0

Featured Post

Setting up LaraDock for Laravel

Learn how to set up LaraDock in a Laravel project - LaraDock gives us an easy way to run a Laravel application using Docker in a single command.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
In threads here at EE, each comment has a unique Identifier (ID). It is easy to get the full path for an ID via the right-click context menu. However, we often want to post a short link within a thread rather than the full link. This article shows a…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
Suggested Courses

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question