List of Unique values

I need (preferably) a Perl script that

- reads input from stdin ... Input is a huge volume of records ... Fields/Columns in each record are tab separated ... Number of fields is not known in advance but all records will have same number of fields.
- accepts column numbers as command line arguments
- outputs all unique values seen in the input for the specified columns

e.g.
input file
A       22      78      rest
E       22      90      best
A       32      55      lest

./myscript.pl 1 4
ie, output all unique values in column 1 and column 4 ... output would look something like

COLUMN 1
A
E

COLUMN 4
rest
best
lest

While in most cases number of unique values must fit in the memory, there are some cases where they may be too big to fit in ... If such cases can be handled - well and good ... in case such cases cannot be handled, it would be good enough if a message saying "too many values in column n" is displayed.
LVL 45
sunnycoderAsked:
Who is Participating?
 
ozoConnect With a Mentor Commented:
perl -alne 'BEGIN{@c=splice @ARGV}$c{$_}{$F[$_-1]}++for@c;END{print join"\n","COLUMN $_",keys %{$c{$_}},""for @c}' 1 4 < input
0
 
sunnycoderAuthor Commented:
Perfect again ... thanks!!
0
 
sunnycoderAuthor Commented:
Sorry about my complete unfamiliarity with perl ... how do I convert this above command into a script that accepts arguments.
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

 
ozoCommented:
this only keeps one column in memory at a time

perl -alne 'BEGIN{@c=@ARGV; @ARGV=(pop @c)x@c}print"COLUMN ",$c=shift @c and %s=() if 1..1; $s{$_}++||print for $F[$c-1];close ARGV&&print""if eof' 1 4  input
0
 
sunnycoderAuthor Commented:
Its okay if it keeps all column in memory at the same time ... What I want is to be able to put it in a script

./myscript.pl 1 4

instead of

perl -alne .....
0
 
ozoCommented:
#!/usr/bin/perl
while( <STDIN> ){
   $c=0;
   $c[$c++]{$_}++ for ('',split)[@ARGV];
}
$\=$/;
for( @ARGV ){
   print "COLUMN $_";
   print for keys %{shift @c};
}
0
 
sunnycoderAuthor Commented:
perfect ... thanks a ton
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.