Solved

List of Unique values

Posted on 2008-06-24
7
734 Views
Last Modified: 2010-04-21
I need (preferably) a Perl script that

- reads input from stdin ... Input is a huge volume of records ... Fields/Columns in each record are tab separated ... Number of fields is not known in advance but all records will have same number of fields.
- accepts column numbers as command line arguments
- outputs all unique values seen in the input for the specified columns

e.g.
input file
A       22      78      rest
E       22      90      best
A       32      55      lest

./myscript.pl 1 4
ie, output all unique values in column 1 and column 4 ... output would look something like

COLUMN 1
A
E

COLUMN 4
rest
best
lest

While in most cases number of unique values must fit in the memory, there are some cases where they may be too big to fit in ... If such cases can be handled - well and good ... in case such cases cannot be handled, it would be good enough if a message saying "too many values in column n" is displayed.
0
Comment
Question by:sunnycoder
  • 4
  • 3
7 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 21862697
perl -alne 'BEGIN{@c=splice @ARGV}$c{$_}{$F[$_-1]}++for@c;END{print join"\n","COLUMN $_",keys %{$c{$_}},""for @c}' 1 4 < input
0
 
LVL 45

Author Closing Comment

by:sunnycoder
ID: 31470458
Perfect again ... thanks!!
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862729
Sorry about my complete unfamiliarity with perl ... how do I convert this above command into a script that accepts arguments.
0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 
LVL 84

Expert Comment

by:ozo
ID: 21862763
this only keeps one column in memory at a time

perl -alne 'BEGIN{@c=@ARGV; @ARGV=(pop @c)x@c}print"COLUMN ",$c=shift @c and %s=() if 1..1; $s{$_}++||print for $F[$c-1];close ARGV&&print""if eof' 1 4  input
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862782
Its okay if it keeps all column in memory at the same time ... What I want is to be able to put it in a script

./myscript.pl 1 4

instead of

perl -alne .....
0
 
LVL 84

Expert Comment

by:ozo
ID: 21862824
#!/usr/bin/perl
while( <STDIN> ){
   $c=0;
   $c[$c++]{$_}++ for ('',split)[@ARGV];
}
$\=$/;
for( @ARGV ){
   print "COLUMN $_";
   print for keys %{shift @c};
}
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862831
perfect ... thanks a ton
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Recently I have been answering a lot of questions like this in IT forums that I frequent. The question posed is usually something along the lines of "We have software X installed and need to uninstall it for reason Y" or some other variant of the sa…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)

919 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now