Solved

List of Unique values

Posted on 2008-06-24
7
737 Views
Last Modified: 2010-04-21
I need (preferably) a Perl script that

- reads input from stdin ... Input is a huge volume of records ... Fields/Columns in each record are tab separated ... Number of fields is not known in advance but all records will have same number of fields.
- accepts column numbers as command line arguments
- outputs all unique values seen in the input for the specified columns

e.g.
input file
A       22      78      rest
E       22      90      best
A       32      55      lest

./myscript.pl 1 4
ie, output all unique values in column 1 and column 4 ... output would look something like

COLUMN 1
A
E

COLUMN 4
rest
best
lest

While in most cases number of unique values must fit in the memory, there are some cases where they may be too big to fit in ... If such cases can be handled - well and good ... in case such cases cannot be handled, it would be good enough if a message saying "too many values in column n" is displayed.
0
Comment
Question by:sunnycoder
  • 4
  • 3
7 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 21862697
perl -alne 'BEGIN{@c=splice @ARGV}$c{$_}{$F[$_-1]}++for@c;END{print join"\n","COLUMN $_",keys %{$c{$_}},""for @c}' 1 4 < input
0
 
LVL 45

Author Closing Comment

by:sunnycoder
ID: 31470458
Perfect again ... thanks!!
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862729
Sorry about my complete unfamiliarity with perl ... how do I convert this above command into a script that accepts arguments.
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 84

Expert Comment

by:ozo
ID: 21862763
this only keeps one column in memory at a time

perl -alne 'BEGIN{@c=@ARGV; @ARGV=(pop @c)x@c}print"COLUMN ",$c=shift @c and %s=() if 1..1; $s{$_}++||print for $F[$c-1];close ARGV&&print""if eof' 1 4  input
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862782
Its okay if it keeps all column in memory at the same time ... What I want is to be able to put it in a script

./myscript.pl 1 4

instead of

perl -alne .....
0
 
LVL 84

Expert Comment

by:ozo
ID: 21862824
#!/usr/bin/perl
while( <STDIN> ){
   $c=0;
   $c[$c++]{$_}++ for ('',split)[@ARGV];
}
$\=$/;
for( @ARGV ){
   print "COLUMN $_";
   print for keys %{shift @c};
}
0
 
LVL 45

Author Comment

by:sunnycoder
ID: 21862831
perfect ... thanks a ton
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question