Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win


List of Unique values

Posted on 2008-06-24
Medium Priority
Last Modified: 2010-04-21
I need (preferably) a Perl script that

- reads input from stdin ... Input is a huge volume of records ... Fields/Columns in each record are tab separated ... Number of fields is not known in advance but all records will have same number of fields.
- accepts column numbers as command line arguments
- outputs all unique values seen in the input for the specified columns

input file
A       22      78      rest
E       22      90      best
A       32      55      lest

./myscript.pl 1 4
ie, output all unique values in column 1 and column 4 ... output would look something like



While in most cases number of unique values must fit in the memory, there are some cases where they may be too big to fit in ... If such cases can be handled - well and good ... in case such cases cannot be handled, it would be good enough if a message saying "too many values in column n" is displayed.
Question by:sunnycoder
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
LVL 84

Accepted Solution

ozo earned 2000 total points
ID: 21862697
perl -alne 'BEGIN{@c=splice @ARGV}$c{$_}{$F[$_-1]}++for@c;END{print join"\n","COLUMN $_",keys %{$c{$_}},""for @c}' 1 4 < input
LVL 45

Author Closing Comment

ID: 31470458
Perfect again ... thanks!!
LVL 45

Author Comment

ID: 21862729
Sorry about my complete unfamiliarity with perl ... how do I convert this above command into a script that accepts arguments.
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

LVL 84

Expert Comment

ID: 21862763
this only keeps one column in memory at a time

perl -alne 'BEGIN{@c=@ARGV; @ARGV=(pop @c)x@c}print"COLUMN ",$c=shift @c and %s=() if 1..1; $s{$_}++||print for $F[$c-1];close ARGV&&print""if eof' 1 4  input
LVL 45

Author Comment

ID: 21862782
Its okay if it keeps all column in memory at the same time ... What I want is to be able to put it in a script

./myscript.pl 1 4

instead of

perl -alne .....
LVL 84

Expert Comment

ID: 21862824
while( <STDIN> ){
   $c[$c++]{$_}++ for ('',split)[@ARGV];
for( @ARGV ){
   print "COLUMN $_";
   print for keys %{shift @c};
LVL 45

Author Comment

ID: 21862831
perfect ... thanks a ton

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Batch, VBS, and scripts in general are incredibly useful for repetitive tasks.  Some tasks can take a while to complete and it can be annoying to check back only to discover that your script finished 5 minutes ago.  Some scripts may complete nearly …
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Suggested Courses

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question