Unix Shell script perl ( sed awk grep ) to screen out duplicate keys: Unix Shell script perl ( sed awk grep )

Unix Shell script perl ( sed awk grep ) to screen out duplicate keys

Asked by: sunhux

I have the following file sorted by the first column as the sort key :

111 found in platter 3297
111 found in platter 3322
1122 found in platter 3232
1129 found in platter 3297
1133 found in platter 3238
1133 found in platter 3297
1133 found in platter 3211
1255 found in platter 3213
1258 found in platter 3289
.....

I need a script that reads the above file & produces only those
lines / records that have the sort key occurring more than once,
so based on the above sample, the output should be :

111 found in platter 3297
111 found in platter 3322
1133 found in platter 3238
1133 found in platter 3297
1133 found in platter 3211

Instant Access to this Solution

Plus...

30 Day FREE access, no risk, no obligation

Collaborate with the world's top tech experts

Unlimited access to our exclusive solution database

Never be left without tech help again

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

"The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.

"Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin

"Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Thanks KeremE.

I've created another question, if you have the time, perhaps can help take a look
if you are available, thanks very much

Title: Unix Shell Perl script to filter out repeated sort key within a large file ( sed awk grep wc )

Question:

I have following files (with millions of lines) which is currently not sorted
by first column value & may have preceding space(s) in front :

113 found in platter 3222
127 found in platter 3922
113 found in platter 3735
1323 found in platter 3213
1323 found in platter 3898
53323 found in platter 3288
1323 found in platter 3223
....
(127 & 53323 each occurs only once in this file)

The output should be (ie those records with first column value that
occur once only will be eliminated & sorted by first column as primary
key followed by 5th column value as secondary sort key, so only
those lines with repeated first column value will be retained) :

113 found in platter 3222
113 found in platter 3735
1323 found in platter 3213
1323 found in platter 3223
1323 found in platter 3898
....

Question

Unix Shell script perl ( sed awk grep ) to screen out duplicate keys

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution