Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

linux Sort Unique by Column

Posted on 2013-06-04
5
Medium Priority
?
603 Views
Last Modified: 2013-06-04
Environment:

Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 GNU/Linux

I need to sort a file and eliminate duplicates based on the first column of a text file.

The text file looks like this (partial):

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  

What I want the output to be after the sort is:

ADEYEFP2~emghlp1071~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1038~145.32  

--essentially eliminating duplicate rows based only on the first column as delimited by "~".

I have tried various combinations of sort, but I am unable to get it to work.

Any help will be much appreciated!

Thanks
0
Comment
Question by:dhite99
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 39219943
try this

sed 's/\~/ /' filename | sort -u -n -k1,1
0
 

Author Comment

by:dhite99
ID: 39220007
It seems to only output the first line of the input file:

oracle@ebhlx001:/u01/app/oracle/dev> cat test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM8UETC2~emghlc1028~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP1~emghlp1059~213.32
AM9WAFP1~emghlp1060~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002~29.36
oracle@ebhlx001:/u01/app/oracle/dev>

Open in new window

0
 
LVL 40

Expert Comment

by:omarfarid
ID: 39220059
please try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1

If you don't want to see rest of the fields of lines

try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1 | cut -d ' ' -f 1
0
 
LVL 23

Accepted Solution

by:
nemws1 earned 2000 total points
ID: 39220155
I'd suggest this:

sort -u -t~ -k1,1 test.txt

Open in new window


But I have a caveat with your example.  From these 2 lines, you want the first one:

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38

That's all good.  But from these 2, you say you want the *second* one:

ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  


Without additional info on how you are making that decision, we can't reproduce your results.
0
 

Author Comment

by:dhite99
ID: 39220254
omarfarid: It still only returns one row:

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /g' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002 29.36          

nemws1: That worked, thanks

oracle@ebhlx001:/u01/app/oracle/dev> sort -u -t~ -k1,1 test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
0

Featured Post

URL rewriting in AWS CloudFront

A quick how-to guide to implement with a Lambda function!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question