Solved

linux Sort Unique by Column

Posted on 2013-06-04
5
590 Views
Last Modified: 2013-06-04
Environment:

Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 GNU/Linux

I need to sort a file and eliminate duplicates based on the first column of a text file.

The text file looks like this (partial):

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  

What I want the output to be after the sort is:

ADEYEFP2~emghlp1071~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1038~145.32  

--essentially eliminating duplicate rows based only on the first column as delimited by "~".

I have tried various combinations of sort, but I am unable to get it to work.

Any help will be much appreciated!

Thanks
0
Comment
Question by:dhite99
  • 2
  • 2
5 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 39219943
try this

sed 's/\~/ /' filename | sort -u -n -k1,1
0
 

Author Comment

by:dhite99
ID: 39220007
It seems to only output the first line of the input file:

oracle@ebhlx001:/u01/app/oracle/dev> cat test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM8UETC2~emghlc1028~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP1~emghlp1059~213.32
AM9WAFP1~emghlp1060~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002~29.36
oracle@ebhlx001:/u01/app/oracle/dev>

Open in new window

0
 
LVL 40

Expert Comment

by:omarfarid
ID: 39220059
please try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1

If you don't want to see rest of the fields of lines

try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1 | cut -d ' ' -f 1
0
 
LVL 23

Accepted Solution

by:
nemws1 earned 500 total points
ID: 39220155
I'd suggest this:

sort -u -t~ -k1,1 test.txt

Open in new window


But I have a caveat with your example.  From these 2 lines, you want the first one:

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38

That's all good.  But from these 2, you say you want the *second* one:

ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  


Without additional info on how you are making that decision, we can't reproduce your results.
0
 

Author Comment

by:dhite99
ID: 39220254
omarfarid: It still only returns one row:

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /g' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002 29.36          

nemws1: That worked, thanks

oracle@ebhlx001:/u01/app/oracle/dev> sort -u -t~ -k1,1 test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question