Solved

linux Sort Unique by Column

Posted on 2013-06-04
5
596 Views
Last Modified: 2013-06-04
Environment:

Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 GNU/Linux

I need to sort a file and eliminate duplicates based on the first column of a text file.

The text file looks like this (partial):

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  

What I want the output to be after the sort is:

ADEYEFP2~emghlp1071~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1038~145.32  

--essentially eliminating duplicate rows based only on the first column as delimited by "~".

I have tried various combinations of sort, but I am unable to get it to work.

Any help will be much appreciated!

Thanks
0
Comment
Question by:dhite99
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 39219943
try this

sed 's/\~/ /' filename | sort -u -n -k1,1
0
 

Author Comment

by:dhite99
ID: 39220007
It seems to only output the first line of the input file:

oracle@ebhlx001:/u01/app/oracle/dev> cat test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM8UETC2~emghlc1028~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP1~emghlp1059~213.32
AM9WAFP1~emghlp1060~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002~29.36
oracle@ebhlx001:/u01/app/oracle/dev>

Open in new window

0
 
LVL 40

Expert Comment

by:omarfarid
ID: 39220059
please try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1

If you don't want to see rest of the fields of lines

try

sed 's/\~/ /g' test.txt | sort -u -n -k1,1 | cut -d ' ' -f 1
0
 
LVL 23

Accepted Solution

by:
nemws1 earned 500 total points
ID: 39220155
I'd suggest this:

sort -u -t~ -k1,1 test.txt

Open in new window


But I have a caveat with your example.  From these 2 lines, you want the first one:

ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38

That's all good.  But from these 2, you say you want the *second* one:

ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32  


Without additional info on how you are making that decision, we can't reproduce your results.
0
 

Author Comment

by:dhite99
ID: 39220254
omarfarid: It still only returns one row:

oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /g' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002 29.36          

nemws1: That worked, thanks

oracle@ebhlx001:/u01/app/oracle/dev> sort -u -t~ -k1,1 test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Configuring Apache Camel to send and recieive from QPID Broker 2 74
Linux boot cd to do hardware report on PC? 3 99
Internal CA server 6 134
umask commands 5 15
Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Recently, an awarded photographer, Selina De Maeyer (http://www.selinademaeyer.com/), completed a photo shoot of a beautiful event (http://www.sintjacobantwerpen.be/verslag-en-fotoreportage-van-de-sacramentsprocessie-door-antwerpen#thumbnails) in An…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question