dhite99
asked on
linux Sort Unique by Column
Environment:
Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 GNU/Linux
I need to sort a file and eliminate duplicates based on the first column of a text file.
The text file looks like this (partial):
ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32
What I want the output to be after the sort is:
ADEYEFP2~emghlp1071~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1038~145.32
--essentially eliminating duplicate rows based only on the first column as delimited by "~".
I have tried various combinations of sort, but I am unable to get it to work.
Any help will be much appreciated!
Thanks
Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64 GNU/Linux
I need to sort a file and eliminate duplicates based on the first column of a text file.
The text file looks like this (partial):
ADEYEFP2~emghlp1071~103.38
ADEYEFP2~emghlp1072~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1037~145.32
ADEYETC2~emghlc1038~145.32
What I want the output to be after the sort is:
ADEYEFP2~emghlp1071~103.38
ADEYETC1~emghlc054~57.11
ADEYETC2~emghlc1038~145.32
--essentially eliminating duplicate rows based only on the first column as delimited by "~".
I have tried various combinations of sort, but I am unable to get it to work.
Any help will be much appreciated!
Thanks
ASKER
It seems to only output the first line of the input file:
oracle@ebhlx001:/u01/app/oracle/dev> cat test.txt
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM8UETC2~emghlc1028~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP1~emghlp1059~213.32
AM9WAFP1~emghlp1060~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
oracle@ebhlx001:/u01/app/oracle/dev> sed 's/\~/ /' test.txt | sort -u -n -k1,1
AM8UETC1 emghlc002~29.36
oracle@ebhlx001:/u01/app/oracle/dev>
please try
sed 's/\~/ /g' test.txt | sort -u -n -k1,1
If you don't want to see rest of the fields of lines
try
sed 's/\~/ /g' test.txt | sort -u -n -k1,1 | cut -d ' ' -f 1
sed 's/\~/ /g' test.txt | sort -u -n -k1,1
If you don't want to see rest of the fields of lines
try
sed 's/\~/ /g' test.txt | sort -u -n -k1,1 | cut -d ' ' -f 1
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
omarfarid: It still only returns one row:
oracle@ebhlx001:/u01/app/o
AM8UETC1 emghlc002 29.36
nemws1: That worked, thanks
oracle@ebhlx001:/u01/app/o
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
oracle@ebhlx001:/u01/app/o
AM8UETC1 emghlc002 29.36
nemws1: That worked, thanks
oracle@ebhlx001:/u01/app/o
AM8UETC1~emghlc002~29.36
AM8UETC2~emghlc1027~99.96
AM9WAFP1~emghlp1058~213.32
AM9WAFP2~emghlp017~339.95
AM9WAFP3~emghlp017~8.59
AM9WARP2~emghlp044~324.5
AM9WASC1~emghlc002~444.59
sed 's/\~/ /' filename | sort -u -n -k1,1