Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Easy-peasy:  Script to delete duplicate lines in a file

Posted on 2006-11-21
2
985 Views
Last Modified: 2012-05-05
This should be really simple, i know, but i'm quite new to scripting and i just don't have any idea how to do it.

Imagine a process that returns a list of paths, with the form:

aaa/bbb/ccc
aaa/bbb/ccc
aaa/bbb/ccc
ddd/eee/fff
ddd/eee/fff
ddd/eee/fff
bbb/ccc/ddd
bbb/ccc/ddd
.
.
.

what script will reduce that to a file that just has:

aaa/bbb/ccc
ddd/eee/fff
bbb/ccc/ddd

and nothing else?

Currrently i've tried this:

# FileList is the file that's being created.

: > FileList
<first process> |
while read "b"
        do
        c=`dirname "$b"`
        d=`sed -n '/"$c"/p' FileList`
        if [ "$d" = "$c" ]
                then
                        continue
                else
                        echo "$c" >> List
                fi
        done

I've tried a few other variations, all based around this theme, but can't seem to get it to work.  Any and all ideas are welcome, and wordy explanations are preferred.  ;-)
0
Comment
Question by:kyle_in_taiwan
2 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
ID: 17987476
try: uniq original > uniqued.txt
Note: uniq only deletes duplicates that follows one by another. ie if Your file have
aaa/bbb/aaa
aaa/ccc/aaa
aaa/bbb/aaa

uniq will do nothing.
0
 
LVL 5

Author Comment

by:kyle_in_taiwan
ID: 17987675
Cool.  I've been able to wrangle a solution out of that one already.  Thanks.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Help Linux centos 5.7 3 94
Linux daemon 11 356
rhel6 C system() call to zip to archive has problems 25 245
Trying to install Node.js on Linux instance. Erroring.... 2 77
Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question