Solved

Easy-peasy:  Script to delete duplicate lines in a file

Posted on 2006-11-21
2
983 Views
Last Modified: 2012-05-05
This should be really simple, i know, but i'm quite new to scripting and i just don't have any idea how to do it.

Imagine a process that returns a list of paths, with the form:

aaa/bbb/ccc
aaa/bbb/ccc
aaa/bbb/ccc
ddd/eee/fff
ddd/eee/fff
ddd/eee/fff
bbb/ccc/ddd
bbb/ccc/ddd
.
.
.

what script will reduce that to a file that just has:

aaa/bbb/ccc
ddd/eee/fff
bbb/ccc/ddd

and nothing else?

Currrently i've tried this:

# FileList is the file that's being created.

: > FileList
<first process> |
while read "b"
        do
        c=`dirname "$b"`
        d=`sed -n '/"$c"/p' FileList`
        if [ "$d" = "$c" ]
                then
                        continue
                else
                        echo "$c" >> List
                fi
        done

I've tried a few other variations, all based around this theme, but can't seem to get it to work.  Any and all ideas are welcome, and wordy explanations are preferred.  ;-)
0
Comment
Question by:kyle_in_taiwan
2 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
ID: 17987476
try: uniq original > uniqued.txt
Note: uniq only deletes duplicates that follows one by another. ie if Your file have
aaa/bbb/aaa
aaa/ccc/aaa
aaa/bbb/aaa

uniq will do nothing.
0
 
LVL 5

Author Comment

by:kyle_in_taiwan
ID: 17987675
Cool.  I've been able to wrangle a solution out of that one already.  Thanks.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
This Micro Tutorial demonstrates using Microsoft Excel pivot tables, how to reverse engineer competitors' marketing strategies through backlinks.

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now