Solved

Easy-peasy:  Script to delete duplicate lines in a file

Posted on 2006-11-21
2
982 Views
Last Modified: 2012-05-05
This should be really simple, i know, but i'm quite new to scripting and i just don't have any idea how to do it.

Imagine a process that returns a list of paths, with the form:

aaa/bbb/ccc
aaa/bbb/ccc
aaa/bbb/ccc
ddd/eee/fff
ddd/eee/fff
ddd/eee/fff
bbb/ccc/ddd
bbb/ccc/ddd
.
.
.

what script will reduce that to a file that just has:

aaa/bbb/ccc
ddd/eee/fff
bbb/ccc/ddd

and nothing else?

Currrently i've tried this:

# FileList is the file that's being created.

: > FileList
<first process> |
while read "b"
        do
        c=`dirname "$b"`
        d=`sed -n '/"$c"/p' FileList`
        if [ "$d" = "$c" ]
                then
                        continue
                else
                        echo "$c" >> List
                fi
        done

I've tried a few other variations, all based around this theme, but can't seem to get it to work.  Any and all ideas are welcome, and wordy explanations are preferred.  ;-)
0
Comment
Question by:kyle_in_taiwan
2 Comments
 
LVL 43

Accepted Solution

by:
ravenpl earned 500 total points
ID: 17987476
try: uniq original > uniqued.txt
Note: uniq only deletes duplicates that follows one by another. ie if Your file have
aaa/bbb/aaa
aaa/ccc/aaa
aaa/bbb/aaa

uniq will do nothing.
0
 
LVL 5

Author Comment

by:kyle_in_taiwan
ID: 17987675
Cool.  I've been able to wrangle a solution out of that one already.  Thanks.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now