Go Premium for a chance to win a PS4. Enter to Win


How to compare two directories and remove the difference in files

Posted on 2004-04-20
Medium Priority
Last Modified: 2010-04-20
What I want to do is this:

I have 2 directories 'image' & 'thumbnail' and I want to compare the 2 directories *recursively* so that I can find the files that are still in 'thumbnail' but not in 'image'. Please note that I just want to compare the filenames (ie doesn't matter if their sizes are different). Then I want to delete the files that are still in 'thumbnail' but not in 'image'.

At the moment I have experimented with diff command such that:

'diff -r image thumbnail'

this yields:

Binary files image/009.jpg and thumbnail/009.jpg differ
Only in thumbnail: 010.jpg
Binary files image/sunset.jpg and thumbnail/sunset.jpg differ

I only want to 'capture' the second line of the output (i.e. Only in thumbnail)  and remove (rm -rf) those file(s).

Thanx for your help in advance
Question by:bjai
  • 4
  • 4
  • 4
  • +1
LVL 45

Expert Comment

ID: 10875484
> Then I want to delete the files that are still in 'thumbnail' but not in 'image'.

find <thumbnail dir> type -f | while read name
           newname=`echo $name | sed 's/<thumbnail dir>/<image dir>/'`
           if [ ! -f "$newname" ]
                      echo $name

this will print the names of all such files ... If you wish to remove them, replace the echo $name with rm -f $name

Expert Comment

ID: 10877866
A Quicker one:

find image -type f|sed 's/^image/thumbnail/'|xargs rm  -i

rm -i ask for confirm.

Peace, R.

Assisted Solution

robipolli earned 400 total points
ID: 10878031
sorry! I misunderstood the question:
This should work

thumbnails# diff -r . ../images  |awk -F\: '/Only in \./ {print $2}'| xargs rm -i

 the sunnycoder one is ok, but you should check
<thumbnail dir> being sure it's the first word . This to avoid problems with filenames like
thumbnail/thumbnail.jpg which becomes image/image.jpg

Pax, R.
 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks


Author Comment

ID: 10879935
the one liner concept is what I am looking for but robipolli please explain the awk part, the line doesn't seem to work... It seems that I need to use the awk to extract the path names of the result of diff, please explain how the awk options and arguments work.

Author Comment

ID: 10880049
robipolli, i also tried using your command with 'xargs -print0' at the end just to check what files are being extracted but the command just echoes '/bin/echo ?...'

Assisted Solution

tolgadalkilic earned 200 total points
ID: 10884070
Let me explain :)
-F\: means ":" character is the delimiter,
 '/Only in \./ {print $2}' means it outputs the second part (oarst are seperated with delimiter ":" ) of the line that contains the string "Only in" and pipes it to remove command. The idea is basically this i guess. You can look at the manuel at:
for  awk. I suggest "awk" and "sed" together for searching and editing strings in files.

Author Comment

ID: 10885284
I have realized another problem with the diff result output, I wonder if there is an option in diff to change the output format eg delimiter => fullpath of the file or a tab (instead of ': ')  coz if a sub-directory name of the thumbnail or image contains a ':' the regular expression will extract the wrong stuff. I would like to see if fullpath format can be outputted from diff

LVL 45

Accepted Solution

sunnycoder earned 400 total points
ID: 10885803

>This to avoid problems with filenames like
>thumbnail/thumbnail.jpg which becomes image/image.jpg
not unless you specify g in the sed command :o)


>, I wonder if there is an option in diff to change the output format eg delimiter => fullpath of the
>file or a tab (instead of ': '
run it through sed

diff <> <> | sed 's/:/=>/'

Expert Comment

ID: 10886096
Hi all!
I'm in Italy so I've just read your msgs!
4 sunnycoder:
 yes! You're right! sorry!

4 bjai:
  I tested my script on linux and it works for directories having the same tree, maybe it won't fit your problem but should be syntax-correct
  you can do little about changing output format of diff, and dat little  depends on your operating system. [ man diff ]
  it would be easier  if  you explain the pattern you want to match and we'll try to write back a regexp

Peace, R.

Author Comment

ID: 10886147
I think the main problem with using diff is that diff is meant to be used for comparing file (contents). The safest way is to use the find command on <thumbnail dir> and loop thru each file to see if the same filepath occurs in <image dir>. (similar to solution of sunny code). However I wonder what will be the load on the system when there are lots of files to compare.

A final comment on the loading of the system with such method is appreciated (tho i have given out the points). :)

I think I may also look at a different direction by creating thumbnail (and image) index files that contain all the thumbnail/image file paths. Then by comparing the 2 files (now it's time to use the diff command) for different pathnames can find out which thumbnails are redundant.

Expert Comment

ID: 10886318
the load of the system during diff depends on diff options and of the size of 'almost equal' files. find should not affect too much workload if you have a fast HD.
another  approach, but the sunnycoder one seems to perform better.

1) copy  all redundant files in another directory (tar  is needed for preservig directory structure
/home/test#  mkdir new_thumb
/home/test/images# find . -type f|xargs tar cf - -C ../thumb |tar -C ../newthumb -tvf-

Peace, R
LVL 45

Expert Comment

ID: 10886493

C: Because Experts' reliability are often judged by their grading records, many Experts would like the opportunity to clarify if you have questions about their solutions. If you have given the Expert(s) ample time to respond to your clarification posts and you have responded to each of their posts providing requested information; or if the answers, after clarification, lack finality or do not completely address the issue presented, then a "C" grade is an option. You also have the option here of just asking Community Support to delete the question.

Remember, the Expert helping you today is probably going to be helping you next time you post a question. Give them a fair chance to earn an 'Excellent!' grade and they'll provide you with some amazing support. It's also true that a "C" is the lowest grade you can give, and the Experts know that -- so use it judiciously.
LVL 45

Expert Comment

ID: 10886521
>However I wonder what will be the load on the system when there are lots of files to compare.
When there are lots of files to compare, the system load will be high irrespective of the method you use ... Efficient methods may keep the duration of high load short but it cannot be avoided all together (unless you decide to do it in very small pieces)


Featured Post

Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month9 days, 14 hours left to enroll

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question