How to compare two directories and remove the difference in files

Posted on 2004-04-20
Last Modified: 2010-04-20
What I want to do is this:

I have 2 directories 'image' & 'thumbnail' and I want to compare the 2 directories *recursively* so that I can find the files that are still in 'thumbnail' but not in 'image'. Please note that I just want to compare the filenames (ie doesn't matter if their sizes are different). Then I want to delete the files that are still in 'thumbnail' but not in 'image'.

At the moment I have experimented with diff command such that:

'diff -r image thumbnail'

this yields:

Binary files image/009.jpg and thumbnail/009.jpg differ
Only in thumbnail: 010.jpg
Binary files image/sunset.jpg and thumbnail/sunset.jpg differ

I only want to 'capture' the second line of the output (i.e. Only in thumbnail)  and remove (rm -rf) those file(s).

Thanx for your help in advance
Question by:bjai
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
  • 4
  • +1
LVL 45

Expert Comment

ID: 10875484
> Then I want to delete the files that are still in 'thumbnail' but not in 'image'.

find <thumbnail dir> type -f | while read name
           newname=`echo $name | sed 's/<thumbnail dir>/<image dir>/'`
           if [ ! -f "$newname" ]
                      echo $name

this will print the names of all such files ... If you wish to remove them, replace the echo $name with rm -f $name

Expert Comment

ID: 10877866
A Quicker one:

find image -type f|sed 's/^image/thumbnail/'|xargs rm  -i

rm -i ask for confirm.

Peace, R.

Assisted Solution

robipolli earned 200 total points
ID: 10878031
sorry! I misunderstood the question:
This should work

thumbnails# diff -r . ../images  |awk -F\: '/Only in \./ {print $2}'| xargs rm -i

 the sunnycoder one is ok, but you should check
<thumbnail dir> being sure it's the first word . This to avoid problems with filenames like
thumbnail/thumbnail.jpg which becomes image/image.jpg

Pax, R.
Veeam gives away 10 full conference passes

Veeam is a VMworld 2017 US & Europe Platinum Sponsor. Enter the raffle to get the full conference pass. Pass includes the admission to all general and breakout sessions, VMware Hands-On Labs, Solutions Exchange, exclusive giveaways and the great VMworld Customer Appreciation Part


Author Comment

ID: 10879935
the one liner concept is what I am looking for but robipolli please explain the awk part, the line doesn't seem to work... It seems that I need to use the awk to extract the path names of the result of diff, please explain how the awk options and arguments work.

Author Comment

ID: 10880049
robipolli, i also tried using your command with 'xargs -print0' at the end just to check what files are being extracted but the command just echoes '/bin/echo ?...'

Assisted Solution

tolgadalkilic earned 100 total points
ID: 10884070
Let me explain :)
-F\: means ":" character is the delimiter,
 '/Only in \./ {print $2}' means it outputs the second part (oarst are seperated with delimiter ":" ) of the line that contains the string "Only in" and pipes it to remove command. The idea is basically this i guess. You can look at the manuel at:
for  awk. I suggest "awk" and "sed" together for searching and editing strings in files.

Author Comment

ID: 10885284
I have realized another problem with the diff result output, I wonder if there is an option in diff to change the output format eg delimiter => fullpath of the file or a tab (instead of ': ')  coz if a sub-directory name of the thumbnail or image contains a ':' the regular expression will extract the wrong stuff. I would like to see if fullpath format can be outputted from diff

LVL 45

Accepted Solution

sunnycoder earned 200 total points
ID: 10885803

>This to avoid problems with filenames like
>thumbnail/thumbnail.jpg which becomes image/image.jpg
not unless you specify g in the sed command :o)


>, I wonder if there is an option in diff to change the output format eg delimiter => fullpath of the
>file or a tab (instead of ': '
run it through sed

diff <> <> | sed 's/:/=>/'

Expert Comment

ID: 10886096
Hi all!
I'm in Italy so I've just read your msgs!
4 sunnycoder:
 yes! You're right! sorry!

4 bjai:
  I tested my script on linux and it works for directories having the same tree, maybe it won't fit your problem but should be syntax-correct
  you can do little about changing output format of diff, and dat little  depends on your operating system. [ man diff ]
  it would be easier  if  you explain the pattern you want to match and we'll try to write back a regexp

Peace, R.

Author Comment

ID: 10886147
I think the main problem with using diff is that diff is meant to be used for comparing file (contents). The safest way is to use the find command on <thumbnail dir> and loop thru each file to see if the same filepath occurs in <image dir>. (similar to solution of sunny code). However I wonder what will be the load on the system when there are lots of files to compare.

A final comment on the loading of the system with such method is appreciated (tho i have given out the points). :)

I think I may also look at a different direction by creating thumbnail (and image) index files that contain all the thumbnail/image file paths. Then by comparing the 2 files (now it's time to use the diff command) for different pathnames can find out which thumbnails are redundant.

Expert Comment

ID: 10886318
the load of the system during diff depends on diff options and of the size of 'almost equal' files. find should not affect too much workload if you have a fast HD.
another  approach, but the sunnycoder one seems to perform better.

1) copy  all redundant files in another directory (tar  is needed for preservig directory structure
/home/test#  mkdir new_thumb
/home/test/images# find . -type f|xargs tar cf - -C ../thumb |tar -C ../newthumb -tvf-

Peace, R
LVL 45

Expert Comment

ID: 10886493

C: Because Experts' reliability are often judged by their grading records, many Experts would like the opportunity to clarify if you have questions about their solutions. If you have given the Expert(s) ample time to respond to your clarification posts and you have responded to each of their posts providing requested information; or if the answers, after clarification, lack finality or do not completely address the issue presented, then a "C" grade is an option. You also have the option here of just asking Community Support to delete the question.

Remember, the Expert helping you today is probably going to be helping you next time you post a question. Give them a fair chance to earn an 'Excellent!' grade and they'll provide you with some amazing support. It's also true that a "C" is the lowest grade you can give, and the Experts know that -- so use it judiciously.
LVL 45

Expert Comment

ID: 10886521
>However I wonder what will be the load on the system when there are lots of files to compare.
When there are lots of files to compare, the system load will be high irrespective of the method you use ... Efficient methods may keep the duration of high load short but it cannot be avoided all together (unless you decide to do it in very small pieces)


Featured Post

Database Solutions Engineer FAQs

In this series, we will discuss common questions received as a database Solutions Engineer at Percona. In this role, we speak with a wide array of MySQL and MongoDB users responsible for both extremely large and complex environments to smaller single-server environments.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Join Greg Farro and Ethan Banks from Packet Pushers ( and Greg Ross from Paessler ( for a discussion about smart network …
Fine Tune your automatic Updates for Ubuntu / Debian
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses

635 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question