?
Solved

Need a shell script to cycle through a directory of images and copy images to diff directory based on filename.

Posted on 2007-10-15
15
Medium Priority
?
1,195 Views
Last Modified: 2008-01-09
I have a directory filled with a TON of images.  Probably over a million.  I'm setting up a system to break the images up based on their filename.  The image filenames are just numbers... 1- 500,000.  And there are two images for each.  A thumb and a full image.

So something like...

/images/1.jpg
/images/1_t.jpg
/images/2.jpg
/images/2_t.jpg
/images/567.jpg
/images/567_t.jpg
etc..

Not every number 1 - 500,00 is taken however.  Some might be missing.  So, I don't know if it's better to have the script cycle through the actual images somehow... or just to loop hard set numbers.. and if we get to a number and the file doesnt exist.. it will just skip it.

As for the directories.  I set them up in a way where  it's based on the number.  So, image 5678.jpg.  Would go into:

/new_images/5/6/7/8/5678.jpg

34.jpg would go to:

/new_images/3/4/34.jpg

450321.jpg would go to:

/new_images/4/5/0/3/2/1/450321/jpg

so the script would have to read the filename.. or just the number that we are on and then issue a copy command based on the number.  I set up the directories using this Quesion:
http://www.experts-exchange.com/Q_22885506.html

I think that about covers it.  Anyone know of the best way to do this?
0
Comment
Question by:tyleradam
15 Comments
 
LVL 48

Expert Comment

by:Tintin
ID: 20081187
The following will work, but won't be blindingly fast (although you only need to run it once)

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file|sed 's#\(.\)#\1/#g'`
  mkdir -p /newimages/$dir
  mv $file /newimages/$dir
done
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20081197
Just a thought, what about the thumbnail images?

Should

/images/567_t.jpg

end up in

/newimages/5/6/7/_t/567_t.jpg

or

/newimages/5/6/7/567_t.jpg

?
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 20081373
Hi,

Tintin:

If you allow me, I will add to your script since it will "digitize" the file extension as well, which is not desired:

find . -name "*.jpg" | while read file
do
  dir=`echo $file | awk -F"." '{print $1}' | sed 's#\(.\)#\1/#g'`
  mkdir -p /newimages/$dir
  mv $file /newimages/$dir
done
0
 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks

 

Author Comment

by:tyleradam
ID: 20081376
The second way:

/newimages/5/6/7/567_t.jpg
0
 

Author Comment

by:tyleradam
ID: 20081391
With that script.. I see that it is making a directory.  It's making a directory based off the filename?  Can I skip that part if I already have the directories created?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20081432


Change

mkdir -p /newimages/$dir

to

[ -d /newimages/$dir ] || mkdir -p /newimages/$dir

or simply

mkdir -p /newimages/$dir 2>/dev/null

Putting the whole thing together with the thumbnails going in the same dir as the full sized image:

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  mkdir -p /newimages/$dir 2>/dev/null
  mv $file /newimages/$dir
done


The 2nd option will be a little quicker.

0
 

Author Comment

by:tyleradam
ID: 20082094
For this one:
#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  mkdir -p /newimages/$dir 2>/dev/null
  mv $file /newimages/$dir
done

I commented out the lines to mkdir and mv.  And added a line to echo 'dir' just to see what's going on.  And it cycles through... but it's not echoing anything.  Is that okay?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20082182
So you're saying you have:

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  echo $dir
done

echo Finished.

0
 

Author Comment

by:tyleradam
ID: 20082192
Correct
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20082224
Does the script finish without displaying anything?

Remember that it will take a while for it to run as the find command will take some time.
0
 

Author Comment

by:tyleradam
ID: 20082235
I actually did something like this:


#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  echo "Go "
  echo $dir
done



and it displays a lot of Go 's followed by a blank new line after each one.  So I figure it's cycling through.  But its not echoing the $dir.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20082610
when you put a

echo $file

in the loop, what do you get?
0
 

Author Comment

by:tyleradam
ID: 20082628
It lists the Go line and the filename and then a blank line...


./354185_m.jpg
Go

./356068_t.jpg
Go

./526306_t.jpg
Go

./513079_m.jpg
Go

./416399_xt.jpg
Go

./351447_t.jpg
Go

./2848_m.jpg
Go

./539100_xt.jpg
Go

./539409_m.jpg
Go

./24464_t.jpg
Go
0
 
LVL 48

Accepted Solution

by:
Tintin earned 2000 total points
ID: 20082640
Sorry, my mistake.  I forgot about the dot slash at the beginning of the file names.

Assuming the original images are all in /images, ie: no subdirs, change

dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`

to

dir=`basename $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
0
 
LVL 9

Expert Comment

by:ghostdog74
ID: 20082685

#!/bin/sh
if [ ! -d newimages ];then
     mkdir newimages
fi
for files in *jpg
do
     num=${files%%.jpg} #get number in front of .jpg
     if [ -f "${num}_t.jpg" -a -f "$files"  ];then
        dir2make=`echo "$num" | awk 'BEGIN{FS=""}{for(i=1;i<=NF;i++)printf $i"/"}'`
        mkdir -p "newimages/$dir2make" > /dev/null 2>&1
        cp $files newimages/$dir2make     #change to mv as necessary
        #copy thumbs.
        cp "${num}_t.jpg" newimages/$dir2make         #change to mv as necessary
     fi
done
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Fine Tune your automatic Updates for Ubuntu / Debian
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
The viewer will learn how to count occurrences of each item in an array.
Suggested Courses
Course of the Month14 days, 17 hours left to enroll

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question