Need a shell script to cycle through a directory of images and copy images to diff directory based on filename.

I have a directory filled with a TON of images.  Probably over a million.  I'm setting up a system to break the images up based on their filename.  The image filenames are just numbers... 1- 500,000.  And there are two images for each.  A thumb and a full image.

So something like...

/images/1.jpg
/images/1_t.jpg
/images/2.jpg
/images/2_t.jpg
/images/567.jpg
/images/567_t.jpg
etc..

Not every number 1 - 500,00 is taken however.  Some might be missing.  So, I don't know if it's better to have the script cycle through the actual images somehow... or just to loop hard set numbers.. and if we get to a number and the file doesnt exist.. it will just skip it.

As for the directories.  I set them up in a way where  it's based on the number.  So, image 5678.jpg.  Would go into:

/new_images/5/6/7/8/5678.jpg

34.jpg would go to:

/new_images/3/4/34.jpg

450321.jpg would go to:

/new_images/4/5/0/3/2/1/450321/jpg

so the script would have to read the filename.. or just the number that we are on and then issue a copy command based on the number.  I set up the directories using this Quesion:
http://www.experts-exchange.com/Q_22885506.html

I think that about covers it.  Anyone know of the best way to do this?
tyleradamAsked:
Who is Participating?
 
TintinCommented:
Sorry, my mistake.  I forgot about the dot slash at the beginning of the file names.

Assuming the original images are all in /images, ie: no subdirs, change

dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`

to

dir=`basename $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
0
 
TintinCommented:
The following will work, but won't be blindingly fast (although you only need to run it once)

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file|sed 's#\(.\)#\1/#g'`
  mkdir -p /newimages/$dir
  mv $file /newimages/$dir
done
0
 
TintinCommented:
Just a thought, what about the thumbnail images?

Should

/images/567_t.jpg

end up in

/newimages/5/6/7/_t/567_t.jpg

or

/newimages/5/6/7/567_t.jpg

?
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
omarfaridCommented:
Hi,

Tintin:

If you allow me, I will add to your script since it will "digitize" the file extension as well, which is not desired:

find . -name "*.jpg" | while read file
do
  dir=`echo $file | awk -F"." '{print $1}' | sed 's#\(.\)#\1/#g'`
  mkdir -p /newimages/$dir
  mv $file /newimages/$dir
done
0
 
tyleradamAuthor Commented:
The second way:

/newimages/5/6/7/567_t.jpg
0
 
tyleradamAuthor Commented:
With that script.. I see that it is making a directory.  It's making a directory based off the filename?  Can I skip that part if I already have the directories created?
0
 
TintinCommented:


Change

mkdir -p /newimages/$dir

to

[ -d /newimages/$dir ] || mkdir -p /newimages/$dir

or simply

mkdir -p /newimages/$dir 2>/dev/null

Putting the whole thing together with the thumbnails going in the same dir as the full sized image:

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  mkdir -p /newimages/$dir 2>/dev/null
  mv $file /newimages/$dir
done


The 2nd option will be a little quicker.

0
 
tyleradamAuthor Commented:
For this one:
#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  mkdir -p /newimages/$dir 2>/dev/null
  mv $file /newimages/$dir
done

I commented out the lines to mkdir and mv.  And added a line to echo 'dir' just to see what's going on.  And it cycles through... but it's not echoing anything.  Is that okay?
0
 
TintinCommented:
So you're saying you have:

#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  echo $dir
done

echo Finished.

0
 
tyleradamAuthor Commented:
Correct
0
 
TintinCommented:
Does the script finish without displaying anything?

Remember that it will take a while for it to run as the find command will take some time.
0
 
tyleradamAuthor Commented:
I actually did something like this:


#!/bin/bash
cd /images

find . -name "*.jpg" | while read file
do
  dir=`echo $file | sed -e 's#\([0-9]\)#\1/#g' -e 's/[_\.].*//'`
  echo "Go "
  echo $dir
done



and it displays a lot of Go 's followed by a blank new line after each one.  So I figure it's cycling through.  But its not echoing the $dir.
0
 
TintinCommented:
when you put a

echo $file

in the loop, what do you get?
0
 
tyleradamAuthor Commented:
It lists the Go line and the filename and then a blank line...


./354185_m.jpg
Go

./356068_t.jpg
Go

./526306_t.jpg
Go

./513079_m.jpg
Go

./416399_xt.jpg
Go

./351447_t.jpg
Go

./2848_m.jpg
Go

./539100_xt.jpg
Go

./539409_m.jpg
Go

./24464_t.jpg
Go
0
 
ghostdog74Commented:

#!/bin/sh
if [ ! -d newimages ];then
     mkdir newimages
fi
for files in *jpg
do
     num=${files%%.jpg} #get number in front of .jpg
     if [ -f "${num}_t.jpg" -a -f "$files"  ];then
        dir2make=`echo "$num" | awk 'BEGIN{FS=""}{for(i=1;i<=NF;i++)printf $i"/"}'`
        mkdir -p "newimages/$dir2make" > /dev/null 2>&1
        cp $files newimages/$dir2make     #change to mv as necessary
        #copy thumbs.
        cp "${num}_t.jpg" newimages/$dir2make         #change to mv as necessary
     fi
done
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.