asked on

Copy files WITH verification process for .sh script

I have a directory with many sub-directories. Each sub-directory contains thousands of files (for a total of 3.5 million). I want to move each file to a single destination directory (remove from sub-directories). However, a process runs on files received so I don't want to dump millions of files at once. I would prefer that it move files from the source to the destination in a bit of a controlled fashion as follows:

Instead of just blasting them through and waiting for 10 seconds to loop as indicated in the script I pasted below, is it possible to have a file move and not have the next file move until the last one is gone from the new destination folder? The destination folder is a cache folder. The files there will not remain very long. Once processed, they are moved again. So, I would like to move each file from /Users/Shared/Source to /Users/usrnm/Documents/Test as indicated in the script below, but I don't want it to move each next file until it verifies that the file it just moved into /Users/usrnm/Documents/Test is no longer present. So, sort of a check and balance control. move all files from A to B, but don't proceed until each last file disappears from B.

Thanks in advance!

#!/bin/sh
while :
do
cd "/Users/Shared/SOURCE"
sudo mv * /Users/usrnm/Documents/Test
 sleep 10
done

Open in new window

wilcoxon

You talk about many sub-directories but your script only appears to move files from one (/Users/Shared/SOURCE). Where do the many sub-directories come in?

This should work...

#!/bin/csh -f
# can be done in other shells - I just find the csh syntax easier to remember

setenv prev '_does_not_exist_'

# set for now since looping not in original
setenv dir /Users/Shared/SOURCE
# loop over sub-directories
#foreach dir (`find /some/path -type d -prune -print`)
    # loop over files
    foreach fil ($dir/*)
        # wait for last file to be gone
        while (-f $prev)
            sleep 5
        end
        mv $dir/$fil /Users/usrnm/Documents/Test/.
        setenv $prev $dir/$fil
    end
#end

Open in new window

SOLUTION

Steven Carnahan

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Steven Carnahan

@wilcoxon: You beat me to it. :) I didn't account for the original comment about multiple source directories.

wilcoxon

I did notice two minor issues in my script (1) forgot to sudo the mv, 2) find in commented out loop over dirs) so may as well fix them before the author tries it.

#!/bin/csh -f
# can be done in other shells - I just find the csh syntax easier to remember

setenv prev '_does_not_exist_'

# set for now since looping not in original
setenv dir /Users/Shared/SOURCE
# loop over sub-directories
# added * to path - otherwise -prune will stop on parent dir
#foreach dir (`find /some/path/* -type d -prune -print`)
    # loop over files
    foreach fil ($dir/*)
        # wait for last file to be gone
        while (-f $prev)
            sleep 5
        end
        sudo mv $dir/$fil /Users/usrnm/Documents/Test/.
        setenv $prev $dir/$fil
    end
#end

Open in new window

ASKER CERTIFIED SOLUTION

wilcoxon

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

weklica

ASKER

Sorry! Forgot to close the question out. Very helpful!!