Link to home
Create AccountLog in
Avatar of weklica
weklica

asked on

Copy files WITH verification process for .sh script

I have a directory with many sub-directories.  Each sub-directory contains thousands of files (for a total of 3.5 million).  I want to move each file to a single destination directory (remove from sub-directories).  However, a process runs on files received so I don't want to dump millions of files at once.  I would prefer that it move files from the source to the destination in a bit of a controlled fashion as follows:

Instead of just blasting them through and waiting for 10 seconds to loop as indicated in the script I pasted below, is it possible to have a file move and not have the next file move until the last one is gone from the new destination folder?  The destination folder is a cache folder.  The files there will not remain very long.  Once processed, they are moved again.  So, I would like to move each file from /Users/Shared/Source to /Users/usrnm/Documents/Test as indicated in the script below, but I don't want it to move each next file until it verifies that the file it just moved into /Users/usrnm/Documents/Test is no longer present.  So, sort of a check and balance control.  move all files from A to B, but don't proceed until each last file disappears from B.

Thanks in advance!
#!/bin/sh
while :
do
cd "/Users/Shared/SOURCE"
sudo mv * /Users/usrnm/Documents/Test
 sleep 10
done

Open in new window

Avatar of wilcoxon
wilcoxon
Flag of United States of America image

You talk about many sub-directories but your script only appears to move files from one (/Users/Shared/SOURCE).  Where do the many sub-directories come in?

This should work...
#!/bin/csh -f
# can be done in other shells - I just find the csh syntax easier to remember

setenv prev '_does_not_exist_'

# set for now since looping not in original
setenv dir /Users/Shared/SOURCE
# loop over sub-directories
#foreach dir (`find /some/path -type d -prune -print`)
    # loop over files
    foreach fil ($dir/*)
        # wait for last file to be gone
        while (-f $prev)
            sleep 5
        end
        mv $dir/$fil /Users/usrnm/Documents/Test/.
        setenv $prev $dir/$fil
    end
#end

Open in new window

SOLUTION
Avatar of Steven Carnahan
Steven Carnahan
Flag of United States of America image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
@wilcoxon:   You beat me to it.  :)   I didn't account for the original comment about multiple source directories.
I did notice two minor issues in my script (1) forgot to sudo the mv, 2) find in commented out loop over dirs) so may as well fix them before the author tries it.
#!/bin/csh -f
# can be done in other shells - I just find the csh syntax easier to remember

setenv prev '_does_not_exist_'

# set for now since looping not in original
setenv dir /Users/Shared/SOURCE
# loop over sub-directories
# added * to path - otherwise -prune will stop on parent dir
#foreach dir (`find /some/path/* -type d -prune -print`)
    # loop over files
    foreach fil ($dir/*)
        # wait for last file to be gone
        while (-f $prev)
            sleep 5
        end
        sudo mv $dir/$fil /Users/usrnm/Documents/Test/.
        setenv $prev $dir/$fil
    end
#end

Open in new window

ASKER CERTIFIED SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of weklica
weklica

ASKER

Sorry!  Forgot to close the question out.  Very helpful!!