Shell Script: How can I recreate the directory tree from source on destination of SCP command?

I am writing a script to read a relative path from a text file, check to see if the file exists in the source directory, and then copy the file from source to desination. So far, I can read, check, and copy files just fine but when the files get to the destination directory, only the files are there and the directory structure is lost. Here is my script so far:

SOURCE='/c/source'
DEST='/c/dest'
YOURFILE="filestomove.txt"
while read -r path
do
if [ -f "$SOURCE$path" ]
then
scp -r "$SOURCE$path" $DEST/new/
else
echo "$SOURCE$path" >> notfound.txt
fi
done < $YOURFILE

Here are the contents of filestomove.txt:
/sub/test1.txt
/sub/test2.txt
/sub/sub 1/test3.txt
/sub/test4.txt
/sub/sub 2/test5.txt
/sub/test6.txt
EOF

Right now I end up with a destination directory containing:
test1.txt
test2.txt
test3.txt
test4.txt
test5.txt
test6.txt

I can't just do a recursive scp because I do not want to move all contents of any directory. I only want to copy specific files and they will reside in various directories.

Thanks,
Adam
iccadminAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

iccadminAuthor Commented:
Note:
On line 8 of the script, "/new/" should be removed from the end.
Sorry for the confusion.

Thanks,
Adam
0
omarfaridCommented:
I think you need to create the sub dir before you copy, and append its name to $DEST while copying.

user command

dir=`dirname "$path"`
mkdir -p $DEST/$dir
scp -r "$SOURCE$path" $DEST$dir
0
iccadminAuthor Commented:
The text file will be updated regularly with files to push. The files will sometimes reside in existing directories, and sometimes new directories. I do not want to have to update my script for each push. Would defeat the purpose of scripting. Is there a way to pull the path out of the file path specified in the text file so that it can create it if needed?
0
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

TintinCommented:
Why are you using scp to try to copy files locally?
0
iccadminAuthor Commented:
Local is for testing. I will copy remotely for production.
0
TintinCommented:
OK, that's where it was getting confusing.  Here's a version of the script that will work when you copy remotely.

#!/bin/sh
SOURCE=/c/source
DEST=/c/dest
YOURFILE=filestomove.txt
 
while read -r path
do
  if [ -f $SOURCE$path ]
  then
     dir=`dirname $path`
     ssh user@server "mkdir -p $DEST$dir"
     scp $SOURCE$path user@server:$DEST/$dir
  else
     echo "$SOURCE$path" >>notfound.txt
  fi
done < $YOURFILE

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Gabriel OrozcoSolution ArchitectCommented:
maybe its better to create the directory structure on the destination server before starting the actual copy. that way the ssh will be called only once and the script will be much more efficient.

maybe a little cosmetic changes there:
#!/bin/sh
# $FILELIST must have files in relative position to $SOURCE.
#
FILELIST=filestomove.txt
 
SRCPATH=/c/source
DSTPATH=/c/dest
DSTSRV=user@server
 
# copy the filelist
scp $FILELIST $DSTSRV:/tmp/tmpfilepath
# create the directory structure
ssh $DSTSRV "while read -r path; do dir=`dirname $path`; [ -d $DSTPATH/$dir ] || mkdir -p $DSTPATH/$dir; done < /tmp/tmpfilepath"
# do the file copying. errors will show in estandard error.
while read -r path; do
  if [ -f $SRCPATH/$path ]; then
     scp $SRCPATH/$path $DSTSRV:$DSTPATH/$path
  else
     echo "$SRCPATH/$path not found in local"
  fi
done < $FILELIST
# delete the directory structure file in destination
ssh $DSTSRV "rm /tmp/tmpfilepath"

Open in new window

0
wide_awakeCommented:
wouldn't it be easier to use rsync?
0
Gabriel OrozcoSolution ArchitectCommented:
well, I have googled for user defined lists and rsync but I was unsuccessful.

I know some commercial products that can do user defined lists (UDL) but I think they can be overkilling since a simple script can do that.
0
wide_awakeCommented:
Try the "--files-from=FILE" option to rsync -- sounds like it should do just what you want.
0
iccadminAuthor Commented:
Redimido:
The only problem I see with your solution is that your are assigning my text file to $FILELIST. When it tries to create this on the dest side, it will try to use the full path including the filename and extension, which wont work.

All-
I have done a great deal of digging, and FINALLY wrote my solution. Please see script below:

GOODLIST="/cygdrive/d/scripts/files_deployed.txt"
BADLIST="/cygdrive/d/scripts/not_found.txt"
FILESTOMOVE="/cygdrive/d/scripts/filestomove.txt"

# cd to the web root
cd /cygdrive/d/www_cmpgns

# verify that the specified files exist
for file in $(cat $FILESTOMOVE); do
  if [ ! -d "$file" -a -f "$file" ]; then
     echo "$file" >> $GOODLIST
  else
     echo "$file" >> $BADLIST
  fi
done

# PWEB21 - create tar and transfer
tar cf - $(cat $GOODLIST | xargs) | ssh cruiseadm@192.168.1.202 "tar -C /d/www/campaigns -xf -"
ssh cruiseadm@192.168.1.202 "cd /d/www/campaigns; chmod -R 0755 *;"
0
Gabriel OrozcoSolution ArchitectCommented:
great idea to use tar to circunvent the creation of files =)

and yes, my solution required you to list files as relative to the origin directory  (see comments on the script)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.