Shell Script: How can I recreate the directory tree from source on destination of SCP command?

I am writing a script to read a relative path from a text file, check to see if the file exists in the source directory, and then copy the file from source to desination. So far, I can read, check, and copy files just fine but when the files get to the destination directory, only the files are there and the directory structure is lost. Here is my script so far:

SOURCE='/c/source'
DEST='/c/dest'
YOURFILE="filestomove.txt"
while read -r path
do
if [ -f "$SOURCE$path" ]
then
scp -r "$SOURCE$path" $DEST/new/
else
echo "$SOURCE$path" >> notfound.txt
fi
done < $YOURFILE

Here are the contents of filestomove.txt:
/sub/test1.txt
/sub/test2.txt
/sub/sub 1/test3.txt
/sub/test4.txt
/sub/sub 2/test5.txt
/sub/test6.txt
EOF

Right now I end up with a destination directory containing:
test1.txt
test2.txt
test3.txt
test4.txt
test5.txt
test6.txt

I can't just do a recursive scp because I do not want to move all contents of any directory. I only want to copy specific files and they will reside in various directories.

Thanks,
Adam
iccadminAsked:
Who is Participating?
 
TintinConnect With a Mentor Commented:
OK, that's where it was getting confusing.  Here's a version of the script that will work when you copy remotely.

#!/bin/sh
SOURCE=/c/source
DEST=/c/dest
YOURFILE=filestomove.txt
 
while read -r path
do
  if [ -f $SOURCE$path ]
  then
     dir=`dirname $path`
     ssh user@server "mkdir -p $DEST$dir"
     scp $SOURCE$path user@server:$DEST/$dir
  else
     echo "$SOURCE$path" >>notfound.txt
  fi
done < $YOURFILE

Open in new window

0
 
iccadminAuthor Commented:
Note:
On line 8 of the script, "/new/" should be removed from the end.
Sorry for the confusion.

Thanks,
Adam
0
 
omarfaridCommented:
I think you need to create the sub dir before you copy, and append its name to $DEST while copying.

user command

dir=`dirname "$path"`
mkdir -p $DEST/$dir
scp -r "$SOURCE$path" $DEST$dir
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
iccadminAuthor Commented:
The text file will be updated regularly with files to push. The files will sometimes reside in existing directories, and sometimes new directories. I do not want to have to update my script for each push. Would defeat the purpose of scripting. Is there a way to pull the path out of the file path specified in the text file so that it can create it if needed?
0
 
TintinCommented:
Why are you using scp to try to copy files locally?
0
 
iccadminAuthor Commented:
Local is for testing. I will copy remotely for production.
0
 
Gabriel OrozcoConnect With a Mentor Solution ArchitectCommented:
maybe its better to create the directory structure on the destination server before starting the actual copy. that way the ssh will be called only once and the script will be much more efficient.

maybe a little cosmetic changes there:
#!/bin/sh
# $FILELIST must have files in relative position to $SOURCE.
#
FILELIST=filestomove.txt
 
SRCPATH=/c/source
DSTPATH=/c/dest
DSTSRV=user@server
 
# copy the filelist
scp $FILELIST $DSTSRV:/tmp/tmpfilepath
# create the directory structure
ssh $DSTSRV "while read -r path; do dir=`dirname $path`; [ -d $DSTPATH/$dir ] || mkdir -p $DSTPATH/$dir; done < /tmp/tmpfilepath"
# do the file copying. errors will show in estandard error.
while read -r path; do
  if [ -f $SRCPATH/$path ]; then
     scp $SRCPATH/$path $DSTSRV:$DSTPATH/$path
  else
     echo "$SRCPATH/$path not found in local"
  fi
done < $FILELIST
# delete the directory structure file in destination
ssh $DSTSRV "rm /tmp/tmpfilepath"

Open in new window

0
 
wide_awakeConnect With a Mentor Commented:
wouldn't it be easier to use rsync?
0
 
Gabriel OrozcoSolution ArchitectCommented:
well, I have googled for user defined lists and rsync but I was unsuccessful.

I know some commercial products that can do user defined lists (UDL) but I think they can be overkilling since a simple script can do that.
0
 
wide_awakeConnect With a Mentor Commented:
Try the "--files-from=FILE" option to rsync -- sounds like it should do just what you want.
0
 
iccadminAuthor Commented:
Redimido:
The only problem I see with your solution is that your are assigning my text file to $FILELIST. When it tries to create this on the dest side, it will try to use the full path including the filename and extension, which wont work.

All-
I have done a great deal of digging, and FINALLY wrote my solution. Please see script below:

GOODLIST="/cygdrive/d/scripts/files_deployed.txt"
BADLIST="/cygdrive/d/scripts/not_found.txt"
FILESTOMOVE="/cygdrive/d/scripts/filestomove.txt"

# cd to the web root
cd /cygdrive/d/www_cmpgns

# verify that the specified files exist
for file in $(cat $FILESTOMOVE); do
  if [ ! -d "$file" -a -f "$file" ]; then
     echo "$file" >> $GOODLIST
  else
     echo "$file" >> $BADLIST
  fi
done

# PWEB21 - create tar and transfer
tar cf - $(cat $GOODLIST | xargs) | ssh cruiseadm@192.168.1.202 "tar -C /d/www/campaigns -xf -"
ssh cruiseadm@192.168.1.202 "cd /d/www/campaigns; chmod -R 0755 *;"
0
 
Gabriel OrozcoConnect With a Mentor Solution ArchitectCommented:
great idea to use tar to circunvent the creation of files =)

and yes, my solution required you to list files as relative to the origin directory  (see comments on the script)
0
All Courses

From novice to tech pro — start learning today.