We help IT Professionals succeed at work.

Shell Script: How can I recreate the directory tree from source on destination of SCP command?

Medium Priority
1,128 Views
Last Modified: 2012-05-05
I am writing a script to read a relative path from a text file, check to see if the file exists in the source directory, and then copy the file from source to desination. So far, I can read, check, and copy files just fine but when the files get to the destination directory, only the files are there and the directory structure is lost. Here is my script so far:

SOURCE='/c/source'
DEST='/c/dest'
YOURFILE="filestomove.txt"
while read -r path
do
if [ -f "$SOURCE$path" ]
then
scp -r "$SOURCE$path" $DEST/new/
else
echo "$SOURCE$path" >> notfound.txt
fi
done < $YOURFILE

Here are the contents of filestomove.txt:
/sub/test1.txt
/sub/test2.txt
/sub/sub 1/test3.txt
/sub/test4.txt
/sub/sub 2/test5.txt
/sub/test6.txt
EOF

Right now I end up with a destination directory containing:
test1.txt
test2.txt
test3.txt
test4.txt
test5.txt
test6.txt

I can't just do a recursive scp because I do not want to move all contents of any directory. I only want to copy specific files and they will reside in various directories.

Thanks,
Adam
Comment
Watch Question

Author

Commented:
Note:
On line 8 of the script, "/new/" should be removed from the end.
Sorry for the confusion.

Thanks,
Adam
CERTIFIED EXPERT
Top Expert 2007

Commented:
I think you need to create the sub dir before you copy, and append its name to $DEST while copying.

user command

dir=`dirname "$path"`
mkdir -p $DEST/$dir
scp -r "$SOURCE$path" $DEST$dir

Author

Commented:
The text file will be updated regularly with files to push. The files will sometimes reside in existing directories, and sometimes new directories. I do not want to have to update my script for each push. Would defeat the purpose of scripting. Is there a way to pull the path out of the file path specified in the text file so that it can create it if needed?
CERTIFIED EXPERT
Top Expert 2007

Commented:
Why are you using scp to try to copy files locally?

Author

Commented:
Local is for testing. I will copy remotely for production.
CERTIFIED EXPERT
Top Expert 2007
Commented:
OK, that's where it was getting confusing.  Here's a version of the script that will work when you copy remotely.

#!/bin/sh
SOURCE=/c/source
DEST=/c/dest
YOURFILE=filestomove.txt
 
while read -r path
do
  if [ -f $SOURCE$path ]
  then
     dir=`dirname $path`
     ssh user@server "mkdir -p $DEST$dir"
     scp $SOURCE$path user@server:$DEST/$dir
  else
     echo "$SOURCE$path" >>notfound.txt
  fi
done < $YOURFILE

Open in new window

Gabriel OrozcoSolution Architect
Commented:
maybe its better to create the directory structure on the destination server before starting the actual copy. that way the ssh will be called only once and the script will be much more efficient.

maybe a little cosmetic changes there:
#!/bin/sh
# $FILELIST must have files in relative position to $SOURCE.
#
FILELIST=filestomove.txt
 
SRCPATH=/c/source
DSTPATH=/c/dest
DSTSRV=user@server
 
# copy the filelist
scp $FILELIST $DSTSRV:/tmp/tmpfilepath
# create the directory structure
ssh $DSTSRV "while read -r path; do dir=`dirname $path`; [ -d $DSTPATH/$dir ] || mkdir -p $DSTPATH/$dir; done < /tmp/tmpfilepath"
# do the file copying. errors will show in estandard error.
while read -r path; do
  if [ -f $SRCPATH/$path ]; then
     scp $SRCPATH/$path $DSTSRV:$DSTPATH/$path
  else
     echo "$SRCPATH/$path not found in local"
  fi
done < $FILELIST
# delete the directory structure file in destination
ssh $DSTSRV "rm /tmp/tmpfilepath"

Open in new window

wouldn't it be easier to use rsync?
Gabriel OrozcoSolution Architect

Commented:
well, I have googled for user defined lists and rsync but I was unsuccessful.

I know some commercial products that can do user defined lists (UDL) but I think they can be overkilling since a simple script can do that.
Try the "--files-from=FILE" option to rsync -- sounds like it should do just what you want.

Author

Commented:
Redimido:
The only problem I see with your solution is that your are assigning my text file to $FILELIST. When it tries to create this on the dest side, it will try to use the full path including the filename and extension, which wont work.

All-
I have done a great deal of digging, and FINALLY wrote my solution. Please see script below:

GOODLIST="/cygdrive/d/scripts/files_deployed.txt"
BADLIST="/cygdrive/d/scripts/not_found.txt"
FILESTOMOVE="/cygdrive/d/scripts/filestomove.txt"

# cd to the web root
cd /cygdrive/d/www_cmpgns

# verify that the specified files exist
for file in $(cat $FILESTOMOVE); do
  if [ ! -d "$file" -a -f "$file" ]; then
     echo "$file" >> $GOODLIST
  else
     echo "$file" >> $BADLIST
  fi
done

# PWEB21 - create tar and transfer
tar cf - $(cat $GOODLIST | xargs) | ssh cruiseadm@192.168.1.202 "tar -C /d/www/campaigns -xf -"
ssh cruiseadm@192.168.1.202 "cd /d/www/campaigns; chmod -R 0755 *;"
Gabriel OrozcoSolution Architect
Commented:
great idea to use tar to circunvent the creation of files =)

and yes, my solution required you to list files as relative to the origin directory  (see comments on the script)

Explore More ContentExplore courses, solutions, and other research materials related to this topic.