?
Solved

How to write a backup script for two alternating USB drives?

Posted on 2008-11-19
9
Medium Priority
?
1,088 Views
Last Modified: 2013-12-16
I want to use two USB drives with FAT32 filesystems to backup data, but I have a problem with mounting and dismounting the drives and I am not sure how to fix it.

Attached is my first attempt at a backup script.
I think my problem stems from several areas:
1.  Variable declaration $BACKUPDEVICE=/dev/sdc1
2.  Mounting the $BACKUPDEVICE
3.  I think I need to prevent the system from automounting the USB disks to the /media folder so that my script can do it's own mount and umount to the /mnt/usbdrive mount point for the backup job only.

Here is the current mount list:
/dev/md0 on / type reiserfs (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/sda1 on /boot type ext3 (rw,acl,user_xattr)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sdc1 on /media/usb-160500022Dc5:0:0:0p1 type subfs (rw,noexec,nosuid,nodev,sync,procuid,iocharset=utf8) <=== a double up
/dev/sdd1 on /media/usb-160500022Dc5:0:0:0p1 type subfs (rw,noexec,nosuid,nodev,sync,procuid,iocharset=utf8) <=== a double up
> /dev/sdc1 and /dev/sdd1 are mounted to the same moutpoint!!?

Note from the above that nothing is mounted at the $BACKUPMOUNTPOINT and yet the backups have been running to the $BACKUPMOUNTPOINT and fill the disk!! :-\

What is the best way for me to mount these USB file systems to my $BACKUPMOUNTPOINT with consistency to that it can run in an automatic fashion.  I'd like to be able to simply unplug one disk each morning and replace without all this manual intervention.
Also if it is related, what checks could be written for the second paragraph of my script that may assist help with this problem?

Cheers for anyones help.
#!/bin/sh
#VARIABLES
 DAYOFWEEK=`date +%a`
 BACKUPDEVICE=/dev/sdc1 #FAT32 USB drive
 BACKUPMOUNTPOINT=/mnt/usbdrive
 BACKUPFOLDER=$BACKUPMOUNTPOINT/backup/$DAYOFWEEK
 BACKUPLOG=/home/$DAYOFWEEK-backup.log
#SUMMARY-BACKUPLOG
 
# Checks
# Is device available for mounting? If not write to $BACKUPLOG
# Is backup disk device already mounted?
 
# Mount the USB disk .
#mount [-fnrsvw] [-o options [,...]] device | dir
 mount -o shortname=winnt $BACKUPDEVICE $BACKUPMOUNTPOINT
 
#Ensure backup folder exists
 mkdir -p $BACKUPFOLDER
 
# Backup data
# /home, /etc ,/var and /root to the FAT32 external USB drive.
# rsync
# -v   verbose output
# -r   recurse into directories
# -l   copy symlinks as symlinks
# -t   preserve times
# --delete delete destination files that don't exist at source
# --modify-window=1 for fat32 compatibility
# no -og options to avoid chown errors
 
 echo "### Backup- /home/data ###"                                        >> $BACKUPLOG
 rsync -vrlt --stats --delete --modify-window=1 /home/data  $BACKUPFOLDER  > $BACKUPLOG
#tail to summary backup log
 echo "### End /home/data ###"                                            >> $BACKUPLOG
 echo ""                                                                  >> $BACKUPLOG
 
cp /home/$DAYOFWEEK-backup.log $BACKUPMOUNTPOINT/
 
# Unmount the backup disk.
 umount $BACKUPDEVICE

Open in new window

0
Comment
Question by:blokeman
  • 5
  • 4
9 Comments
 
LVL 21

Accepted Solution

by:
Daniel McAllister earned 2000 total points
ID: 22997145
First of all, you need to add error-checking to your script... that's why you're filling up your / drive! (Your mount is failing, but the script continues to run).

The simplest test to see if a command ran successfully is to check its RETURN CODE. In *nix, the return code is 0 if things went as planned, anything else is an error code. The shell variable you'll use is $?

So, your error checking could look like:

--- Start code example ---
mount -o shortname=winnt $BACKUPDEVICE $BACKUPMOUNTPOINT
ECODE=$?
 if [ $ECODE != 0 ] ; then
  echo "$0 ERROR: Mount failed on devvice $BACAKUPDEVICE to location $BACKUPMOUNTPOINT 1>&2
  echo " - mount encountered error # $ECODE, and $0 is unable to continue" 1>&2
  exit $ECODE
fi
--- stop code example ---

A few items to cover here:
 1) as noted above, the $? is the return code of the most previous command -- but it gets reset after EACH command, so the FIRST thing I do is store that code into the variable ECODE
 2) I check if ECODE is 0 -- if it IS zero, then we skip to the next step, if it is not, we do the inside of the IF
 3) the ECHO commands are self-explanatory -- except for the 1>&2 part -- which is directing the OUTPUT of the echo command to STDERR instead of STDOUT
 4) Lastly, I exit with a non-zero return code (so a calling shell will know I failed). In this case, I used the mount error code, but I could have used ANYTHING other than zero

Now... there is a FASTER way, but it's not as READABLE or straightforward -- and you'll lose the error code from mount:

--- Start code example ---
if ! mount -o shortname=winnt $BACKUPDEVICE $BACKUPMOUNTPOINT ; then
  echo "$0 ERROR: Mount failed on devvice $BACAKUPDEVICE to location $BACKUPMOUNTPOINT 1>&2
  exit 1
fi
--- Stop code example ---

The key here is that we let IF just use the return code of the mount command, rather than doing a comparison. However, in doing so the value of $? becomes the return of the IF command -- which will be 0 regardless of whether the mount was successful or not.

The same principle needs to be applied throughout your script to check for failed steps.

OH... one more thing -- you need an extra > on your rsync line -- you're blowing away the "old" backup log file at that point.

I hope this helps!!

Dan
IT4SOHO
0
 

Author Comment

by:blokeman
ID: 23000561
That error checking looks like it could be handy.   But how do I exit the cron job script if ECODE !=0?

Also whenever the USB disks are swapped, each day, the Suse Linux Enterprise 9 server keeps mounting them to /media automatically.  As shown below...

/dev/sdc1 on /media/usb-160500022Dc5:0:0:0p1 type subfs (rw,noexec,nosuid,nodev,sync,procuid,iocharset=utf8) <=== a double up
/dev/sdd1 on /media/usb-160500022Dc5:0:0:0p1 type subfs (rw,noexec,nosuid,nodev,sync,procuid,iocharset=utf8) <=== a double up

Strangely too  /dev/sdc1 and /dev/sdd1 are mounted to the same moutpoint.  This could prevent issues because my script only deals with /dev/sdc1.
Can my USB disks always mount to sdc1 so my script cannot fail to find them because it looks like at some point when swapping USB disks, the device was /dev/sdd1.  I read that you can't label FAT32 partitions like EXT3 ones, but is there another way around this?
Cheers
Blokeman
0
 
LVL 21

Expert Comment

by:Daniel McAllister
ID: 23014405
First of all, I would think you'd rather use the script to FIND the auto-mounted USB device (try the df or mount commands)... but if you insist on manually mounting plug-in devices, simply stop the udevmonitor service. (It'll need to be stopped BEFORE you plug in the USB drive!)

One thing I did for a client who wanted to rotate hot-swap SATA drives for a daily backup was to place 2 files in the root of each disk -- a LABEL file & a BACKUP folder. Then I could look for mounted drives that had a /LABEL file & compare the value within to the expected value (actually, we didn't care about an order except that it couldn't be the same as the PREVIOUS one, which was recorded onto the system logs).

Here's how it went for me --

--- begin snippet --- NOTE: Edited to be more readable, but less efficient
FOUNDFS=               # clear variable name
df -h | grep "^/dev" | # get a list mounted filesystems from /dev devices
  awk '{print $6}' |     # take the 6th field (the mountpoint)
  while read FS ; do # loop over each one
     if [ -f "$FS ] ; then FOUNDFS=$FS ; fi
  done

if [ -z $FOUNDFS ] ; then
  echo "$0 error: no appropriate filesystem found" 1>&2
  exit 1
fi

OLDLABEL=`grep LABEL $BACKUPLOGFILE |  tail -1 | sed 's/LABEL=//'`
NEWLABEL=`tail -1 "$FOUNDFS/LABEL"`

if [ "$OLDLABEL" == "$NEWLABEL" ] ; then
  echo "$0 error: backing up to same device as last time" 1>&2
  echo "$0 aborting" 1>&2
  exit 2
fi

# Found a drive with an acceptable label -- let's do our backup
echo "$0 starting backup to $FOUNDFS at `date`" | tee >> $BACKUPLOGFILE
echo "LABEL=$NEWLABEL" >> $BACKUPLOGFILE
--- end snippet ---

I'll say again -- the scriptlet above is written for READABILITY, not execution speed. PLEASE: no comments about making it faster!

I hope this helps...

Dan
IT4SOHO
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:blokeman
ID: 23034273
Hey Dan

<gasp>
I am a little overwhelmed by your 'expert' response.  Makes my script seem simple!
Don't worry, I won't comment on how to make it faster! :-D
I obviously have a lot to learn about scripting.

In you script, when you do a check like this one
if [ -z $FOUNDFS ] ; then
  echo "$0 error: no appropriate filesystem found" 1>&2
  exit 1
fi

Will 'exit 1' quit your script at that point? I thought that this would be good for my testing of whethe a USB drive is connected and if not abort my backup script.

And what is 1>&2?

Also out of curiosity, I have heard of AWK and SED.  Are they worth learning and using in bash scripts or can I do everything I need with the standard bash commands.

Cheers
Blokeman
0
 
LVL 21

Expert Comment

by:Daniel McAllister
ID: 23042761
Blokeman,

OK, first, with regards to the FOUNDFS and the -z:
  Note that I first CLEARED the value of FOUNDFS (just in case), then I only SET the value of FOUNDFS *IF* I actually find the "file LABEL on it. (NOTE: You may want something a little more -- like BACKUPLABEL or something -- to make sure there isn't an accidental match...

Which brings me to an OOPS... in the code snippet I threw out there, I didn't test it and so left one IMPORTANT part out... the IF test is SUPPOSED to read:
     if [ -f "$FS/LABEL ] ; then FOUNDFS=$FS ; fi
After all, the mount point will NEVER be a "regular" file.

So, at the end of the loop, either FOUNDFS has the last discovered filesystem that had the file LABEL in the root, OR it is blank because none was found.

That's where the -z comes in... when you look in the MAN pages (or DOC), remember that if (the bash operator) only checks the return code of the following command -- which means that "[" is actually a command! (it is an alias for the TEST command)... so do a MAN on TEST to read all about the options to the "[" command. In any case, the -z tests to see if the argument afterwards has ZERO length...  which leads me to ANOTHER error in my script... the IF test with the -z line should read:
     if [ -z "$FOUNDFS" ] ; then
-- in other words, I SHOULD have put $FOUNDFS in double-quotes (not single-quotes). That way, if the variable truly IS still blank, -z will actually SEE it.

Next question -- the EXIT command:
  The exit command will cause the current shell to exit with the error code given (or 0 if none is given). Now remember that *nix systems see an exit code of non-zero as a FAILURE. Furthermore, you need to know that shell scripts are run in a separate shell (called a sub-shell). So, in the script I think we're analyzing, the shell script will return a ZERO if everything works OK, or a 1 if NO filesystem was found, or a 2 if the filesystem is the same one as last time. So -- yes, the exit aborts the script, but more importantly, it allows you to set an error code that may be used by the calling program/script/user.

Next question -- the 1>&2:
  All *nix programs have 3 "streams" opened for them by the OS when they start. FD0 (File Descriptor 0) is the input stream called STDIN (sometimes referred to as "standard in", and defaults to the input TTY device (aka: keyboard)). FD1 is an output stream called STDOUT ("standard output", which defaults to the screen). And FD2 is another output stream called STDERR ("standard error", which also defaults to the screen).

You're probably already familiar with the > symbol -- it REDIRECTS STDOUT (FD1) to a file. There is a cousin symbol, < that can be used to redirect STDIN (FD0) from a file (for programs that read only from STDIN). Then there is also the cousin | that redirects the STDOUT of one program to the STDIN of another (called a pipeline, or pipe for short). BUT! There is also a "2nd cousin, twice removed" for both the > & | symbols.... stick a 2 in front of them, and they operate on STDERR instead of STDOUT.

So, when I run the line
  echo "$0 error: backing up to same device as last time" 1>&2
I'm printing the name of the shell I'm running ($0), and my text... but the output of echo (which normally goes to STDOUT is being REDIRECTED to STDERR... so 1>&2 broken down means take the output of STDOUT and connect it to whatever STDERR currently is.

Finally, you inquire as to awk & sed... I ABSOLUTELY think they're important if you're going to write shell scripts. I probably write 2-3 scripts a week, and 90% of them use one or the other or both.

AWK is particularly powerful, and there are WHOLE BOOKS written for all of it's intricacies. About 90% of the time I use it is for FIELD selection. In this case, whitespace is the field delimiter (the output of df), so there are no options, just the command to print the 6th field. However, suppose I wanted to see the shell script used by every user in /etc/passwd (HINT: It's the 7th field, but the fields are separated by colons, not white space).
  awk -F: '{print $7}'
would tell me... OH! but you want the username there too? Try
  awk -F: '{print $1,$7}'
... I could write a 4-page paper on just the use of options to format the output, but you get the general idea.

Sed, on the other hand, allows me to change items on a line (or delete lines)... there are LOTS of sed options too, but the most common ones used are s (for substitution), and d (for delete)... some examples:
  sed 's/bash/csh/' /etc/passwd  # changes the FIRST occurrence of bash on each line of /etc/passwd to csh (thus pissing off all your users by forcing them to use csh instead of bash!)... the output comes to STDOUT, so no change is made to the REAL /etc/passwd.
  sed 's/bash/csh/g' /etc/passwd # changes ALL occurrences of bash found in /etc/passwd to csh
  sed '/bash/q' /etc/passwd  # prints lines in /etc/passwd until bash is found, then quits
  sed '/bash/d' /etc/passwd  # deletes all lines in /etc/passwd where bash is found (like grep -v)

Again... there are LOTS of options -- I used to TEACH this stuff, so I know more than I should!

I hope this helps!

Dan
IT4SOHO
 
0
 

Author Comment

by:blokeman
ID: 23053405
I am impressed and appreciative of your indepth answer!  Even if a lot is beyond me, but I am sure I can have my backup script running much better with your excellent suggestions.

Thanks very much

- BLOKEMAN
0
 

Author Closing Comment

by:blokeman
ID: 31518349
"Partially" only because of my newbness.
Thanks again!!
0
 

Author Comment

by:blokeman
ID: 23601021
I have just got around to fully testing the above scripts and hit a road block...IT seems that variables lose their value when they exit a login loop like WHILE. See comments below...

#!/bin/sh
FOUNDFS=               # clear variable name
df -h | grep "^/dev" | # get a list mounted filesystems from /dev devices  
awk '{print $6}'     | # take the 6th field (the mountpoint)
while read FS ; do       # loop over each one
           echo $FS
      ls $FS/disklabel*
      echo
     if [ -f $FS/disklabel-backup ] ; then FOUNDFS=$FS; echo "* * * fs = $FS and foundfs = $FOUNDFS * * *" ; fi  # This picks up my disk label correctly, but see next comment...
done
echo
echo FoundFS = $FOUNDFS #But $FOUNDFS is blank here ???
echo END
0
 
LVL 21

Expert Comment

by:Daniel McAllister
ID: 23603444
You are correct that shell variables do not "live" outside of some BASH shell loops -- this is a quirk of the BASH shell, in that a sub-shell is often used to execute the loop(s). While it may not APPEAR that you're using a BASH shell (you've specified a BOURNE shell in your script above), on most Linux systems /bin/sh is a link to /bin/bash -- so you ARE using a BASH shell.

There are a few things you can do:
 1) use the KORN shell instead (personally, I abide by the rule-of-thumb: use KORN for scripts, BASH for interactive shells)
 2) Since you're only dealing with ONE variable, capture the value some way... either send it to a file, or use STDOUT

Some examples:
1) To use the KORN shell as an interpreter, first make sure you HAVE a KORN shell (# which ksh), then make the first line of your script "#! /bin/ksh" (or the path to the ksh file you discovered with the which command earlier)

So your script looks like:

#! /bin/kshFOUNDFS=               # clear variable name
df -h | grep "^/dev" | # get a list mounted filesystems from /dev devices  
awk '{print $6}'     | # take the 6th field (the mountpoint)
while read FS ; do       # loop over each one
           echo $FS
      ls $FS/disklabel*
      echo
     if [ -f $FS/disklabel-backup ] ; then FOUNDFS=$FS; echo "* * * fs = $FS and foundfs = $FOUNDFS * * *" ; fi  # This picks up my disk label correctly, but see next comment...
done
echo
echo FoundFS = $FOUNDFS #But $FOUNDFS is blank here ???
echo END

2) Try your loop like this:

FOUNDFS=""
FOUNDFS=`df -h | grep "^/dev" | awk '{print $6}' | while read FS ; do if [ -f ${FS}/disklabel-backup ] ; then echo $FS ; fi ; done`
if [ -z "$FOUNDFS" ] ; then echo $0 ERROR: No Filesystem found with /disklabel-backup ; exit 1 ; fi

(Careful -- run that in a regular shell, and if there is an error, that shell will exit!)

NOTE: You lose all of your debugging... but if you want to insert debugging, send it to STDERR (e.g.: instead of "echo DONE" use "echo DONE >&2"

Or, try something like this

#! /bin/sh
FOUNDFS=               # clear variable name
rm -f /tmp/FOUNDFSes  # clear any old tmp file
df -h | grep "^/dev" | # get a list mounted filesystems from /dev devices  
awk '{print $6}'     | # take the 6th field (the mountpoint)
while read FS ; do       # loop over each one
           echo $FS
      ls $FS/disklabel*
      echo
     if [ -f $FS/disklabel-backup ] ; then echo FOUNDFS=$FS >> /tmp/FOUNDFSes ; echo "* * * fs = $FS and foundfs = $FOUNDFS * * *" ; fi  # This picks up my disk label correctly, but see next comment...
done
. /tmp/FOUNDFSes   # . means "source", and the /tmp file should now have shell command(s) to set FOUNDFS
echo
echo FoundFS = $FOUNDFS
echo END

NOTES:
1 - If you find more than 1 instance, the file /tmp/FOUNDFSes contains a list of them all
2 - Only the LAST found FS is set to the variable FOUNDFS


I hope this helps... someone!!!

Dan
IT4SOHO


0

Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month15 days, 3 hours left to enroll

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question