We help IT Professionals succeed at work.

bash script

sword12
sword12 asked
on
Hi Experts



Scenario
 
In this server, we have a directory called /important  
 
We need to backup this directory every hour to a different path to /backup
 
Then check if the backup successful we need to remove the data from /important  
 
For this, we need you to write for us a shell script, which can do this task for us automatically every hour  
 
Note: we need to specify the time stamp for the backup files


i have this question and i need full answer i manged just to fix the first part which includes making the backup with time stamp

but i need the other part

1-  check if the backup successful remove the data from /important




Solution


#!/bin/bash
backup=/backup/important-`date +%Y-%m-%d-%H-%M`.tar
tar -cvf "$backup" /important




thnaks
Comment
Watch Question

Dr. KlahnPrincipal Software Engineer
CERTIFIED EXPERT

Commented:
This should get you somewhere near what you want.

#!/bin/bash


PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

#
# Create the backup file name
#
PDATE=$(date +"%Y-%m-%d-%H-%M")
echo "--> Date suffix $PDATE"
BFILE=
# Compressed file name
BUFILE=backup/important-$PDATE.tar
echo "--> Proposed backup file $BUFILE"


#
# Make a tar backup
#
echo "--> nice -n 15 tar -cpf $BUFILE --one-file-system /important"
nice -n 15 tar -cpf $BUFILE --one-file-system /important


#
# See if a backup file was actually generated
#
if [ ! -e $BUFILE ]; then
  echo "--> No backup file generated"
  echo "--> Send email to root and stop here"
  /usr/sbin/sendmail root << EOM
From: System_Backup
To: root
Subject: System backup failed

System full backup failed.
No compressed backup file was generated
Should have been:  $BUFILE
EOM
  exit

#
# Backup was successful, purge the directory
#
echo "--> rm -r /important/*
rm -r /important/*

#
# Done
#
echo "--> Done.  Backup and purge complete."

Open in new window

Author

Commented:
thank you

i test it i got this output and this result


[root@linux-lab /]# ./test.sh
--> Date suffix 2020-02-10-16-01
--> Proposed backup file backup/important-2020-02-10-16-01.tar
--> nice -n 15 tar -cpf backup/important-2020-02-10-16-01.tar --one-file-system /important
tar: Removing leading `/' from member names
./test.sh: line 50: unexpected EOF while looking for matching `"'
./test.sh: line 51: syntax error: unexpected end of file
[root@linux-lab /]# cd /important/
[root@linux-lab important]# ll
total 4
drwxr-xr-x. 8 root root 4096 Feb  5 15:05 photos
[root@linux-lab important]# cd /
[root@linux-lab /]# cd /backup/
[root@linux-lab backup]# ll
total 1385752
-rw-r--r--. 1 root root 709498880 Feb 10 15:57 important-2020-02-10-15-57.tar
-rw-r--r--. 1 root root 709498880 Feb 10 16:02 important-2020-02-10-16-01.tar
[root@linux-lab backup]#


where you think the issue




[root@linux-lab /]# less test.sh
#!/bin/bash


PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

#
# Create the backup file name
#
PDATE=$(date +"%Y-%m-%d-%H-%M")
echo "--> Date suffix $PDATE"
BFILE=
# Compressed file name
BUFILE=backup/important-$PDATE.tar
echo "--> Proposed backup file $BUFILE"


#
# Make a tar backup
#
echo "--> nice -n 15 tar -cpf $BUFILE --one-file-system /important"
nice -n 15 tar -cpf $BUFILE --one-file-system /important


#
# See if a backup file was actually generated
#
if [ ! -e $BUFILE ]; then
  echo "--> No backup file generated"
  echo "--> Send email to root and stop here"
  /usr/sbin/sendmail root << EOM
From: System_Backup
To: root
Subject: System backup failed

System full backup failed.
No compressed backup file was generated
Should have been:  $BUFILE
EOM
  exit

#
# Backup was successful, purge the directory
#
echo "--> rm -r /important/*
rm -r /important/*

#
# Done
#
echo "--> Done.  Backup and purge complete."
~
~
~
~
~
~
Dr. KlahnPrincipal Software Engineer
CERTIFIED EXPERT

Commented:
echo "--> nice -n 15 tar --one-file-system -cpf $BUFILE  /important"
nice -n 15 tar --one-file-system -cpf $BUFILE  /important

Open in new window


That may improve things.
David FavorFractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
I'd likely do something like...

tar -cf ~/backup/$timestamp-important-tarball.tar /important

zstd --threads=0 --rm --no-check -19 --long=31 ~/backup/$timestamp-important-tarball.tar

rm -rf /important/*

Open in new window


The specific zstd incantation produces a very fast compression (almost as fast as zip) with far greater compression than xz.

Tip: Using zstd for your backups reduces system load, backup time, disk space usage for backup files.
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

The exit status for all backup creation commands (tar, compress, ...) SHOULD BE tested for success.

Tar will create a file, but if /backup is full the backup is not succesful, the tar does exist...

So adjusting the script from Dr. Klahn: and using zstd compression...


#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin  
# 
# Create the backup file name 
#
PDATE=$(date +"%Y-%m-%d-%H-%M") 
echo "--> Date suffix $PDATE" 

# Compressed file name
BUFILE="backup/important-$PDAT.tar.zst"
echo "--> Proposed backup file $BUFILE"

#
# Make a tar backup
#
echo "--> nice -n 15 tar --one-file-system -cpf $BUFILE  /important"
nice -n 15 tar cpf --one-file-system --zstd $BUFILE  /important
TARRESULT=$?

#
# See if a backup file was actually generated
#
if [ "$TARRESULT" != "0" ] || [ ! -e "$BUFILE" ]
then
   echo "--> No backup file generated"
   echo "--> Send email to root and stop here"
   /usr/sbin/sendmail root << EOM
From: System_Backup
To: root
Subject: System backup failed

System full backup failed. No compressed backup file was generated Should have been:  $BUFILE
Tar exit status: $TARRESULT
EOM

   exit  1
fi
# 
# Backup was successful, purge the directory 
# 

echo "--> rm -r /important/* 
rm -rf /important/*  
# 
# Done 
# 
echo "--> Done.  Backup and purge complete."


Author

Commented:
sorry but till now dose not work


[root@linux-lab /]# ./test.sh
--> Date suffix 2020-02-11-08-27
--> Proposed backup file backup/important-.tar.zst
--> nice -n 15 tar --one-file-system -cpf backup/important-.tar.zst  /important
tar: unrecognized option '--zstd'
Try 'tar --help' or 'tar --usage' for more information.
--> No backup file generated
--> Send email to root and stop here
./test.sh: line 28: From:: No such file or directory
./test.sh: line 29: To:: command not found
./test.sh: line 30: Subject:: command not found
./test.sh: line 32: System: command not found
./test.sh: line 33: Tar: command not found
./test.sh: line 34: EOM: command not found
[root@linux-lab /]# cd backup/
[root@linux-lab backup]# ll
total 0
[root@linux-lab backup]# cd /important/
[root@linux-lab important]# ll
total 4
-rw-r--r--. 1 root root 34 Feb 11 08:25 hi
[root@linux-lab important]#
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

Ok 

i repaired the script  above if: "<<EOM" is written it is removed from view....  "<< EOM" does show corerctly. EE Bug i already reported. see below:

blah <<EOM
asdf
ad
EOM
 
blah << EOM
adsf
ad
EOM
 





Appearantly your tar doesn't known zstd compression, what version are you using?

You mention bash/linux versions of linux can help. There is a ifference between versions as each of the componenets gets upgraded continuously.

zstd has been available to tar for about 2 years... and update might be in place.

RHEL 7 doesn't have it as it uses a tar is from 2013.


Check with man tar which compression can be used if it doesn't know --zstd then try --xz if that isn't available try --bzip2 if that files try --gz.

the file extentions for tar need  then need to be resp. .tar.xz, .tar.bz2 or .tar.gz.


Author

Commented:
Hi noci

thank you because you are helping me

i cheked man tar and i found --xz   supported so i change it in the script

and i removed the space << EOM   no look like this  <<EOM

then i test it but dose not make any backup file and removed nothing

please check the attached file

and please try to test it on any centos box you have

try to make /importnat   and /backup  and test this script on your box then send me the corrected one

thanks a lot
3.jpg
Dr. KlahnPrincipal Software Engineer
CERTIFIED EXPERT

Commented:
Try issuing the tar command manually from the working directory and see what happens.

Try the tar command without the compression switch.  If that works, the compression (whatever kind you prefer) can be added later on in the script per below.

#
# Compress the backup
# At present 7-Zip produces the best compression
# test notes:  Bzip2 compresses 40% bigger than 7z
#              gzip compresses 50% bigger than 7z
#
echo "--> nice -n 15 7z a -t7z (target-file-name) (tar-file-name)"
nice -n 15 7z a -t7z (target-file-name) (tar-file-name)
# Remove the tar file
echo "--> rm -f (tar-file-name)"
rm -f (tar-file-name)

Open in new window

Author

Commented:
dear Dr Klahn

the first version you wrote worked very well

it create the tar file  but the first version dose not delete the data from the source

so the first part about creating the backup (( tar file )) worked

but the problem with check if the backup done successfully then remove the data from the source

so please try to help me and fix you script and test it in any linux box you have then give it to me


thanks
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

This should definitely work....

#!/bin/bash 
 
IMPORTANT=/important 
BACKUP=/backup 
COMPRESSION=--xz 
EXT=xz 
 
if [ -z "$BACKUP" ] || [ ! -d $BACKUP ] 
then 
  echo "Verify backup destination settings" 
fi 
if [ -z "$IMPORTANT" ] || [ ! -d $IMPORTANT ] 
then 
  echo "Verify backup source settings" 
fi 
 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 
# 
# Create the backup file name 
# 
PDATE=$(date +"%Y-%m-%d-%H-%M") 
echo "--> Date suffix $PDATE" 
 
# Compressed file name 
BUFILE="$BACKUP/important-$PDATE.tar.$EXT" 
echo "--> Proposed backup file $BUFILE" 
 
# 
# Make a tar backup 
# 
echo "--> nice -n 15 tar cf $BUFILE --one-file-system $COMPRESSION  $IMPORTANT" 
nice -n 15 tar cf $BUFILE --one-file-system $COMPRESSION $IMPORTANT 
TARRESULT=$? 
 
# 
# See if a backup file was actually generated 
# 
if [ "$TARRESULT" != "0" ] || [ ! -e "$BUFILE" ] 
then 
   echo "--> No backup file generated" 
   echo "--> Send email to root and stop here" 
   /usr/sbin/sendmail root << EOM 
From: System_Backup 
To: root 
Subject: System backup failed 
 
System full backup failed. No compressed backup file was generated Should have been:  $BUFILE 
Tar exit status: $TARRESULT 
EOM 
 
   exit  1 
fi 
# 
# Backup was successful, purge the directory 
# 
 
echo "--> rm -rf $IMPORTANT/*" 
rm -rf $IMPORTANT/* 
# 
# Done 
# 
echo "--> Done.  Backup and purge complete." 

Author

Commented:
yes it work partly

please check the attached file

the source data has to be removed
4.jpg
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

Somehow the EOM on line 49 is not seen....

Consequently the rm is missing and you got the errors about line 62.... EOF.


The EOM needs to be on the LEFT, no spaces in front!


The EOM on line 42 doesn't matter if there is a blank in front.

Author

Commented:
hi noci

i would like to thank you because you are investing time to help me

the issue still even if played with EOM . (( please check the attached file ))

i think in order to fix this issue you can just connect to any Linux box you have and

1- create direct called /important
2- create one txt file under /important
3- create another directory /backup

then test your script this will save your time and this will help me to get a tested script

thanks a lot
5.jpg
We like KT across the universe.  C C++ shell script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT
CERTIFIED EXPERT
Commented:
@sword12
Base version from noci:
Modifications:
1. Removed blank lines
2. Included following lines(PS4 and set -x) for debugging:
#!/bin/bash
export PS4='$0 $LINENO: '
set -x
IMPORTANT=/important
3. Initially validating directory and execute the script only if they exist:
Replace:
if [ -z "$BACKUP" ] || [ ! -d $BACKUP ]
then
        echo "Verify backup destination settings"
fi
if [ -z "$IMPORTANT" ] || [ ! -d $IMPORTANT ]
then
        echo "Verify backup source settings"
        exit 1
fi

Open in new window

With:
if [ -z "$BACKUP" ] || [ ! -d $BACKUP ]
then
        echo "Verify backup destination settings"
        exit 1
fi
if [ -z "$IMPORTANT" ] || [ ! -d $IMPORTANT ]
then
        echo "Verify backup source settings"
        exit 1
fi

Open in new window

4. Remove space at end of the lines which are having space
Replace:
IMPORTANT=/important 
BACKUP=/backup 

Open in new window

With:
IMPORTANT=/important
BACKUP=/backup

Open in new window

5. Removed lines starting and ending with # character
6. Created a directory manually:
mkdir /backup /important
7. Commented sendmail related commands since no email sendmail and mail here.
Not writing more comments
copying and pasting updated code:
#!/bin/bash
unalias SETBINPATH >/dev/null 2>&1
unset -f SETBINPATH
SETBINPATH ()
{
        if [ -f /usr/sbin/sendmail ]
        then
                SENDMAIL="/usr/sbin/sendmail"
        elif [ -f /sbin/sendmail ]
        then
                SENDMAIL="/sbin/sendmail"
        else
                echo "Update this script for correct location of sendmail"
                exit 5
        fi
        if [ -f /usr/bin/sleep ]
        then
                SLEEP="/usr/bin/sleep"
        elif [ -f /usr/bin/sleep ]
        then
                SLEEP="/usr/bin/sleep"
        else
                echo "Update this script for correct location of sleep"
                exit 5
        fi
        if [ -f /usr/bin/date ]
        then
                DATE="/usr/bin/date"
        elif [ -f /usr/bin/date ]
        then
                DATE="/usr/bin/date"
        else
                echo "Update this script for correct location of date"
                exit 5
        fi
        if [ -f /usr/bin/tar ]
        then
                TAR="/usr/bin/tar"
        elif [ -f /usr/bin/tar ]
        then
                TAR="/usr/bin/tar"
        else
                echo "Update this script for correct location of tar"
                exit 5
        fi
        if [ -f /usr/bin/rm ]
        then
                RM="/usr/bin/rm"
        elif [ -f /usr/bin/rm ]
        then
                RM="/usr/bin/rm"
        else
                echo "Update this script for correct location of rm"
                exit 5
        fi
}
IMPORTANT=/important
BACKUP=/backup
EXT=xz
if [ "-bash" = "$0" ] || [ "-ksh" = "$0" ]
then
        echo "Executing this script at current shell not allowed"
else
        SETBINPATH $@
        if [ -z "$BACKUP" ]
        then
                echo "BACKUP environment variable not set."
                exit 1
        fi
        if [ ! -d $BACKUP ]
        then
                echo "$BACKUP directory not present"
                exit 2
        fi
        if [ -z "$IMPORTANT" ]
        then
                echo "IMPORTANT environment variable not set."
                exit 3
        fi
        if [ ! -d $IMPORTANT ]
        then
                echo "$IMPORTANT directory not present"
                echo "Verify backup source settings"
                exit 4
        fi
        PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
        echo "--> Date suffix $PDATE"
        while [ 1 ]
        do
                PDATE=$($DATE +"%Y_%m_%d_%H_%M_%S") # Create the backup file name using _ as seperator.
                BUFILE="$BACKUP/important_$PDATE.tar.$EXT" # Compressed file name
                if [ ! -f "$BUFILE" ]
                then
                        break
                fi
                $SLEEP 1 # SLEEP for 1 second to change PDATE again.
                echo "--> $SLEEP 1"
        done
        echo "--> Proposed backup file $BUFILE"
        echo "--> $TAR -czf $BUFILE ./*" # Make a $TAR backup
        cd $IMPORTANT # To remove / directory warning when using TAR.
        $TAR -czf $BUFILE ./* >/dev/null 2>&1
        TARRESULT=$?
        if [ $TARRESULT -ne 0 ] || [ ! -e "$BUFILE" ] # See if a backup file was actually generated
        then
                echo "--> No backup file generated"
                # I have disabled $SENDMAIL
                #echo "--> $SENDMAIL to root and stop here"
#$SENDMAIL root << EOM
#From: System_Backup
#To: root
#Subject: System backup failed
#System full backup failed. No compressed backup file was generated Should have been:  $BUFILE
#Tar exit status: $TARRESULT
#EOM
                exit  1
        fi
        echo "--> $RM -rf $IMPORTANT/*" # Backup was successful, purge the directory
        # You can remove echo here after completing unit testing.
        echo $RM -rf $IMPORTANT/* # Done
        #$RM -rf $IMPORTANT/* # Done
        echo "--> Done.  Backup and purge complete."
fi

Open in new window

Author

Commented:
great job

but the files still available under /important  so some how  

rm -rf /important/*

dose not take effects  so where i have to make changes to let it work


[root@linux-lab /]# ./sword.sh
--> Date suffix
--> Proposed backup file /backup/important_2020_02_12_10_47_59.tar.xz
--> /usr/bin/tar -czf /backup/important_2020_02_12_10_47_59.tar.xz ./*
--> /usr/bin/rm -rf /important/*
/usr/bin/rm -rf /important/gggg /important/hh /important/hi /important/jj
--> Done.  Backup and purge complete.
[root@linux-lab /]# cd important/
[root@linux-lab important]# ll
total 8
drwxr-xr-x. 2 root root 4096 Feb 12 08:19 gggg
-rw-r--r--. 1 root root    0 Feb 12 08:19 hh
-rw-r--r--. 1 root root   34 Feb 11 08:25 hi
-rw-r--r--. 1 root root    0 Feb 12 08:19 jj
[root@linux-lab important]# cd /
[root@linux-lab /]# cd backup/
[root@linux-lab backup]# ll
total 4
-rw-r--r--. 1 root root 204 Feb 12 10:47 important_2020_02_12_10_47_59.tar.xz
[root@linux-lab backup]# pwd
Murugesan NWe like KT across the universe.  C C++ shell script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT
CERTIFIED EXPERT

Commented:
@sword12
I have written following comment:
# You can remove echo here after completing unit testing.
        echo $RM -rf $IMPORTANT/* # Done
        #$RM -rf $IMPORTANT/* # Done

Open in new window

Hence
replace above code to:
# You can remove echo here after completing unit testing.
#echo $RM -rf $IMPORTANT/* # Done
$RM -rf $IMPORTANT/* # Done

Open in new window

Author

Commented:
it worked like a charm

thanks a lot for all of you
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

spaces at the end of the lines seem to be inserted by the EE website. They were not in the scripts i pasted here.

Script was running on my system..  

Anyway it works now. 

Murugesan NWe like KT across the universe.  C C++ shell script Automation /bin/bash /bin/bash.exe /bin/ksh /bin/mksh.exe AIX CYGWIN_NT HP-UX Linux MINGW32 MINGW64 SunOS Windows_NT
CERTIFIED EXPERT

Commented:
@sword12

>> thanks a lot for all of you


>> comments not only from noci from others too.
I was thinking the same when you accepted the solution.

In my code
Replace:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
        echo "--> Date suffix $PDATE"
        while [ 1 ]
        do
                PDATE=$($DATE +"%Y_%m_%d_%H_%M_%S") # Create the backup file name using _ as seperator.
                BUFILE=
With:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
        while [ 1 ]
        do
                PDATE=$($DATE +"%Y_%m_%d_%H_%M_%S") # Create the backup file name using _ as seperator.
                echo "--> Date suffix $PDATE"
                BUFILE=

If you feel to change the points, you can change(share :) ) the given points to others if required.