?
Solved

Automation backup process using cron

Posted on 2007-08-02
28
Medium Priority
?
527 Views
Last Modified: 2013-12-16
Here is the scenario I do every day

==============================
- log into my machine as root
- rename file1.htm to file2.htm
- rename file3.htm to file1.htm
- rename folder main to secondary
- run command: mysqldump database_name | gzip -c > /home/backups/backup_date.sql.gz
the mysqldump command normally takes 5 minutes, once finished, continue
- run command: service mysql restart
- rename folder secondary to main
- rename file1.htm to file3.htm
- rename file2.htm to file1.htm
===================
How to automate these steps to run everyday at specific time.  I know it can be done through cron job, but I think I should first put all these commands into a file which will be run by the cron job.

Please note that the rename of the two files and folder is a MUST, and also the "backup_date.sql.gz" file should have different date everytime it is created, so the word "date" should be replaced by the current date (e.g. 03Aug2007, or 04Aug2007 for tom.), so the output file should look like: backup_03Aug2007.sql.gz

Any help with this!
0
Comment
Question by:mafairuz
  • 15
  • 13
28 Comments
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19622198
These are all standard shell commands.  Start with a bash shell script.  Enhance it as you work with scripting for validation, errors, success, etc like you would do if you did it manually.  I typically make a backup of the data files and directories before manipulating them.

Here is a crude example of putting your commands into a script.  I use perl more often than bash but both work.  Make sure that all paths to programs, files and directories are specified.  I presume the location of programs in the script below based upon where they are on my workstation

[root]# nano /home/backups/bin/mysql-backup.sh

#################################
#!/bin/bash

logfile=/home/backup/mysql-backup.log

# This presumes that all files and directories
# are in the same subdirectory

cd /location/of/files

echo "MySql Backup `date'" $logfile

# the if statements verify that the files and directories exist to determine which action to take

if [ -f /location/of/files/file1.htm ]; then
        mv -f file1.htm file2.htm
        echo "File1 renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

if [ -f /location/of/files/file3.htm ]; then
        mv -f file3.htm file1.htm
        echo "File3 renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

if [ -d /location/of/files/main ] ; then
        mv -f main secondary
        echo "Main directory renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_date.sql.gz

service mysql restart

cd /location/of/files
        mv -f secondary main
        mv -f file1.htm file3.htm
        mv -f file2.htm file1.htm

# consider mailing the logfile so that you don't have to log in to read it
########################################

[root]# chmod 755 /home/backups/bin/mysql-backup.sh
[root]# crontab -e
0 1 * * * /home/backups/bin/mysql-backup.sh > /dev/null 2>&1

<save the cron entry and exit>

View your cron entry, make sure cron is running, start it *if* it isn't running:

[root]# crontab -l
[root]# ps awx | grep cron
[root]# service crond start

This will run the cron job every day at 1 am.
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19622225
modify for date:

underneath the 'logfile' variable, add this line ->

mydate=`date +%d%b%y`

change this line to and put on all one line unless you escape it for continuation->

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19623715
thanks jesper

that was amazing .. just before I go ahead with implementation .. I've got few queries

if the files and the folder are not located in the same place, can I use the following:
====================
        mv -f /home/backups/secondary /home/backups/main
        mv -f /home/one/file1.htm /home/one/file3.htm
        mv -f /home/two/file2.htm /home/two/file1.htm
====================

Also with regards to the following two commands:

=================
/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz
service mysql restart
==================
will the second command start after the first one finishes or they will start together ? may sound a stupid question, but just to make sure things are a bit clear to me :)

Also what do you mean by: # do something here: log it, send email, abort script?
is there any example you may give ..

Last maybe, can I write this script using notepad and then uploaded using ftp or it has to be written using some sort of LINUX writting programs!

Thanks for help
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 29

Expert Comment

by:Jan Springer
ID: 19624421
Yes, every place that there is a path, like to file1.htm, make sure that the absolute path is correct.  Down the road, you can create a variable to define the path and just change the variable.

It's not a stupid question.  In shell scripts the commands will be process as you specify and the next command will wait until the previous command is complete.  I have used the 'sleep' command in some scripts under different circumstances.

The 'else' comments identify what you want to do if the file does not exist. Those additions with corrections in the script below.  

Replace the 'user@domain.com' with your email address.

***Before you let it go to cron
   1) make a backup copy of all files and directories being manipulated to something different
         like file1.htm.orig
   2) run the command manually to check for accuracy and errors
       [root]# /home/backups/bin/mysql-backup.sh
   3) Check that everything processed as it should


#######################################
#!/bin/bash

logfile=/home/backup/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`date +%d%b%y`


# This presumes that all files and directories
# are in the same subdirectory

echo "MySql Backup `date'" $logfile

if [ -f /path/to/file1.htm ]; then
        mv -f /path/to/file1.htm /path/to/file2.htm
        echo "File1 renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -f /path/to/file3.htm ]; then
        mv -f /path/to/file3.htm /path/to/file1.htm
        echo "File3 renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -d /path/to/main ] ; then
        mv -f /path/to/main /path/to/secondary
        echo "Main directory renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz

service mysql restart

mv -f /path/to/secondary /path/to/main
mv -f /path/to/file1.htm /path/to/file3.htm
mv -f /path/to/file2.htm /path/to/file1.htm

/bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
###################################
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19629422
hi jesper

I've done everything and tried to run the script without cron using /home/backups/bin/mybackup.sh command, but I got the following error:
bash: /home/backups/bin/mybackup.sh: Permission denied

Also I tried to test the e-mail sending command by itself as follows:
/bin/cat /home/backups/mysql-backup.log | mailx -s MySql_Logs -F backup.log test@mydomain.com

but I got the following error:
======================
mailx: invalid option -- F
Usage: mail [-iInv] [-s subject] [-c cc-addr] [-b bcc-addr] to-addr ...
            [- sendmail-options ...]
       mail [-iInNv] -f [name]
       mail [-iInNv] [-u user]
===========================

any help!
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19631070
1) are you root when  you run the script?  if you need to be another user
    log  in as that user to create the script, set the permissions and do the 'crontab -e' as that user
2) change the mail command to:
    /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com

Let me know if you still run into problems.
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19631087
Thanks jesper

Actually I log as root .. I tried to run the script using "sh" command and it seems it worked.  However, I got the following error message:

==========================================
root@serv01 [/]# sh /home/backups/bin/mybackup.sh
: command not foundckup.sh: line 3:
: command not foundckup.sh: line 5:
: command not foundckup.sh: line 8:
: command not foundckup.sh: line 10:
: command not foundckup.sh: line 11:
: command not foundckup.sh: line 14:
/home/backups/bin/mybackup.sh: line 15: unexpected EOF while looking for matching ``'
/home/backups/bin/mybackup.sh: line 54: syntax error: unexpected end of file
====================================================
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19631154
Did you specify the full path to foundckup.sh ?

Can you post the actual script that you are using?

And, you don't need to call the 'sh' to run the script.  That's supplied in the first line of the script.
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19631172
I don't know what is the foundchup.sh .. it is not included in the script at all

here is the script, after changing the actual email address back to user@domain.com, and actual user name to user in /home/user/:

#######################################
#!/bin/bash

logfile=/home/backups/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`date +%d%b%y`


# This presumes that all files and directories
# are in the same subdirectory

echo "MySql Backup `date'" $logfile

if [ -f /home/user/www/404.shtml ]; then
        mv -f /home/user/www/404.shtml /home/user/www/404_original.shtml
        echo "404.shtml to 404_original.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404.shtml to 404_orginal.shtml rename step"
 >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -f /home/user/www/404_mod.shtml ]; then
        mv -f /home/user/www/404_mod.shtml /home/user/www/404.shtml
        echo "404_mod.shtml to 404.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404_mod.shtml to 404.shtml rename step" >>
${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -d /home/user/www/main ] ; then
        mv -f /home/user/www/main /home/user/www/main_sleep
        echo "main to main_sleep directory rename successfully" >> $logfile
else
        echo "MySql Backup failed at main to main_sleep directory rename" >> ${log
file}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

/usr/bin/mysqldump database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql
.gz

service mysql restart

mv -f /home/user/www/avb_sleep /home/user/www/avb
mv -f /home/user/www/404.shtml /home/user/www/404_mod.shtml
mv -f /home/user/www/404_original.shtml /home/user/www/404.shtml

/bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
###################################
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19631189
And this script is actually called :
   /home/backups/bin/mybackup.sh

Permissions are rwxr-x-r-x on the file?
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19631203
also can you verify that where the lines in the script wrap around that they are actually only on one line?

one change (line 14) :

 echo "MySql Backup ${mydate}" $logfile
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19631221
I made the change requested, and by testing another script, it seems the permission denied is no longer appears after I chmod to 755

I can't test the script now as it is peak hours .. I will give it a try at midnight and come back with results
thanks jasper for the time and help :)
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19631238
I've made the changes previously suggested and here is the new one (adjust for your paths and fix line wraps):

#!/bin/bash

logfile=/home/backups/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`/bin/date +%d%b%y`

echo "MySql Backup ${mydate}" $logfile

if [ -f /home/user/www/404.shtml ]; then
        mv -f /home/user/www/404.shtml /home/user/www/404_original.shtml
        echo "404.shtml to 404_original.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404.shtml to 404_orginal.shtml rename step" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi

if [ -f /home/user/www/404_mod.shtml ]; then
        mv -f /home/user/www/404_mod.shtml /home/user/www/404.shtml
        echo "404_mod.shtml to 404.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404_mod.shtml to 404.shtml rename step" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi

if [ -d /home/user/www/main ] ; then
        mv -f /home/user/www/main /home/user/www/main_sleep
        echo "main to main_sleep directory rename successfully" >> $logfile
else
        echo "MySql Backup failed at main to main_sleep directory rename" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi


##this line wraps, fix it
/usr/bin/mysqldump database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz

service mysql restart

mv -f /home/user/www/avb_sleep /home/user/www/avb
mv -f /home/user/www/404.shtml /home/user/www/404_mod.shtml
mv -f /home/user/www/404_original.shtml /home/user/www/404.shtml

/bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19631250
thanks I will for sure
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19632437
hi jesper

the script worked out perfectly .. files and folders got renamed, database got backedup as required.
However :), only the last command (sending mysql-backup.log contents by e-mail) did not work, and I got the following error:
=========================
/bin/cat: /home/backups//home/backups/mysql-backup.log: No such file or directory
Null message body; hope that's ok
==========================

The file /home/backups/mysql-backup.log does exist and its contents shows clearly that the process worked out.   Maybe the last command be modified to look like:
/bin/cat ${logfile} | mail -s MySql_Logs user@domain.com

Okey final issue, how to schedule this script to work out at 3 midnight (my time) every day using cron?
by the way, my time is 9 hours ahead server time .. for example, 3 midnight my time, is 6pm yesterday server time ..
This actually created a problem with backup file, the file was named (backup_04Aug07 instead of backup_05Aug07) which I think can be fixed in mydate definition line, but I don't know how.

just these two bits and I would be thankful for your time jasper
0
 
LVL 29

Accepted Solution

by:
Jan Springer earned 1200 total points
ID: 19634236
We need to schedule the cron job by the server time.  If you want the job to run at 03:00 your time which is 18:00 server time, then the cron entry needs to be:

0 18 * * * /home/backups/bin/mysql-backup.sh > /dev/null 2>&1


Now we need to add a day to the filename.  Delete the mydate variable and in its place put:

date=`/bin/date`
mydate=$(date +%d%b%y --date='Tomorrow')


And last, you surely caught my error with the logfile.  We also want to add the path to the mail program:

/bin/cat ${logfile} | /bin/mail -s MySql_Logs user@domain.com
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19635935
Thanks again jesper
When I run the script again after the changes you said, I got the following error :s

=================
/home/backups/bin/mysql-backup.sh: line 9: date+%d%b%y: command not found
=================

what should be the problem there !
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636034
That is saying that it can't find the date command.

Does the preceding line say:

date=`/bin/date`  

?
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19636040
yes, here follows what I have in the shell script

===========
date=`/bin/date`
mydate=$(date+%d%b%y --date='Tomorrow')
===========
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636042
Do this as (any user is fine, root is shown as an example):

[root]# nano test-date

#!/bin/bash
date=`/bin/date`
mydate=$(date +%d%b%y --date='Tomorrow')
echo $mydate

<save the file>

[root]# chmod 755 test-date
[root]# ./test-date

What is the output?
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636045
Can you verify that 'date' lives in /bin?
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636048
Wait.  I don't see a space between $(date and the +
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19636050
yes maybe the problem was with the space as the test.sh worked fine
let me change on the main script and try out .. it is off-peak now :)
just a moment
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19636055
Thanks jesper .. :)
things are just perfect right now
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636059
Wonderful!  Anything else we need to do with this?
0
 
LVL 3

Author Comment

by:mafairuz
ID: 19636062
yes, I will add a new separate question for you later .. I will need to pass my backup file to another server using any way .. like ftp or anything else .. I hope there is a way for this to be done automaticlly

Thanks again
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 19636064
sounds good.  we can go that.  i'll set a filter for your name and watch for it.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month14 days, 7 hours left to enroll

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question