Automation backup process using cron

Here is the scenario I do every day

==============================
- log into my machine as root
- rename file1.htm to file2.htm
- rename file3.htm to file1.htm
- rename folder main to secondary
- run command: mysqldump database_name | gzip -c > /home/backups/backup_date.sql.gz
the mysqldump command normally takes 5 minutes, once finished, continue
- run command: service mysql restart
- rename folder secondary to main
- rename file1.htm to file3.htm
- rename file2.htm to file1.htm
===================
How to automate these steps to run everyday at specific time.  I know it can be done through cron job, but I think I should first put all these commands into a file which will be run by the cron job.

Please note that the rename of the two files and folder is a MUST, and also the "backup_date.sql.gz" file should have different date everytime it is created, so the word "date" should be replaced by the current date (e.g. 03Aug2007, or 04Aug2007 for tom.), so the output file should look like: backup_03Aug2007.sql.gz

Any help with this!
LVL 3
mafairuzAsked:
Who is Participating?
 
Jan SpringerConnect With a Mentor Commented:
We need to schedule the cron job by the server time.  If you want the job to run at 03:00 your time which is 18:00 server time, then the cron entry needs to be:

0 18 * * * /home/backups/bin/mysql-backup.sh > /dev/null 2>&1


Now we need to add a day to the filename.  Delete the mydate variable and in its place put:

date=`/bin/date`
mydate=$(date +%d%b%y --date='Tomorrow')


And last, you surely caught my error with the logfile.  We also want to add the path to the mail program:

/bin/cat ${logfile} | /bin/mail -s MySql_Logs user@domain.com
0
 
Jan SpringerCommented:
These are all standard shell commands.  Start with a bash shell script.  Enhance it as you work with scripting for validation, errors, success, etc like you would do if you did it manually.  I typically make a backup of the data files and directories before manipulating them.

Here is a crude example of putting your commands into a script.  I use perl more often than bash but both work.  Make sure that all paths to programs, files and directories are specified.  I presume the location of programs in the script below based upon where they are on my workstation

[root]# nano /home/backups/bin/mysql-backup.sh

#################################
#!/bin/bash

logfile=/home/backup/mysql-backup.log

# This presumes that all files and directories
# are in the same subdirectory

cd /location/of/files

echo "MySql Backup `date'" $logfile

# the if statements verify that the files and directories exist to determine which action to take

if [ -f /location/of/files/file1.htm ]; then
        mv -f file1.htm file2.htm
        echo "File1 renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

if [ -f /location/of/files/file3.htm ]; then
        mv -f file3.htm file1.htm
        echo "File3 renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

if [ -d /location/of/files/main ] ; then
        mv -f main secondary
        echo "Main directory renamed successfully" $logfile
else
        # do something here: log it, send email, abort script?
fi

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_date.sql.gz

service mysql restart

cd /location/of/files
        mv -f secondary main
        mv -f file1.htm file3.htm
        mv -f file2.htm file1.htm

# consider mailing the logfile so that you don't have to log in to read it
########################################

[root]# chmod 755 /home/backups/bin/mysql-backup.sh
[root]# crontab -e
0 1 * * * /home/backups/bin/mysql-backup.sh > /dev/null 2>&1

<save the cron entry and exit>

View your cron entry, make sure cron is running, start it *if* it isn't running:

[root]# crontab -l
[root]# ps awx | grep cron
[root]# service crond start

This will run the cron job every day at 1 am.
0
 
Jan SpringerCommented:
modify for date:

underneath the 'logfile' variable, add this line ->

mydate=`date +%d%b%y`

change this line to and put on all one line unless you escape it for continuation->

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz
0
Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

 
mafairuzAuthor Commented:
thanks jesper

that was amazing .. just before I go ahead with implementation .. I've got few queries

if the files and the folder are not located in the same place, can I use the following:
====================
        mv -f /home/backups/secondary /home/backups/main
        mv -f /home/one/file1.htm /home/one/file3.htm
        mv -f /home/two/file2.htm /home/two/file1.htm
====================

Also with regards to the following two commands:

=================
/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz
service mysql restart
==================
will the second command start after the first one finishes or they will start together ? may sound a stupid question, but just to make sure things are a bit clear to me :)

Also what do you mean by: # do something here: log it, send email, abort script?
is there any example you may give ..

Last maybe, can I write this script using notepad and then uploaded using ftp or it has to be written using some sort of LINUX writting programs!

Thanks for help
0
 
Jan SpringerCommented:
Yes, every place that there is a path, like to file1.htm, make sure that the absolute path is correct.  Down the road, you can create a variable to define the path and just change the variable.

It's not a stupid question.  In shell scripts the commands will be process as you specify and the next command will wait until the previous command is complete.  I have used the 'sleep' command in some scripts under different circumstances.

The 'else' comments identify what you want to do if the file does not exist. Those additions with corrections in the script below.  

Replace the 'user@domain.com' with your email address.

***Before you let it go to cron
   1) make a backup copy of all files and directories being manipulated to something different
         like file1.htm.orig
   2) run the command manually to check for accuracy and errors
       [root]# /home/backups/bin/mysql-backup.sh
   3) Check that everything processed as it should


#######################################
#!/bin/bash

logfile=/home/backup/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`date +%d%b%y`


# This presumes that all files and directories
# are in the same subdirectory

echo "MySql Backup `date'" $logfile

if [ -f /path/to/file1.htm ]; then
        mv -f /path/to/file1.htm /path/to/file2.htm
        echo "File1 renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -f /path/to/file3.htm ]; then
        mv -f /path/to/file3.htm /path/to/file1.htm
        echo "File3 renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -d /path/to/main ] ; then
        mv -f /path/to/main /path/to/secondary
        echo "Main directory renamed successfully" >> $logfile
else
        echo "MySql Backup failed at [specify location of script]" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

/usr/bin/mysqldump /location/of/database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz

service mysql restart

mv -f /path/to/secondary /path/to/main
mv -f /path/to/file1.htm /path/to/file3.htm
mv -f /path/to/file2.htm /path/to/file1.htm

/bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
###################################
0
 
mafairuzAuthor Commented:
hi jesper

I've done everything and tried to run the script without cron using /home/backups/bin/mybackup.sh command, but I got the following error:
bash: /home/backups/bin/mybackup.sh: Permission denied

Also I tried to test the e-mail sending command by itself as follows:
/bin/cat /home/backups/mysql-backup.log | mailx -s MySql_Logs -F backup.log test@mydomain.com

but I got the following error:
======================
mailx: invalid option -- F
Usage: mail [-iInv] [-s subject] [-c cc-addr] [-b bcc-addr] to-addr ...
            [- sendmail-options ...]
       mail [-iInNv] -f [name]
       mail [-iInNv] [-u user]
===========================

any help!
0
 
Jan SpringerCommented:
1) are you root when  you run the script?  if you need to be another user
    log  in as that user to create the script, set the permissions and do the 'crontab -e' as that user
2) change the mail command to:
    /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com

Let me know if you still run into problems.
0
 
mafairuzAuthor Commented:
Thanks jesper

Actually I log as root .. I tried to run the script using "sh" command and it seems it worked.  However, I got the following error message:

==========================================
root@serv01 [/]# sh /home/backups/bin/mybackup.sh
: command not foundckup.sh: line 3:
: command not foundckup.sh: line 5:
: command not foundckup.sh: line 8:
: command not foundckup.sh: line 10:
: command not foundckup.sh: line 11:
: command not foundckup.sh: line 14:
/home/backups/bin/mybackup.sh: line 15: unexpected EOF while looking for matching ``'
/home/backups/bin/mybackup.sh: line 54: syntax error: unexpected end of file
====================================================
0
 
Jan SpringerCommented:
Did you specify the full path to foundckup.sh ?

Can you post the actual script that you are using?

And, you don't need to call the 'sh' to run the script.  That's supplied in the first line of the script.
0
 
mafairuzAuthor Commented:
I don't know what is the foundchup.sh .. it is not included in the script at all

here is the script, after changing the actual email address back to user@domain.com, and actual user name to user in /home/user/:

#######################################
#!/bin/bash

logfile=/home/backups/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`date +%d%b%y`


# This presumes that all files and directories
# are in the same subdirectory

echo "MySql Backup `date'" $logfile

if [ -f /home/user/www/404.shtml ]; then
        mv -f /home/user/www/404.shtml /home/user/www/404_original.shtml
        echo "404.shtml to 404_original.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404.shtml to 404_orginal.shtml rename step"
 >> ${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -f /home/user/www/404_mod.shtml ]; then
        mv -f /home/user/www/404_mod.shtml /home/user/www/404.shtml
        echo "404_mod.shtml to 404.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404_mod.shtml to 404.shtml rename step" >>
${logfile}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

if [ -d /home/user/www/main ] ; then
        mv -f /home/user/www/main /home/user/www/main_sleep
        echo "main to main_sleep directory rename successfully" >> $logfile
else
        echo "MySql Backup failed at main to main_sleep directory rename" >> ${log
file}
        /bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
        exit
fi

/usr/bin/mysqldump database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql
.gz

service mysql restart

mv -f /home/user/www/avb_sleep /home/user/www/avb
mv -f /home/user/www/404.shtml /home/user/www/404_mod.shtml
mv -f /home/user/www/404_original.shtml /home/user/www/404.shtml

/bin/cat /home/backups/${logfile} | mailx -s MySql_Logs -F backup.log user@domain.com
###################################
0
 
Jan SpringerCommented:
And this script is actually called :
   /home/backups/bin/mybackup.sh

Permissions are rwxr-x-r-x on the file?
0
 
Jan SpringerCommented:
also can you verify that where the lines in the script wrap around that they are actually only on one line?

one change (line 14) :

 echo "MySql Backup ${mydate}" $logfile
0
 
mafairuzAuthor Commented:
I made the change requested, and by testing another script, it seems the permission denied is no longer appears after I chmod to 755

I can't test the script now as it is peak hours .. I will give it a try at midnight and come back with results
thanks jasper for the time and help :)
0
 
Jan SpringerCommented:
I've made the changes previously suggested and here is the new one (adjust for your paths and fix line wraps):

#!/bin/bash

logfile=/home/backups/mysql-backup.log

rm -f ${logfile}
touch ${logfile}

mydate=`/bin/date +%d%b%y`

echo "MySql Backup ${mydate}" $logfile

if [ -f /home/user/www/404.shtml ]; then
        mv -f /home/user/www/404.shtml /home/user/www/404_original.shtml
        echo "404.shtml to 404_original.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404.shtml to 404_orginal.shtml rename step" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi

if [ -f /home/user/www/404_mod.shtml ]; then
        mv -f /home/user/www/404_mod.shtml /home/user/www/404.shtml
        echo "404_mod.shtml to 404.shtml renamed successfully" >> $logfile
else
        echo "MySql Backup failed at 404_mod.shtml to 404.shtml rename step" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi

if [ -d /home/user/www/main ] ; then
        mv -f /home/user/www/main /home/user/www/main_sleep
        echo "main to main_sleep directory rename successfully" >> $logfile
else
        echo "MySql Backup failed at main to main_sleep directory rename" >> ${logfile}
        /bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
        exit
fi


##this line wraps, fix it
/usr/bin/mysqldump database_name | /bin/gzip -c > /home/backups/backup_${mydate}.sql.gz

service mysql restart

mv -f /home/user/www/avb_sleep /home/user/www/avb
mv -f /home/user/www/404.shtml /home/user/www/404_mod.shtml
mv -f /home/user/www/404_original.shtml /home/user/www/404.shtml

/bin/cat /home/backups/${logfile} | mail -s MySql_Logs user@domain.com
0
 
mafairuzAuthor Commented:
thanks I will for sure
0
 
mafairuzAuthor Commented:
hi jesper

the script worked out perfectly .. files and folders got renamed, database got backedup as required.
However :), only the last command (sending mysql-backup.log contents by e-mail) did not work, and I got the following error:
=========================
/bin/cat: /home/backups//home/backups/mysql-backup.log: No such file or directory
Null message body; hope that's ok
==========================

The file /home/backups/mysql-backup.log does exist and its contents shows clearly that the process worked out.   Maybe the last command be modified to look like:
/bin/cat ${logfile} | mail -s MySql_Logs user@domain.com

Okey final issue, how to schedule this script to work out at 3 midnight (my time) every day using cron?
by the way, my time is 9 hours ahead server time .. for example, 3 midnight my time, is 6pm yesterday server time ..
This actually created a problem with backup file, the file was named (backup_04Aug07 instead of backup_05Aug07) which I think can be fixed in mydate definition line, but I don't know how.

just these two bits and I would be thankful for your time jasper
0
 
mafairuzAuthor Commented:
Thanks again jesper
When I run the script again after the changes you said, I got the following error :s

=================
/home/backups/bin/mysql-backup.sh: line 9: date+%d%b%y: command not found
=================

what should be the problem there !
0
 
Jan SpringerCommented:
That is saying that it can't find the date command.

Does the preceding line say:

date=`/bin/date`  

?
0
 
mafairuzAuthor Commented:
yes, here follows what I have in the shell script

===========
date=`/bin/date`
mydate=$(date+%d%b%y --date='Tomorrow')
===========
0
 
Jan SpringerCommented:
Do this as (any user is fine, root is shown as an example):

[root]# nano test-date

#!/bin/bash
date=`/bin/date`
mydate=$(date +%d%b%y --date='Tomorrow')
echo $mydate

<save the file>

[root]# chmod 755 test-date
[root]# ./test-date

What is the output?
0
 
Jan SpringerCommented:
Can you verify that 'date' lives in /bin?
0
 
Jan SpringerCommented:
Wait.  I don't see a space between $(date and the +
0
 
mafairuzAuthor Commented:
yes maybe the problem was with the space as the test.sh worked fine
let me change on the main script and try out .. it is off-peak now :)
just a moment
0
 
mafairuzAuthor Commented:
Thanks jesper .. :)
things are just perfect right now
0
 
Jan SpringerCommented:
Wonderful!  Anything else we need to do with this?
0
 
mafairuzAuthor Commented:
yes, I will add a new separate question for you later .. I will need to pass my backup file to another server using any way .. like ftp or anything else .. I hope there is a way for this to be done automaticlly

Thanks again
0
 
Jan SpringerCommented:
sounds good.  we can go that.  i'll set a filter for your name and watch for it.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.