Solved

Write linux script for automated backup and restore

Posted on 2010-09-04
12
879 Views
Last Modified: 2013-12-15

I have fedora 13 64bit on a sata 320 GB hard disk (sda). I have a second 512 GB sata hard disk sdb.

I want to partition sdb in such way that I have about 2 50 gb partitions of which one is ntfs for windows installation (mounted on fedora as /media/cdrive ) and other 50 gb for linux use from fedora. The remaining space 412 GB is used for backup as detailed below.
LVM scheme can be used on sdb. Lets call this 412 GB physical partition as sdb3.

Please write a backup script that would achieve the following tasks:

1) Runs automatically every night at 3 am on fedora if fedora machine is on.
 
2) Makes bit for bit disk image or snapshot of entire /dev/sda used space ( not the free space on /dev/sda) and save on sdb3 disk.

3) Must do some type of compression like gzip.

4) The program should backup in such a way that sdb3 contains disk images of most recent 5 days ( 5 disk images). That means on day 6, the disk image that was created on day 1 is overwritten and on day 7 , disk image that was created on day 2 is overwritten and so on. Should also account for days in which power went out or simply the system was off and skip that day automatically.

5) This program must be reliable to withstand sudden power outage in case that happens while its running.


Please write a restore script that would achieve the following tasks:

In case sda is fried up, I install a new hard disk in its place and boot the system using any linux live cd like fedora or ubuntu. From this live cd I access and run the restore script.
Restore script should achive the following:

1) I can choose which disk image ( of the 5 images in sdb3)  to use as the source for restore
2) sda now should have exactly same partions/logical volumes and data at the time the backup program above created
the disk image.
3) after this I remove the live cd and boot the system and should be able to boot to fedora.



0
Comment
Question by:zenguru
  • 4
  • 2
  • 2
  • +4
12 Comments
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 33605169
Is this homework?
0
 

Author Comment

by:zenguru
ID: 33609262
No its not.
0
 
LVL 7

Expert Comment

by:greg_vander
ID: 33609606
so you better take a dedicated backup product like veritas Netbackup (symantech).
0
 

Author Comment

by:zenguru
ID: 33612419
Please provide the solution exactly as described instead pointing me to enterprise products costing $1000s. I dont need that for this purpose as you should know.

I expect some one who is expert in this subject matter ( an expert having extensive experience
in linux backup and restore -- if he did not extensive experience he would not be called
expert) in 15 minutes or less or he may already have such a script that needs some editing
to achieve the solution to the problem.

Please do not post comment unless you have extensive experience in linux based backup and restore.
0
 
LVL 4

Expert Comment

by:rajendraone
ID: 33619087
Read through some of your other posts too. What an attitude !
Now good luck with your answers.
0
 
LVL 18

Expert Comment

by:TobiasHolm
ID: 33621525
Hi!

Look at Bacula. It's a free backup program that's very powerful.

Ref: http://www.bacula.org/en/

Regards, Tobias
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:zenguru
ID: 33622214
about bacula:  its from bacula website:

 However, if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, the Bacula project does not recommend using Bacula as it is much more difficult to setup and use than tar or dump.

so it does not suit my needs. Thanks anyway. Again please do not point me to complex enterprise programs. I just need that linux script which will meet the conditions given.

0
 
LVL 14

Expert Comment

by:cjl7
ID: 33628364
Dear zenguru,

You should probably ask a bit more specific question and not ask people to do all of your work for you. And also not bash people if they recommend other solutions then the ones you had in mind. (IMHO)

But anyway...

"A simple script" - if you want to do a bit-wise comparison there are no simple things available. If you are OK with a copy of the data, then your script might look like this:

(see code)

I haven't tested this at all, it's up to you...


#!/bin/bash
DATE=$(date %Y%m%d)
TARGET=/my/backup/device
SNAP=${TARGET}/snap
mkdir -p ${TARGET}/${DATE}
mkdir -p ${SNAP}

# main backup
rsync -avz /source/* /${TARGET}/${DATE}/

# do "snapshot" if successful

if [ $? -eq 0 ]; then
tar -zcf ${SNAP}/${DATE}.tar.gz ${TARGET}/${DATE}
else;
echo "didnt work"
fi

Open in new window

0
 
LVL 25

Expert Comment

by:madunix
ID: 33644205
you can use multiple tools to image/backup linux system  in my datacenter, i use the following:
1- use Mondo Rescue to clone linux systems:
http://www.mondorescue.org

2- You can use dd also. dd if=/dev/somedisk of=/dev/somedestination

3- rsync scripting

4- netcat
On the source server: (cd / && tar -cvf - ) | nc 192.168.1.2 2000
On the destination:: nc -l -p 2000 | tar -xvf -

5- http://sourceforge.net/projects/g4l/

6- http://www.bacula.org/en/

7- http://clonezilla.org/

8- ibm tools
http://www-01.ibm.com/software/tivoli/products/prov-mgr-os-deploy/
IBM® Tivoli® Provisioning Manager for OS Deployment helps organizations reduce the time and costs associated with desk-side support and direct interaction with target systems.

http://www-01.ibm.com/software/tivoli/products/storage-mgr/
IBM Tivoli® Storage Manager 6 provides a wide range of storage management capabilities from a single point of control, helping companies ride the information tidal wave.
0
 
LVL 25

Accepted Solution

by:
madunix earned 500 total points
ID: 33644216
as said rsync is a good tool to build a relation between master and slave, for example my qmail server master and slave are replucated by
copying  these directories:
- - /home/vpopmail/
- - /var/qmail/control/
- - /var/qmail/alias
and other directories that exists in both the servers and needs to be replicated, using rsync command:
rsync -avz -e ssh -R /home/vpopmail user@host:/home/rsync/backupdir

on the other-hand it depends on you application, If the files and file systems are too heavy I/O _very_ busy or locked by applications copy/ftp/rsync... commands are not useful specially for DB and it is better to use mirror/shadow/standby features of database.

regarding rsync i have created my own script to move data, feel free to change it
the script i have tested on multiple Linux server RHEL, Centos

#!/bin/bash
#  This script uses Rsync the backup files
#  to a remote server. To use this script
#  you first have to setup ssh to use keys.
#  This elimates the need for a password but
#  still provides security I have ssh on
#  a non standard port so I had to specify
#  the port (ssh -p 2998).
##########################################
#  Simply change the variables list below
#  to match your specifications.
##########################################
#  The Script also logs the transfer time.
#  For automated backup add script to crontab.
# Setup Variables
SOURCEPATH="/../source/"
DESTPATH="/../target/"
DESTHOST="10.6.40.81"
#The user you created with ssh_keys
DESTUSER="root"
LOCKFILE="/var/lock/rsyncbackup.lock"
LOGFILE="/var/log/rsyncbackup.log"
STARTTIME=$(date +"%s")
# Verfiy the last rsync is not still running
if [ -f $LOCKFILE ]; then
ERRORTIME=$(date +"%d/%b/%Y:%H:%M:%S %Z")
echo "[$ERRORTIME] WARNING rsyncbackup.lock exists, aborting..." >> $LOGFILE
exit 1
fi
# If it finished, create a new lock file
touch $LOCKFILE
# The rsync command (for explanation type rsync --help)
rsync -aqrpogtu --delete --bwlimit=512 --rsh='ssh -p 22' $SOURCEPATH $DESTUSER@$DESTHOST:$DESTPATH 2>&1 >> $LOGFILE
# Log when the script was run
DONE=$(date +"%d/%b/%Y:%H:%M:%S %Z")
# Log how long it took the script to run
STOPTIME=$(date +"%s")
SECONDS=$(($STOPTIME - $STARTTIME))
echo "[$DONE]finished: - Tranfer took:$(((SECONDS/60)/60)) hours $(((SECONDS/60)%60)) minutes $((SECONDS%60)) seconds" >> $LOGFILE
# Remove lock file
rm -f $LOCKFILE
# END
#

Open in new window

0
 

Author Comment

by:zenguru
ID: 33653076
Thanks for ur post madunix.  Does rsynch creates compressed disk image of the source drive and does it include bit for bit copy including partitions and MBR ( master boot record)  while source is live and running ( not offline mode ) ?
0
 
LVL 18

Expert Comment

by:TobiasHolm
ID: 33654568
>Does rsynch creates compressed disk image

No.

>does it include bit for bit copy including partitions and MBR

No.

rsync copies files just as they are. No compression are made to the destination file, no partition copy are available.

Regards, Tobias
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now