Solved

BASH, how to delete backup files on a regular basis ?

Posted on 2009-03-29
28
551 Views
Last Modified: 2012-05-06
Hello,

I have a bash script which runs mysqldump daily,

files are named mydbname_`date +%m_%d_%y`.gz
I need to add to my script a feature in which if the number of backups for that db reaches 10 AND there are no currpted files or size 0 files in that dir, then the oldest created backup file gets deleted.

Please alter the script below so it does that:
#!/bin/bash

OUTFILE="/mnt/nas/backups/db_`date +%m_%d_%y`.gz"

MAILTO=myemail@address.com

SERVER=$(uname -n)

mysqldump -hLocalhost -uMyUser -pMyPass--opt --databases --single-transaction MyDB | gzip > $OUTFILE

MSG=$(ls -l $OUTFILE |awk '{print $5/1024/1024, "MB", $9}')

echo $MSG $(date) | mailx -s "Backups on $SERVER" $MAILTO

Open in new window

0
Comment
Question by:tzvish
  • 14
  • 8
  • 5
  • +1
28 Comments
 
LVL 48

Expert Comment

by:Tintin
ID: 24014517

#!/bin/bash

BACKUPDIR=/mnt/nas/backups

OUTFILE="$BACKUPDIR/db_`date +%m_%d_%y`.gz"

MAILTO=myemail@address.com

SERVER=$(uname -n)

mysqldump -hLocalhost -uMyUser -pMyPass--opt --databases --single-transaction MyDB | gzip > $OUTFILE

MSG=$(ls -l $OUTFILE |awk '{print $5/1024/1024, "MB", $9}')

echo $MSG $(date) | mailx -s "Backups on $SERVER" $MAILTO
 

find $BACKUPDIR -name "db_*gz" -size +0 | xargs ls -t >/tmp/$$

[ $(wc -l </tmp/$$) -gt 10 ] && rm -f $(tail -1 /tmp/$$)

rm -f /tmp/$$

Open in new window

0
 
LVL 23

Expert Comment

by:Maciej S
ID: 24014570
In such cases (backups, which can run 1 hour sometimes, but sometimes 1 or more days) I don't trust timestamps. I rather use date from filename.

Unfortunately you have date in month_day_year format, which complicates below script a little. I would be much easier if you use date in year_month_date format. Anyway - my version below (checks if gzipped file is correct with gzip -t). It leaves last 10 correct files.
#!/bin/sh
 

TMP_FILE=/tmp/dates.$$

if [ -e "${TMP_FILE}" ]; then

   rm -f ${TMP_FILE}

fi 
 

for file in `ls -1 mydbname*`; do

   DATE=`echo ${file} | sed 's/.*_\([0-9][0-9]\)_\([0-9][0-9]\)_\([0-9][0-9]\).gz/\3\1\2/'`

   echo "${DATE}:${file}" >> ${TMP_FILE}

done
 

sort ${TMP_FILE} > ${TMP_FILE}.sorted
 

COUNTER=0

for file in `cat ${TMP_FILE}.sorted`; do

   file2=`echo ${file} | cut -d: -f2-`

   if [ ${COUNTER} -gt 10 ]; then

      rm -f ${file2}

   fi 

   gzip -t ${file2} 2> /dev/null && COUNTER=$(( COUNTER + 1 )) || echo File ${file2} corrupted.

done
 

rm ${TMP_FILE}

rm ${TMP_FILE}.sorted

Open in new window

0
 
LVL 1

Author Comment

by:tzvish
ID: 24014897
oklit, i don't mind changing the format of the file name, does the procedure you supplied work with my naming convention or another ?

how do i use it ?
0
 
LVL 23

Expert Comment

by:Maciej S
ID: 24014961
It works with your current filename format.
Just add this to your current script (before mysqldump or after - doesn't matter) or make it separate script - as you wish.
If you add above script to your current one after mysqldump command, your just created backup will be counted of course. If you add it before mysqldump command, it will be counted in next run of the script.

For testing purposes you may add "echo" before "rm -f ${file2}" (line 19) and run it - you will se what files would be deleted. If it is ok for you, remove echo.

I just realized that I made mistake in above code - I used "sort", but there should be "sort -r" (line 13).

With changed filename format, this script can have ca. 50% of size/number of lines of current one.
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24016488
What action do you want to take when there is any zero sized file in the directory?
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24017593
You can use the following code to do this.
I guess this is quite explanatory.
Let me know, if you want any more help/detail on this.

FileCount=`ls "db_*gz" | wc -l`

if [[ $FileCount -gt 10 ]]

then

	# Checking below if there is any file of zero sized using flag NoFileZeroSized        

	NoFileZeroSized=0;	

	for file in `ls "db_*gz"*`

	do

		if [ ! -s $file ]

		then

			NoFileZeroSized=1;

		fi

	done

	if [ $NoFileZeroSized -eq 0 ]

	then

	#If there are no zero sized file as well, then get the oldest file and delete it.

	        FileToDelete=`ls -t "db_*gz"* | tail -1`

		rm -f $FileToDelete;

	fi

fi

Open in new window

0
 
LVL 1

Author Comment

by:tzvish
ID: 24017646
Sorry, for not closing tis yet - had to sleep, will cose this up today
Zero sized files send echo error and are not counted whitin the 10 backup files listing
0
 
LVL 1

Author Comment

by:tzvish
ID: 24020908
oklit,

In your script, anything i need to specify besides mydbname?

Where do i insert the path ?
0
 
LVL 23

Expert Comment

by:Maciej S
ID: 24021889
You mean path to mydbname* file(s)?
0
 
LVL 1

Author Comment

by:tzvish
ID: 24022389
yep
0
 
LVL 23

Accepted Solution

by:
Maciej S earned 251 total points
ID: 24022567
You m ay add another variable to above script, to make in easier to change in the future.
For clarity corrected script below. (echo added before rm - for safety).
With below version, just change "/some/directory/" in BACKUP_FILES variable (leaving mydbname* part of course).
#!/bin/sh
 

BACKUP_FILES="/some/directory/mydbname*"

TMP_FILE=/tmp/dates.$$

if [ -e "${TMP_FILE}" ]; then

   rm -f ${TMP_FILE}

fi
 

for file in `ls -1 ${BACKUP_FILES}`; do

   DATE=`echo ${file} | sed 's/.*_\([0-9][0-9]\)_\([0-9][0-9]\)_\([0-9][0-9]\).gz/\3\1\2/'`

   echo "${DATE}:${file}" >> ${TMP_FILE}

done
 

sort -r ${TMP_FILE} > ${TMP_FILE}.sorted
 

COUNTER=0

for file in `cat ${TMP_FILE}.sorted`; do

   file2=`echo ${file} | cut -d: -f2-`

   if [ ${COUNTER} -gt 10 ]; then

      echo rm -f ${file2}

   fi

   gzip -t ${file2} 2> /dev/null && COUNTER=$(( COUNTER + 1 )) || echo File ${file2} corrupted.

done
 

rm ${TMP_FILE}

rm ${TMP_FILE}.sorted

Open in new window

0
 
LVL 1

Author Comment

by:tzvish
ID: 24023903
Thanks oklit,

I removed the echo on the rm command and the files were successfully deleted,
I created this test directory where i had 12 copies of the same mysqldump file just to try out the script, and it successfully deleted the oldest files.

However, i did get 2 messeges of currupt files,
these were the files that got deleted by script:
File /mnt/nas/backups/mydb_03_30_09.gz corrupted.
File /mnt/nas/backups/mydb_03_29_09.gz corrupted.

this must have something to do with this line:
   gzip -t ${file2} 2> /dev/null && COUNTER=$(( COUNTER + 1 )) || echo File ${file2} corrupted.

i know the script has deleted them but why was were the files specefied as currupt ?

0
 
LVL 23

Expert Comment

by:Maciej S
ID: 24026704
Are you sure, that above script deleted this two corrupted files? It shouldn't. It searches for first 10 good files (corrupted/zero sized files are not counted, but not deleted also). After finding 10 good files (with possibly some corrupted ignored), next files are deleted.
Anyway - "File ... corrupted" message is displayed, when gzip -t (-t is for test - man gzip) exits with some error (so archive isn't ok). It may mean that file is not gzip archive at all (for example zero sized file). I don't know exactly what gzip checks - in man there is only info "Check the compressed file integrity." If it is unsuccessful, then (in script) COUNTER variable is not increased, and this message shows up (but the file is not deleted - if you want to delete it, then we need to make little modification of one line).
0
 
LVL 1

Author Comment

by:tzvish
ID: 24038506
Yep,

Tried it again in the past 24 hours - it seems like that the file which gets deleted for being the oldest, also gets reported for being currupted.

for example - this is the backup directory after the backup script - the first files were duplicated to create a 12 files folder:

-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_21_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_22_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_23_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_24_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_25_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_26_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:45 mydbname_03_27_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:53 mydbname_03_28_09.gz
-rw-r--r-- 1 root     root     971K Mar 30 16:44 mydbname_03_29_09.gz
-rw-r--r-- 1 mydbname mydbname 971K Mar 30 17:10 mydbname_03_30_09.gz
-rw-r--r-- 1 mydbname mydbname 975K Mar 31 17:10 mydbname_03_31_09.gz

And I got this email when the proccess finished:
File /mnt/nas/backups/mydbname_03_20_09.gz corrupted.
(that file was also deleted)

I tried to run gzip -t on the files and they aren't currpted, then why is this happening?

0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24038772
did you try my script? Do you want any changes in that?
0
 
LVL 1

Author Comment

by:tzvish
ID: 24040625
I'd check it out, didn't know there's a gzip -t feature it would be great if it was integrated in your script, along with 0 sized files.

i'd five all 500 points to which ever script gets chosen eventually and to the first one that works
0
 
LVL 5

Assisted Solution

by:vikaskhoria
vikaskhoria earned 249 total points
ID: 24056875
After reading all the comments, I guess I am more clearer on the requirement:
If the there are more than 10 GOOD files, then delete the oldest file.
GOOD here means non-zero and not corrupt file.
Based on this, I have changed my script above (Have also considered gzip -t this time).
Try this script and let me know, if you have any other issue or queries.

Vikas.
fileCount=0; # Counter for no. of files

for file in `ls -t "db_*gz"*`

do

        if [ ! -s $file ] # Checking if file is zero sized.

        then

		# Checking if file is corrupted.

		if [ gzip -t ${file2} 2> /dev/null ] 

		then

			fileCount=$(( fileCount + 1 )) # Incresing count of files, if okay

			oldestFileToDelete=$file;      # Keeping track of oldest GOOD file.

		else

			echo File ${file} is corrupted.			

		fi
 

        fi

done

# Now if the number of Non-Zero and Non-Corrupted files is more than 10, then delete the oldest GOOD File.

if [ $fileCount -gt 10 ]

then

        rm -f $oldestFileToDelete;

fi

Open in new window

0
 
LVL 1

Author Comment

by:tzvish
ID: 24057602
Thanks,

Gonna try this out now - where do i insert the file path ?
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24057752
replace db_*gz at ls -t "db_*gz"*, in the script with you path / filename.
So that line (no. 2) should look something like this:

for file in `ls -t /your/path/"db_*gz"`

db_*gz lists all the concerned files in the given path.
0
 
LVL 1

Author Comment

by:tzvish
ID: 24057763
Ok, thanks - trying it out now with this:
for file in `ls -t /mnt/nas/backups/"mydb_*gz"`
0
 
LVL 1

Author Comment

by:tzvish
ID: 24057795
weird, when i run ths from bash it works, maybe i got the double qoutes misplaced in :
for file in `ls -t /mnt/nas/backups/"mydb_*gz"`

ls: /mnt/nas/backups/mydb_*gz: No such file or directory

0
 
LVL 1

Author Comment

by:tzvish
ID: 24057812
I tried to remove the double quotes completely and didn't get any error message back from cron, however - no files were deleted from the directory
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24057844
What does you file name look like?
I had taken as an EXAPLE mydb_*gz
It depends on what regex you want.
OKAY! I just checked you file list above, is not working because of the following reason:
What we are doing is mydb_*
So it looking for a _ after mydb, thus looking for a name like say 'mydb_jhjsjh.gz'
(This is because I took and example name!! :))
Change that to mydbname_*.gz and it will work.

So, now the line should look like:
for file in `ls -t /mnt/nas/backups/"mydbname_*.gz"`
 
I hope I made my point clear.


Cheers!
vikas.
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24057859
Its because no files are matching our expression
You can remove the "", that does not matter much in this case.
0
 
LVL 1

Author Comment

by:tzvish
ID: 24057928
mydbname or mydb is psedu / container name for what i real use to call the database,

running ls -t /mnt/nas/backups/mydbname_*.gz
in bash shows all the file result set
0
 
LVL 5

Expert Comment

by:vikaskhoria
ID: 24058109
Yes, if this shows all the file result set, then its perfect. Right?
Did you run the script with this line, now?

Now it should work fine!
0
 
LVL 1

Author Comment

by:tzvish
ID: 24067688
Sorry but this doesn't work, Nope, the script doesn't delete the last file
0
 
LVL 1

Author Closing Comment

by:tzvish
ID: 31566598
Both solutions had problems, i fixed oklit, it had the curruption check line misplaced after file was supposed to be deleted, moved it before the deletion and the script now works great.
I couldn't get vikaskhoria's script to work, but it seems like it can with a couple of tweaks.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
In Solr 4.0 it is possible to atomically (or partially) update individual fields in a document. This article will show the operations possible for atomic updating as well as setting up your Solr instance to be able to perform the actions. One major …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now