Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Shell script to delete old backups

Posted on 2015-01-13
13
Medium Priority
?
241 Views
Last Modified: 2015-01-22
Hi,

I need a script to delete old backups created by Cobian Backup on a NAS (for some reason the program does not delete the old backups, gave up trying to find out why).

The NAS is running Synology 5, with ssh access enabled. The built-in shell is ash.

The backups are basically one full, followed by 0 or more incremental backups, then the next full one etc.
The backups are stored, for each user, in /volume1/homes/username/. See the structure below.

What I need is to delete, for each backed up folder, all the backups apart from the most recent full backup and its incremental backups. After the script runs, I should have, for each user and each folder, a single full backup and 0 or more incremental backups with a more recent date than the full backup.

How can I do that?

Thank you.

The file structure is as follows:
volume1/homes/user1/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
volume1/homes/user1/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder2 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder2 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder2 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder2 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder2 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user1/folder2 2014-xx-xx xx;xx;xx (Incremental)
volume1/homes/user1/folder2 2014-12-34 56;78;90 (Full)
volume1/homes/user1/folder2 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user2/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user2/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user2/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user2/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user2/folder1 2014-xx-xx xx;xx;xx (Incremental)
...
volume1/homes/user2/folder1 2014-xx-xx xx;xx;xx (Incremental)
volume1/homes/user2/folder1 2014-12-34 56;78;90 (Full)
volume1/homes/user2/folder1 2014-xx-xx xx;xx;xx (Incremental)
...

Open in new window

0
Comment
Question by:Dan Craciun
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 40546192
How you distinguish between full and incremental backups and are these file names or folder names?
0
 
LVL 35

Author Comment

by:Dan Craciun
ID: 40546197
The full backups end in (Full), the incremental ones in (Incremental). For ex, see below a real listing:

ServerBackup> ls -l
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-01 09;30;17 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-09 09;30;28 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-10 09;30;24 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-13 09;30;19 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-14 09;30;28 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-15 09;30;16 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-16 09;30;24 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-17 09;30;30 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-20 09;30;28 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-21 09;30;24 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-22 09;30;27 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-23 09;30;19 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-24 09;30;24 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-27 09;30;26 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-28 09;30;33 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-29 09;30;28 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-30 09;30;30 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-10-31 09;30;20 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-03 09;30;23 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-04 09;30;27 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-05 09;30;17 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-06 09;30;24 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-07 09;30;27 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-11 09;30;20 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-12 09;30;30 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-13 09;30;24 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-14 09;30;20 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-17 09;30;26 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-18 09;30;26 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-19 09;30;17 (Incremental)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-11-20 09;30;17 (Full)
drwxrwxrwx    3 user1   users         4096 Nov 25  2009 Profiles 2014-12-19 09;30;27 (Incremental)

Open in new window

0
 
LVL 84

Expert Comment

by:ozo
ID: 40546198
should
volume1/homes/user1/folder1 2014-xx-xx xx;xx;xx (Incremental)
be deleted because it is older than
volume1/homes/user1/folder2 2014-12-34 56;78;90 (Full)
or should it be kept because it is more recent than
volume1/homes/user1/folder1 2014-12-34 56;78;90 (Full)
?

i.e. is
volume1/homes/user1/folder2 2014-12-34 56;78;90 (Full)
part of the same backup set as
volume1/homes/user1/folder1 2014-12-34 56;78;90 (Full)
or is it part of a separate backup set that should be pruned separately?
0
Learn how to optimize MySQL for your business need

With the increasing importance of apps & networks in both business & personal interconnections, perfor. has become one of the key metrics of successful communication. This ebook is a hands-on business-case-driven guide to understanding MySQL query parameter tuning & database perf

 
LVL 35

Author Comment

by:Dan Craciun
ID: 40546306
It's a separate backup. Each user has 1 or more folders that they chose to backup, i.e. Desktop, Documents, etc, so for ex folder1 is Desktop, folder 2 is Documents, folder 3 is Pictures etc. And each can have it's own schedule.
0
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 40546369
perl -e 'for( @ARGV ){
  my %b;
  m#(.*?)\s\d{4}-\d\d-\d\d\s\d\d;\d\d;\d\d\s\((Full|Incremental)\)# && unshift @{$b{$1}},$_ for grep -d, <"$_"/*>;
  for( values %b ){
     my($f,$i);
     system "rm","-f",$_ for grep {$i|=$f,$f|=/\(Full\)$/,$i}@$_;
  }
}' volume1/homes/*
0
 
LVL 10

Expert Comment

by:Carlos Ijalba
ID: 40546421
At the end of the day, what you should prune more often will be the full backups, since they will be the biggest hogs.

So why not just do a find to delete fulls older than x days (in this example older than 30 days):

find /volume1/homes/user1 -name *Full* -mtime +30 | xargs rm

Open in new window


For incrementals, you could do similar:

find /volume1/homes/user1 -name *Incremental* -mtime +30 | xargs rm

Open in new window


It's not exactly what you asked, but it uses KISS methodology (Keep It Short and Simple).
0
 
LVL 35

Author Comment

by:Dan Craciun
ID: 40546871
@Carlos: if you notice, all the folder have the same date: Nov 25  2009. That's probably the reason why Cobian does not delete old backups (it's searching for timestamps, not folder names).

@ozo: thank you. I'll try to understand the script first, will post back with what I could not figure out by myself. I'm more of a powershell guy, so Perl is a little greek to me.
0
 
LVL 35

Author Comment

by:Dan Craciun
ID: 40547324
Here's what I got so far:
perl -e 'for( @ARGV ){  # create a short script (-e). 
#Run the block for each of the arguments array (basically for each folder in /volume1/homes/). 
#The current folder name (/volume1/homes/user1) will be stored in $_
  my %b; # declare a hash b. Does it have an initial value?
  m#(.*?)\s\d{4}-\d\d-\d\d\s\d\d;\d\d;\d\d\s\((Full|Incremental)\)# && unshift @{$b{$1}},$_ for grep -d, <"$_"/*>;
#then it becomes fuzzy: for each file in the current path ($_/*) search for names like folder2 2014-12-34 56;78;90 (Full)
#and then prepend the current path (/volume1/homes/user1 for ex) to what array??? $b{$1} contains what? 
# How is the hash constructed?
  for( values %b ){      #for each value of the hash b
     my($f,$i);               #declare 2 local variables, $f and $i. Are they the (key,value) pair for b?
     system "rm","-f",$_ for grep {$i|=$f,$f|=/\(Full\)$/,$i}@$_;
#again, fuzzy: remove (run rm -f) for each file that matches the grep. But I cannot figure out the grep. It reverses the 
#file order and preserves only those after the first folder with "(Full)" in the name?
  }
}' /volume1/homes/*

Open in new window


I tested the script (replacing system "rm", "-f" with system "echo") and it seams to work. But I can't still figure out why :)

Can you enlighten me, ozo, or shall I post this as a new question?
0
 
LVL 84

Expert Comment

by:ozo
ID: 40547930
# declare a hash b. Does it have an initial value?
It is initially empty
# $b{$1} contains what?
for names like folder2 2014-12-34 56;78;90 (Full)
$1 would contains "folder2"
@{$b{$1}} would contain a list associated with folder2

# $f and $i. Are they the (key,value) pair for b?
$_ is the value for b
@{$_} is the list of names in that value
$f is set the first [or last in date order] time we see a (full) backup
$i is set on the next name we see
0
 
LVL 35

Author Comment

by:Dan Craciun
ID: 40555393
Thank you Ozo. I tested the script and it works, with minor tweaks (it's /volume1 and "-rf").

The delete part takes a long time, so I would like to print the name of the folder being deleted as a "proof of life", so I know the script is still running.
Would this work?
perl -e 'for( @ARGV ){
  my %b;
  m#(.*?)\s\d{4}-\d\d-\d\d\s\d\d;\d\d;\d\d\s\((Full|Incremental)\)# && unshift @{$b{$1}},$_ for grep -d, <"$_"/*>;
  for( values %b ){
     my($f,$i);
     for (grep {$i|=$f,$f|=/\(Full\)$/,$i}@$_) {
         system "echo", "Deleting $_";
         system "rm","-rf",$_ ;
     }
  }
}' /volume1/homes/*

Open in new window

0
 
LVL 84

Expert Comment

by:ozo
ID: 40555500
Yes, although
print "Deleting $_\n";
may be easier.
0
 
LVL 10

Expert Comment

by:Carlos Ijalba
ID: 40558668
It probably takes a while deleting since it does multiple "rm file" commands.
It would be a lot faster if you issue a "xargs | rm" command.
0
 
LVL 35

Author Closing Comment

by:Dan Craciun
ID: 40565026
Thank you.
0

Featured Post

Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
This tutorial will show how to configure a single USB drive with a separate folder for each day of the week. This will allow each of the backups to be kept separate preventing the previous day’s backup from being overwritten. The USB drive must be s…
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…

664 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question