Avatar of PearsonAus
PearsonAus asked on

TSM Differentials and Expiration

Hi,

We are running TSM 5.5 and have been running NDMP Backups for our EMC Isilon NAS.
Due to the amount of data that we have (over 100TB) we are unable to schedule in a full backup due to time constraints.
Our original full backup took around 6 days to complete with 2 dedicated Tape Drives.

At present, we are solely running Differential backups and are quickly filling up the tapes.
Does expiry take place on older Differential backups, or does a Full need to take place to implement this?

Below is our copygroup.

Policy        Policy        Mgmt          Copy          Versions     Versions       Retain      Retain
Domain        Set Name      Class         Group             Data         Data        Extra        Only
Name                        Name          Name            Exists      Deleted     Versions     Version
STANDARD      ACTIVE        ISIMC         STANDARD             2            1           30          60

Thanks
Storage SoftwareStorage

Avatar of undefined
Last Comment
PearsonAus

8/22/2022 - Mon
woolmilkporc

Hi,

this is a bit complicated. TSM cannot expire single files inside of a NAS dump image, only the complete image. Reason: TSM would have to create an own server object for each and every file in the dump, which it doesn't for reasons like performance and DB space consumption. Note: File level restore is possible nonetheless, by reading the TOC.

Full and differential backups are grouped, and the full backup is kind of a group leader ("Delta Group Leader" in TSM terms).

You have VEREXISTS = 2, so when you have performed one full and one differential backup then the next differential backup (version 3) will expire the full backup (version 1) ... BUT TSM must keep it internally, because it's needed to restore any of the differential backups.

The next (third) differential (version 4) will then expire the first differential backup after the full backup (version 2), etc, and this image really vanishes from TSM storage in the course of inventory expiration and space reclamation.

Short, the TSM server may store a full backup in excess of the number of versions you specified, and this full backup will stay in the TSM DB (and on media) until all dependent backups have expired.

Again, in other words, with your setting of VEREXISTS=2, when you run one full backup followed by differentials only then you will always have one (hidden) full backup and two active differential backups at a given point in time in TSM storage.

Note: Although the full backup image is still present in TSM you cannot perform a point in time restore from the date of this (in fact expired) full backup. You can only perform PIT restores from full or differential backups that are active (have not yet expired).

Expiry is performed by a background process which runs every "ExpInterval" hours (run Q OPT under dsmadmc to check this value).
If "ExpInterval" is 0 (zero) you must schedule an EXPIRE INVENTORY command on your own, by means of an Admin Schedule.

I hope I could help clarifying things a bit. As I said, it's complicated ...

wmp
ASKER
PearsonAus

great, thanks for that.
Yes this all makes sense.

Yes expiration runs daily, ExpInterval is set to 24.

I have noticed however that when I run a "q vol NAME f=d" the "Scratch Volume?:" is set to "No" on many of the NDMP pool tapes. Could this be a reason as to why data may not be expiring correctly, and thus using up many tapes?
I think some tapes may have been manually defined with "def vol" as opposed to letting the NDMP pool pickup from available libvolume SCRATCH's
ASKER CERTIFIED SOLUTION
woolmilkporc

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Thomas Rush

I know you've probably invested heavily in TSM, but it might be worth your while when your next renewal comes up to consider HP's Data Protector.  It also allows you to perform a backup to disk in a single full, then incremental forever mode.  It keeps track of the pointers and can expire older files and reclaim space easier, it sounds, than TSM can.

With either Data Protector or TSM, remember that you'll want to create periodic Synthetic Full Backups -- typically to tape -- that do a lot of backend magic that resets pointers and keeps things organized and optimized, and which will allow you to do quicker restores, as well as have an economical second (or third) copy of your backups to store off site.

http://www.hp.com/go/dataprotector
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
ASKER
PearsonAus

Great. Just wanted to make sure this was the case.
Thanks for your response. As most of those tapes in the NDMP pool contain data, I am running move data for each, and subsequently will del vol and upd libv to scratch.
ASKER
PearsonAus

Thanks SelfGovern,
This isn't an option for us, as the AUS TSM Servers are part of a global pool of about 30 TSM Servers.