This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.
Course Of The Month
8 days, 20 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
TSM Differentials and Expiration
Posted on 2014-01-14
Hi,
We are running TSM 5.5 and have been running NDMP Backups for our EMC Isilon NAS.
Due to the amount of data that we have (over 100TB) we are unable to schedule in a full backup due to time constraints.
Our original full backup took around 6 days to complete with 2 dedicated Tape Drives.
At present, we are solely running Differential backups and are quickly filling up the tapes.
Does expiry take place on older Differential backups, or does a Full need to take place to implement this?
Below is our copygroup.
Policy Policy Mgmt Copy Versions Versions Retain Retain
Domain Set Name Class Group Data Data Extra Only
Name Name Name Exists Deleted Versions Version
STANDARD ACTIVE ISIMC STANDARD 2 1 30 60
Thanks
We are running TSM 5.5 and have been running NDMP Backups for our EMC Isilon NAS.
Due to the amount of data that we have (over 100TB) we are unable to schedule in a full backup due to time constraints.
Our original full backup took around 6 days to complete with 2 dedicated Tape Drives.
At present, we are solely running Differential backups and are quickly filling up the tapes.
Does expiry take place on older Differential backups, or does a Full need to take place to implement this?
Below is our copygroup.
Policy Policy Mgmt Copy Versions Versions Retain Retain
Domain Set Name Class Group Data Data Extra Only
Name Name Name Exists Deleted Versions Version
STANDARD ACTIVE ISIMC STANDARD 2 1 30 60
Thanks
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
3
2
6 Comments
Active today
Hi,
this is a bit complicated. TSM cannot expire single files inside of a NAS dump image, only the complete image. Reason: TSM would have to create an own server object for each and every file in the dump, which it doesn't for reasons like performance and DB space consumption. Note: File level restore is possible nonetheless, by reading the TOC.
Full and differential backups are grouped, and the full backup is kind of a group leader ("Delta Group Leader" in TSM terms).
You have VEREXISTS = 2, so when you have performed one full and one differential backup then the next differential backup (version 3) will expire the full backup (version 1) ... BUT TSM must keep it internally, because it's needed to restore any of the differential backups.
The next (third) differential (version 4) will then expire the first differential backup after the full backup (version 2), etc, and this image really vanishes from TSM storage in the course of inventory expiration and space reclamation.
Short, the TSM server may store a full backup in excess of the number of versions you specified, and this full backup will stay in the TSM DB (and on media) until all dependent backups have expired.
Again, in other words, with your setting of VEREXISTS=2, when you run one full backup followed by differentials only then you will always have one (hidden) full backup and two active differential backups at a given point in time in TSM storage.
Note: Although the full backup image is still present in TSM you cannot perform a point in time restore from the date of this (in fact expired) full backup. You can only perform PIT restores from full or differential backups that are active (have not yet expired).
Expiry is performed by a background process which runs every "ExpInterval" hours (run Q OPT under dsmadmc to check this value).
If "ExpInterval" is 0 (zero) you must schedule an EXPIRE INVENTORY command on your own, by means of an Admin Schedule.
I hope I could help clarifying things a bit. As I said, it's complicated ...
wmp
this is a bit complicated. TSM cannot expire single files inside of a NAS dump image, only the complete image. Reason: TSM would have to create an own server object for each and every file in the dump, which it doesn't for reasons like performance and DB space consumption. Note: File level restore is possible nonetheless, by reading the TOC.
Full and differential backups are grouped, and the full backup is kind of a group leader ("Delta Group Leader" in TSM terms).
You have VEREXISTS = 2, so when you have performed one full and one differential backup then the next differential backup (version 3) will expire the full backup (version 1) ... BUT TSM must keep it internally, because it's needed to restore any of the differential backups.
The next (third) differential (version 4) will then expire the first differential backup after the full backup (version 2), etc, and this image really vanishes from TSM storage in the course of inventory expiration and space reclamation.
Short, the TSM server may store a full backup in excess of the number of versions you specified, and this full backup will stay in the TSM DB (and on media) until all dependent backups have expired.
Again, in other words, with your setting of VEREXISTS=2, when you run one full backup followed by differentials only then you will always have one (hidden) full backup and two active differential backups at a given point in time in TSM storage.
Note: Although the full backup image is still present in TSM you cannot perform a point in time restore from the date of this (in fact expired) full backup. You can only perform PIT restores from full or differential backups that are active (have not yet expired).
Expiry is performed by a background process which runs every "ExpInterval" hours (run Q OPT under dsmadmc to check this value).
If "ExpInterval" is 0 (zero) you must schedule an EXPIRE INVENTORY command on your own, by means of an Admin Schedule.
I hope I could help clarifying things a bit. As I said, it's complicated ...
wmp
great, thanks for that.
Yes this all makes sense.
Yes expiration runs daily, ExpInterval is set to 24.
I have noticed however that when I run a "q vol NAME f=d" the "Scratch Volume?:" is set to "No" on many of the NDMP pool tapes. Could this be a reason as to why data may not be expiring correctly, and thus using up many tapes?
I think some tapes may have been manually defined with "def vol" as opposed to letting the NDMP pool pickup from available libvolume SCRATCH's
Yes this all makes sense.
Yes expiration runs daily, ExpInterval is set to 24.
I have noticed however that when I run a "q vol NAME f=d" the "Scratch Volume?:" is set to "No" on many of the NDMP pool tapes. Could this be a reason as to why data may not be expiring correctly, and thus using up many tapes?
I think some tapes may have been manually defined with "def vol" as opposed to letting the NDMP pool pickup from available libvolume SCRATCH's
Active today
Volumes assigned to a storage pool by means of DEF VOL are for exclusive use in that storage pool.
Expiration and space reclamation will take place nonetheless, but the tapes in question will not return to the scratch pool but will stay in their storage pool in an "Empty" state.
You can check whether a storage pool is allowed to use additional scratch volumes or whether it is limited to use its assigned volumes with Q STGPOOL <poolname> F=D. The MAXSCRATCH parameter defines the number of scratch tapes a pool can request.
You can check with Q VOL STGPOOL=<poolname> ST=empty how many empty volumes are available to the pool for future use.
To return empty volumes to the scratch pool unassign them from their storage pool with
DEL VOL <volume_name>
and (important if the volume in question had "Scratch=No")
UPD LIBVOL <library_name> <volume_name> STAT=scratch
Again: expiration processing is not adversely affected by assigning tapes to a pool, they will just not return to "Scratch". No other pool than their "home" pool can use such tapes.
Expiration and space reclamation will take place nonetheless, but the tapes in question will not return to the scratch pool but will stay in their storage pool in an "Empty" state.
You can check whether a storage pool is allowed to use additional scratch volumes or whether it is limited to use its assigned volumes with Q STGPOOL <poolname> F=D. The MAXSCRATCH parameter defines the number of scratch tapes a pool can request.
You can check with Q VOL STGPOOL=<poolname> ST=empty how many empty volumes are available to the pool for future use.
To return empty volumes to the scratch pool unassign them from their storage pool with
DEL VOL <volume_name>
and (important if the volume in question had "Scratch=No")
UPD LIBVOL <library_name> <volume_name> STAT=scratch
Again: expiration processing is not adversely affected by assigning tapes to a pool, they will just not return to "Scratch". No other pool than their "home" pool can use such tapes.
Active 1 day ago
I know you've probably invested heavily in TSM, but it might be worth your while when your next renewal comes up to consider HP's Data Protector. It also allows you to perform a backup to disk in a single full, then incremental forever mode. It keeps track of the pointers and can expire older files and reclaim space easier, it sounds, than TSM can.
With either Data Protector or TSM, remember that you'll want to create periodic Synthetic Full Backups -- typically to tape -- that do a lot of backend magic that resets pointers and keeps things organized and optimized, and which will allow you to do quicker restores, as well as have an economical second (or third) copy of your backups to store off site.
http://www.hp.com/go/dataprotector
With either Data Protector or TSM, remember that you'll want to create periodic Synthetic Full Backups -- typically to tape -- that do a lot of backend magic that resets pointers and keeps things organized and optimized, and which will allow you to do quicker restores, as well as have an economical second (or third) copy of your backups to store off site.
http://www.hp.com/go/dataprotector
Featured Post
Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.
One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
Are you looking to recover an email message or a contact you just deleted mistakenly? Or you are searching for a contact that you erased from your MS Outlook ‘Contacts’ folder and now realized that it was important.
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention.
Multiple USB devices need t…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting.
This w…
Suggested Courses
Earn Certification
HTML5 Specialist - Certification
Free withPremium
Course of the Month8 days, 20 hours left to enroll
617 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.