Solved

Automating tape archive of multi-system FSD backup in ARCserve 11

Posted on 2004-08-18
16
2,183 Views
Last Modified: 2008-01-09
Yes, that's a doozy.

I have ARCserve 11 running on a Win2K3 Standard server, with a 2.3 TB array attached for backup-to-disk of several servers running NetWare 6, RedHat ES 2.1, a couple of different versions of Windows and a Snap server.  The rotation backups to disk are going well for the most part, but I have a problem figuring out how to make a daily rotation scheme to create a consolidated tape set for offsite storage.  Tapecopy.exe is not really conducive to that kind of thing.

One problem is that because of the different application requirements of the different servers, they are not all on the same kind of backup rotation on the FSD backups, and for efficiency's sake I have separate device groups for the different types of servers, so they can be backed up simultaneously (if they're all on the same device group, they will wait for one to complete before starting the next...)  Because of that, it's difficult to devise a tapecopy command to grab the latest full backup and all incrementals since, for those that are on full/incremental rotations, and just the latest full backup from those that are on daily full rotations.

I am more than happy to create several jobs to do this task, and schedule them according to day of week and in sequence, appending to the same tape for a given day. If I could, I would have the tapecopy done as an after-backup action in each backup job, but that leads into another problem - I have to change tapes somewhere along the line, and don't want any backups waiting for the tape copy to do its thing.  I wanted to get an LTO-2 drive, but that was vetoed, so instead I have to spend twice as much on AIT-3 tapes - but I digress...

I also have a problem with tapecopy in  that it does not allow serial numbers to stick with a tape, unless you specify the serial number in the tapecopy command.  That doesn't work when you want to automate it.  You essentially have to let it autoname the tape with the date, write the date on the tape to identify it, and manually remove and erase them to feed them back into your tape rotation.  If there's a way around that, I'd like to know.

Ideas?  Solutions?  Scripting examples?  Other ARCserve command-line utilities that can be used?
0
Comment
Question by:ShineOn
  • 8
  • 5
16 Comments
 
LVL 22

Expert Comment

by:dovidmichel
ID: 11841464
First off on the chance you don't know about it the Job Scheduler Wizard (found in the ARCserve program group) can be used to run tapecopy jobs. Select ca_devmgr and then tapecopy.

How about using this option
-qPastTime Queries the number of days specified previous to current date and time.
Setting it for 1 day should get all the previous backups from the group.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 11841833
First, thanks for the response.

second, yes, I have been using the wizard to schedule tapecopy commands.  I want to automate and simplify the operation so anyone can walk in and do the tapes for the day.  With the mix of backups and rotations I'm using on the FSD, it almost seems I have to do something different each time.
Third, what would I use ca_devmgr for?
Fourth, I have been using the -qPastTime query, but that won't help me get a consolidated set that includes all of the latest full backups plus all subsequent incrementals to current.  I think that to do that, I'd have to write a custom script of some sort, but am not sure how such a script would be constructed.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 11841863
Fifth - do you know of any way to get a tapecopy to assign a sequential serial number according to the tape pool being used, like one might expect it to?
0
 
LVL 22

Expert Comment

by:dovidmichel
ID: 11843286
A consolidated set of the latest fulls. Wow

If that is the end goal then perhaps a different setup for the backups would be a better way to go.

For example you could setup one job to do the full backup of all the systems. That way to begin with all the full backups will be consolidated. Then for the other incrementals setup different jobs as per the different needs of the systems, but have the jobs go to the same group and append.

Now you have your consolodation and you have only two groups making tapecopy much easier, yet each system is still backed up as per it's needs.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 11843523
That's a thought.

Just wondering - what would be best, in your opinion, if I wanted to set up a complex script to do exactly what I want?  WSH with VBS, JScript, Perl or Python; good ole BAT files; WinBatch; or ??  We're talking about multiple ARCserve command-line utilities all tied together in a script with variable substitution and error checking.
0
 
LVL 22

Expert Comment

by:dovidmichel
ID: 11843723
good ole BAT files since they work well with the Post job command option.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 11844791
It boggles the mind that CA has not incorporated a "real" FSD-to-tape-consolidation job as part of the backup process.  I do believe that NetVault can do that.  

I guess the best I can do is get it "reasonably" automated using BAT files and the scheduling wizard, and set up a manual rotation, manually labeling the tapes with their creation date and deleting and erasing them when they should be "scratch."

I have tried pre-initializing the tapes, but they end up with serial number="[]" anyway.

An example of tapecopy commands I have used is:

TAPECOPY -d "TAPE-AIT" -qMediaPool "*_WLY" -m "TAPE_ARCW" -qPastTime "6" -b -g

It creates the tape set in the TAPE_ARCW pool, but doesn't assign it a serial number according to the pool's rules.
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 22

Expert Comment

by:dovidmichel
ID: 11845362
Serial numbers are put to a tape via two methods:
1) when barcodes are used - the tape is given a serial number matching the barcode
2) when the tape is formated as part of a job using a media pool.

If it is not already installed then get
QO57427 11.3 MB 07/22/04 NT -R11 DEVICE SUPPORT UPDATE 4

As for the automation thing, don't know, heard rumors of 11.1 having better support for it but I missed the beta and so just don't know for sure what to expect from it yet.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 11946290
What I've ended up doing, at this point anyway, is a WinBatch routine that automatically names the tapes as daily, weekly or monthly, along with the date.  The tapecopy command blows away any serial numbering you give a tape, even if it's assigned to a media pool., so that's the only way I can come up with to easily associate a physical tape with a database entry.  It also uses a holiday file to help calculate whether or not to run - it'll only run Monday-Friday mornings, except on holidays, and will grab all sessions from Sunday on for incrementals and for the previous day for full-only backups.

So far, it's kinda working.  I hope you heard right about 11.1.  If you want, once I clean up my code, I'll shoot you a copy of my WinBatch script.  Let me know.
0
 
LVL 22

Expert Comment

by:dovidmichel
ID: 11947838
Sure I'd like to see it.
0
 
LVL 35

Author Comment

by:ShineOn
ID: 12213099
It's been a wild ride - it's been a while but I'd like to keep this open for a tad yet, so I can post the script I'm using.  It might make a good PAQ then.  Is that OK?
0
 
LVL 32

Expert Comment

by:Luc Franken
ID: 12213106
Not a problem for me, but the moderator following up will decide. Please post your script so this question can be PAQ'd and refunded.

LucF
EE Cleanup Volunteer
0
 
LVL 35

Accepted Solution

by:
ShineOn earned 0 total points
ID: 12237554
Here is the WinBatch script I'm using.  It calls a holiday calendar file to perform exception processing for holidays.

; Tapecopy Automation Script
; Author: J. Stalewski 8/24/2004
;
; Function:  Generate "consolidated" backup tapes from filesystem-device backup datasets, for archival purposes.
;             Uses BrightStor ARCserve Backup 11.0 command-line utilities.  Uses holiday file for exceptions.
;                          data\Holidays.txt
; Note:  Future enhancements include error checking, activity logging and hardcoded rotation override
;
; Modified:
;
;
;
;

;  Winbatch Extender Addins Go Here
;  Example: AddExtender("wwwnt34i.dll")

;  Variable Inits Go Here
sASPath = "D:\BrightStor ARCserve Backup\"
; sASPath = "C:\Program Files\CA\BrightStor ARCserve Backup\"

NumGroups = 6

DiskGroups=ArrDimension(NumGroups, 3)

DiskGroups[0,0] = "NETWARE"
DiskGroups[0,1] = "NW"
DiskGroups[0,2] = "I"
DiskGroups[1,0] = "WINDOZE"
DiskGroups[1,1] = "NT"
DiskGroups[1,2] = "I"
DiskGroups[2,0] = "SNAP"
DiskGroups[2,1] = "SNAP"
DiskGroups[2,2] = "I"
DiskGroups[3,0] = "LINUX"
DiskGroups[3,1] = "LX"
DiskGroups[3,2] = "F"
DiskGroups[4,0] = "LAPTOP"
DiskGroups[4,1] = "LT"  
DiskGroups[4,2] = "F"
DiskGroups[5,0] = "ASDBGP"
DiskGroups[5,1] = "AS"  
DiskGroups[5,2] = "I"
;  Start logic - add log file open and logging start here

; End the job if it's a weekend day, otherwise set full backup select period.
tDayWord = StrSub ( TimeDate(), 1, 3 )
If tDayWord == "Sat" then Goto Theend  ;Don't run on weekend
If tDayWord == "Sun" then Goto Theend  ;Don't run on weekend
If tDayWord == "Mon" then sPastTm = 2  ; Set how far back to look for Sunday's full backup
If tDayWord == "Tue" then
      sPastTime = 3   ; Set how far back to look for Sunday's full backup
;      sPastTmInc = 1  ; Set how far back to look for incrementals since Sunday's full backup
endif
If tDayWord == "Wed" then
      sPastTime = 4  ; Set how far back to look for Sunday's full backup
;      sPastTmInc = 2  ; Set how far back to look for incrementals since Sunday's full backup
endif
If tDayWord == "Thu" then
      sPastTime = 5  ; Set how far back to look for Sunday's full backup
;      sPastTmInc = 3  ; Set how far back to look for incrementals since Sunday's full backup
endif
If tDayWord == "Fri" then
      sPastTime = 6  ; Set how far back to look for Sunday's full backup
;      sPastTmInc = 4  ; Set how far back to look for incrementals since Sunday's full backup
endif

:DateChecks
;  End the job if it's a holiday.
tTodayJul = TimeJulianDay ( TimeYmdHms() ) ; Grab Current Date and make into Jdate
Holiday1 = @FALSE
Holiday2 = @FALSE
:HolidayCheck
hHolidayFile = FileOpen ( "C:\Program Files\WinBatch\Data\Holidays.txt", "READ")
While @TRUE
      x = FileRead(hHolidayFile)
      If x == "*EOF*" Then Break
      If StrSub (x, 1, 1) == "*" then Goto SkipTest
      iMM = StrSub (x, 1, 2)
      iDD = StrSub (x, 4, 2)
      iYYYY = StrSub (x, 7, 4)
      sDateCk = StrCat( iYYYY,":",iMM,":",iDD )
      ;Message ( "sDateCk", "sDateCk = %sDateCk%" )
      tHolJul = TimeJulianDay ( sDateCk )
      If tTodayJul == tHolJul then
            Holiday1 = @TRUE ;Set Holiday1 to true if today is a holiday
            Break
      endif
      if (tTodayJul - 1) == tHolJul then
            Holiday2 = @TRUE
            Break
      endif
:SkipTest
endwhile

if Holiday1 == @TRUE then Goto Theend  ;Don't run on a holiday

;  Set date variables including day number, day of week, month of year, previous month name


aSlash = StrScan ( TimeDate(), "/", 1, @FWDSCAN)
bSlash = StrScan ( TimeDate(), "/", aSlash+1, @FWDSCAN)
DaySiz = bSlash - aSlash -1
tDateMDY = StrTrim ( StrSub ( TimeDate(), 5, 10 ))
tMonth = int ( StrTrim(StrSub ( TimeDate(), 5 ,aSlash-5 )))
tDay = int( StrSub ( TimeDate(), aSlash+1, DaySiz ))
tYear = int ( StrTrim ( StrSub ( TimeDate(), bSlash+1, 4 )))



Switch tMonth
      Case 1
            tMonthWord = "Jan"
            tLastMonthWord = "Dec"
            Break
      Case 2
            tMonthWord = "Feb"
            tLastMonthWord = "Jan"
            Break
      Case 3
            tMonthWord = "Mar"
            tLastMonthWord = "Feb"
            Break
      Case 4
            tMonthWord = "Apr"
            tLastMonthWord = "Mar"
            Break
      Case 5
            tMonthWord = "May"
            tLastMonthWord = "Apr"
            Break
      Case 6
            tMonthWord = "Jun"
            tLastMonthWord = "May"
            Break
      Case 7
            tMonthWord = "Jul"
            tLastMonthWord = "Jun"
            Break
      Case 8
            tMonthWord = "Aug"
            tLastMonthWord = "Jul"
            Break
      Case 9  
            tMonthWord = "Sep"
            tLastMonthWord = "Aug"
            Break
      Case 10
            tMonthWord = "Oct"
            tLastMonthWord = "Sep"
            Break
      Case 11
            tMonthWord = "Nov"
            tLastMonthWord = "Oct"
            Break
      Case 12
            tMonthWord = "Dec"
            tLastMonthWord = "Nov"
            Break
EndSwitch
;Message ( "DateVars", "aSlash = %aSlash%  bSlash = %bSlash% %@CRLF% tDayWord = %tDayWord% %@CRLF% tDateMDY = >%tDateMDY%< %@CRLF% tMonth = %tMonth% %@RCLF% tDay = %tDay% %@CRLF% tYear = %tYear% %@CRLF% tMonthWord = %tMonthWord%  Holiday2 = %Holiday2% ")

;  Check for daily, weekly or monthly and set tape name and pool variable
;  At this point it has been determined that it is not a holiday, Saturday or Sunday.  This is assumed from here on.

TapeType = "Daily"  ;assume it's a Daily unless changed below...
sLasMth = "" ;Set "last month" to null for tape naming process
sTypeLtr = "D"
; The following tests determine if it's a Weekly.  If not, it's still a Daily.
If tDayWord == "Mon" then
      TapeType = "Weekly"  ; If it's Monday, assume it's a Weekly
      sTypeLtr = "W"
Endif
if tDayWord == "Tue" then
      if Holiday2 == @TRUE then
         TapeType = "Weekly"   ;If it's Tuesday and yesterday was a holiday, then assume it's a Weekly
            sTypeLtr = "W"
      Endif
Endif

; At this point it is a Daily or a Weekly.  If it's set for Weekly, it's either a Monday or Tuesday.
;  If it's set for Daily, it's not a Monday.

;The following tests to see if it should be a Monthly.
if tDay > 4 then Goto CheckTape  ; Assume if it's past the 4th, it's not a Monthly. Leave it set to Daily or Weekly.
if tDay == 1 then
      TapeType = "Monthly"                ; If it's the first at this point, then it's a Monthly. End of discussion.
   sTypeLtr = "M"
      sLasMth = tLastMonthWord
      Goto CheckTape
endif

; If it gets to this point, it can only be a monthly if it's after a weekend. This assumes it's a Monday or Tuesday.
if tDayWord == "Mon" || (tDayWord == "Tue" && Holiday2 == @TRUE) then
      TapeType = "Monthly"
      sTypeLtr = "M"
      sLasMth = tLastMonthWord
      Goto CheckTape
endif

:CheckTape
Display ( 3600, "Tape Needed", "%TapeType% %@CRLF%" )

;  Run command to check tape in drive.  If not scratch or blank, eject and ask for scratch or blank.
;  This one is "on hold" for now - the tapecopy command will check and initialize the tape.

:NameTape
; This routine creates the tape name to be used for this run.
sTapeName = StrCat( TapeType,"-",sLasMth,"-",tDateMDY )
;Message ( "TapeName", "TapeName = %sTapeName%" )
:StartCopy
;  Start loop for backup, loop through disk device groups.  Use array for lookup.
For g=0 to 4
      sGroup = DiskGroups[g,0]
      sPool = DiskGroups[g,1]
      sType = DiskGroups[g,2]
      sPoolSet = StrCat ( sPool, "_?LY" )

;       Message ( "DiskGroup", "g = %g% %@CRLF% DiskGroup = %sGroup%" )


;  Run tapecopy command to copy last full backup to tape, ignoring "replicated."
;  This would be the only step for only-full backups.
;  This will assume last full backup to be within last 24 hours for only-full, and calculated back to Sunday for daily/weekly.
;  Monthly will be based on date of last day of the month forward to current date. Use special job if Sunday backup failed.
If sGroup == "LAPTOP" then Goto EndOfLoop ; Don't put laptop backups on the archive tape.
If sType == "F" then sPastTm = 1  ; always set it to 1 for full-only backups
if sType == "I" then sPastTm = sPastTime


sCmdStr = "-d TAPE-AIT -qMediaPool %sPoolSet% -qPastTime %sPastTm% -qIgnoreRep -m TAPE_ARC%sTypeLtr% -c %sTapeName%"


:DoTheCopy
;Tapecopy command for full bkup:
RunWait ( "%sASPath%TapeCopy.EXE", "%sCmdStr%" )
; Message ( "Tapecopy Command String", "%sCmdStr%" )


;  Loop back to query, checking the next device group.
:EndOfLoop
Next g

;  Merge sessions to ARCserve database

RunWait ( "%sASPath%CA_MERGE.EXE", "-tape %sTapeName% -allsessions -group TAPE-AIT" )

;  Eject tape when done

RunWait ( "%sASPath%CA_DEVMGR.EXE", "-eject 3 0 2" )

Message ( "Label Tape(s)", "%sTapeName%" )
;  Close log file (when function has been established)

;  End logic flow - do cleanup - close log file

:Theend
Exit

;  Error subroutines (very rudimentary - future enhancement)

:DateErr
Return  ; if not fatal
Goto Theend   ; if fatal


 
0
 
LVL 35

Author Comment

by:ShineOn
ID: 12238301
Note: the commented-out message commands are for debugging purposes.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now