Slow RMAN backups Oracle 10g with Veritas Netbackup 6.5.3
Hi, I am a Window Systems Admin and I am in charge of the Oracle 10g database backups. Here is the setup:
Backup Server:
Windows 2003, 2.0GHz x2, 2GB RAM, 1Gbps
Veritas netbackup 6.5.3
DLT2 drive tops out at about ~5-6MB/sec (Compression is off)
Oracle Server:
Windows 2003, 2.0GHz x4, 4GB RAM, 1Gbps
Oracle 10g using RMAN to backups, auto control file, 40GB database
We pretty much used the RMAN script from veritas located in the DBext folder on the client. Here are the stats on the backups I am running.
Windows file system backup to tape: ~5MB/s
Oracle RMAN to tape: ~2MB/s
Oracle RMAN to HDD (Veritas virtual tape drive) ~10MB/s
My quersion is, why are the oracle RMAN tape backups so slow? My second question would be, why are my Backups to tape so slow, when I can copy a file over the network @ ~30MB/s over gigabit.
Backup Server:
Windows 2003, 2.0GHz x2, 2GB RAM, 1Gbps
Veritas netbackup 6.5.3
DLT2 drive tops out at about ~5-6MB/sec (Compression is off)
Oracle Server:
Windows 2003, 2.0GHz x4, 4GB RAM, 1Gbps
Oracle 10g using RMAN to backups, auto control file, 40GB database
We pretty much used the RMAN script from veritas located in the DBext folder on the client. Here are the stats on the backups I am running.
Windows file system backup to tape: ~5MB/s
Oracle RMAN to tape: ~2MB/s
Oracle RMAN to HDD (Veritas virtual tape drive) ~10MB/s
My quersion is, why are the oracle RMAN tape backups so slow? My second question would be, why are my Backups to tape so slow, when I can copy a file over the network @ ~30MB/s over gigabit.
@REM $Header: hot_database_backup.cmd,v 1.2 2003/09/11 21:17:14 $
@REM bcpyrght
@REM ***************************************************************************
@REM * $VRTScprght: Copyright 1993 - 2004 VERITAS Software Corporation, All Rights Reserved $ *
@REM ***************************************************************************
@REM ecpyrght
@REM
@REM ---------------------------------------------------------------------------
@REM hot_database_backup.cmd modified by Ram Akella 05/24/06
@REM ---------------------------------------------------------------------------
@REM This script uses Recovery Manager to take a hot (inconsistent) database
@REM backup. A hot backup is inconsistent because portions of the database are
@REM being modified and written to the disk while the backup is progressing.
@REM You must run your database in ARCHIVELOG mode to make hot backups.
@REM
@REM NOTE information for running proxy backups has been included. These
@REM information sections begin with a comment line of PROXY
@REM ---------------------------------------------------------------------------
@setlocal ENABLEEXTENSIONS
@REM ---------------------------------------------------------------------------
@REM No need to echo the commands. modified by Ram Akella 03/24/05
@REM Added echo delete input all to remove all archive logs after backups by RAM 0n 06/22/06
@REM This script works with cluster name placed on both servers orap01 and orap02
@REM when cluster failed over 07/20/06
@REM ---------------------------------------------------------------------------
@echo off
@REM ---------------------------------------------------------------------------
@REM Put output in the same filename, different extension.
@REM ---------------------------------------------------------------------------
@echo off
set curndate=%date%
set day=%curndate:~4,2%
set date=%curndate:~7,2%
set year=%curndate:~10,4%
@set RMAN_LOG_FILE="C:\Program Files\VERITAS\NetBackup\Temp\oracle_daily_prod_cl_archivelogs_del7days-%day%%date%%year%.log"
@REM ---------------------------------------------------------------------------
@REM You may want to delete the output file so that backup information does
@REM not accumulate. If not, delete the following command.
@REM ---------------------------------------------------------------------------
@REM ---------------------------------------------------------------------------
@REM Replace H:\oracle\ora81, below, with the Oracle home path.
@REM ---------------------------------------------------------------------------
@set ORACLE_HOME=C:\Oracle\Ora10g
@REM ---------------------------------------------------------------------------
@REM Replace ora81, below, with the Oracle SID.
@REM ---------------------------------------------------------------------------
@set ORACLE_SID=dbnamehere
@REM ---------------------------------------------------------------------------
@REM Replace sys/manager, below, with the target connect string.
@REM ---------------------------------------------------------------------------
@set TARGET_CONNECT_STR=user/password
@REM ---------------------------------------------------------------------------
@REM Set the Oracle Recovery Manager.
@REM ---------------------------------------------------------------------------
@set RMAN=%ORACLE_HOME%\bin\rman.exe
@REM ---------------------------------------------------------------------------
@REM PROXY
@REM For a PROXY backup, uncomment the line below and replace the value.
@REM
@REM NB_ORA_PC_STREAMS - specifies the number of parallel backup streams
@REM to be started.
@REM ---------------------------------------------------------------------------
@REM @set NB_ORA_PC_STREAMS=3
@REM ---------------------------------------------------------------------------
@REM Log the start of this scripts.
@REM ---------------------------------------------------------------------------
@for /F "tokens=1*" %%p in ('date /T') do @set DATE=%%p %%q
@for /F %%p in ('time /T') do @set DATE=%DATE% %%p
@echo ==== started on %DATE% ==== >> %RMAN_LOG_FILE%
@echo Script name: %0 >> %RMAN_LOG_FILE%
@REM ---------------------------------------------------------------------------
@REM Several RMAN commands use time parameters that require NLS_LANG and
@REM NLS_DATE_FORMAT to be set. This example uses the standard date format.
@REM Replace below with the desired language values.
@REM ---------------------------------------------------------------------------
@set NLS_LANG=american
@set NLS_DATE_FORMAT=YYYY-MM-DD:hh24:mi:ss
@REM ---------------------------------------------------------------------------
@REM Print out environment variables set in this script.
@REM ---------------------------------------------------------------------------
@echo # >> %RMAN_LOG_FILE%
@echo RMAN : %RMAN% >> %RMAN_LOG_FILE%
@echo NLS_LANG : %NLS_LANG% >> %RMAN_LOG_FILE%
@echo ORACLE_HOME : %ORACLE_HOME% >> %RMAN_LOG_FILE%
@echo ORACLE_SID : %ORACLE_SID% >> %RMAN_LOG_FILE%
@echo NLS_DATE_FORMAT : %NLS_DATE_FORMAT% >> %RMAN_LOG_FILE%
@echo RMAN_LOG_FILE : %RMAN_LOG_FILE% >> %RMAN_LOG_FILE%
@REM ---------------------------------------------------------------------------
@REM PROXY
@REM For a PROXY backup, uncomment the line below.
@REM ---------------------------------------------------------------------------
@REM @echo NB_ORA_PC_STREAMS : %NB_ORA_PC_STREAMS% >> %RMAN_LOG_FILE%
@REM ---------------------------------------------------------------------------
@REM Print out environment variables set in bphdb.
@REM ---------------------------------------------------------------------------
@echo NB_ORA_SERV : %NB_ORA_SERV% >> %RMAN_LOG_FILE%
@echo NB_ORA_FULL : %NB_ORA_FULL% >> %RMAN_LOG_FILE%
@echo NB_ORA_INCR : %NB_ORA_INCR% >> %RMAN_LOG_FILE%
@echo NB_ORA_CINC : %NB_ORA_CINC% >> %RMAN_LOG_FILE%
@echo NB_ORA_CLASS : %NB_ORA_CLASS% >> %RMAN_LOG_FILE%
@REM ---------------------------------------------------------------------------
@REM We assume that the database is properly opened. If desired, this would
@REM be the place to verify that.
@REM ---------------------------------------------------------------------------
@REM ---------------------------------------------------------------------------
@REM If this script is executed from a NetBackup schedule, NetBackup
@REM sets an NB_ORA environment variable based on the schedule type.
@REM For example, when:
@REM schedule type is BACKUP_TYPE is
@REM ---------------- --------------
@REM Automatic Full INCREMENTAL LEVEL=0
@REM Automatic Differential Incremental INCREMENTAL LEVEL=1
@REM Automatic Cumulative Incremental INCREMENTAL LEVEL=1 CUMULATIVE
@REM
@REM For user initiated backups, BACKUP_TYPE defaults to incremental
@REM level 0 (Full). To change the default for a user initiated
@REM backup to incremental or incrementatl cumulative, uncomment
@REM one of the following two lines.
@REM @set BACKUP_TYPE="INCREMENTAL LEVEL=1"
@REM @set BACKUP_TYPE="INCREMENTAL LEVEL=1
@REM Ram modified the script to run the incremental backups on 06/01/05
@REM @set BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"
@REM
@REM Note that we use incremental level 0 to specify full backups.
@REM That is because, although they are identical in content, only
@REM the incremental level 0 backup can have incremental backups of
@REM level > 0 applied to it.
@REM ---------------------------------------------------------------------------
@REM ---------------------------------------------------------------------------
@REM What kind of backup will we perform.
@REM ---------------------------------------------------------------------------
@if "%NB_ORA_FULL%" EQU "1" @set BACKUP_TYPE=INCREMENTAL Level=0
@if "%NB_ORA_INCR%" EQU "1" @set BACKUP_TYPE=INCREMENTAL Level=1
@if "%NB_ORA_CINC%" EQU "1" @set BACKUP_TYPE=INCREMENTAL Level=1 CUMULATIVE
@if NOT DEFINED BACKUP_TYPE @set BACKUP_TYPE=INCREMENTAL Level=0
@REM ---------------------------------------------------------------------------
@REM Call Recovery Manager to initiate the backup. This example does not use a
@REM Recovery Catalog. If you choose to use one, remove the option, nocatalog,
@REM from the rman command line below and add a
@REM 'rcvcat <userid>/<passwd>@<tns alias>' statement.
@REM
@REM NOTE WHEN USING TNS ALIAS: When connecting to a database
@REM using a TNS alias, you must use a send command or a parms operand to
@REM specify environment variables. In other words, when accessing a database
@REM through a listener, the environment variables set at the system level are not
@REM visible when RMAN is running. For more information on the environment
@REM variables, please refer to the NetBackup for Oracle Admin. Guide.
@REM
@REM ---------------------------------------------------------------------------
@REM ---------------------------------------------------------------------------
@REM PROXY
@REM For a PROXY backup, you must use a send command to specify
@REM the NB_ORA_PC_STREAMS environment variable. For example,
@REM echo ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
@REM echo SEND 'NB_ORA_PC_STREAMS=%%NB_ORA_PC_STREAMS%%';
@REM
@REM %BACKUP_TYPE% must also be removed and replaced with the PROXY parameter
@REM in the RMAN section associated with the data files. For example,
@REM echo BACKUP
@REM echo PROXY
@REM echo FORMAT 'bk_u%%u_s%%s_p%%p_t%%t'
@REM echo DATABASE;
@REM .
@REM .
@REM Note that the controlfiles and archivelogs are not backed up using proxy
@REM copy method. Rman will initiate non-proxy copy sessions to backup the
@REM controlfile and archivelogs.
@REM ---------------------------------------------------------------------------
@(
echo RUN {
echo ALLOCATE CHANNEL ch00
echo TYPE 'SBT_TAPE';
echo SEND 'NB_ORA_CLIENT=INPHORAP00,NB_ORA_POLICY=PROD_Oracle_Daily,NB_ORA_SERV=INPHBKPP01,NB_ORA_SCHED=Default-Application-Backup';
echo BACKUP
echo %BACKUP_TYPE%
echo FORMAT 'bk_u%%u_s%%s_p%%p_t%%t'
echo DATABASE;
echo RELEASE CHANNEL ch00;
echo ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
echo SEND 'NB_ORA_CLIENT=INPHORAP00,NB_ORA_POLICY=PROD_Oracle_Daily,NB_ORA_SERV=INPHBKPP01,NB_ORA_SCHED=Default-Application-Backup';
echo # Backup all archive logs
echo sql 'alter system archive log current';
echo BACKUP
echo FILESPERSET 40
echo FORMAT 'arch-s%%s-p%%p'
echo ARCHIVELOG ALL
echo ARCHIVELOG UNTIL TIME 'SYSDATE-7' DELETE ALL INPUT;
echo RELEASE CHANNEL ch00;
echo ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
echo SEND 'NB_ORA_CLIENT=INPHORAP00,NB_ORA_POLICY=PROD_Oracle_Daily,NB_ORA_SERV=INPHBKPP01,NB_ORA_SCHED=Default-Application-Backup';
echo # The above one is working
echo BACKUP
echo FORMAT 'bk_u%%u_s%%s_p%%p_t%%t'
echo CURRENT CONTROLFILE;
echo RELEASE CHANNEL ch00;
echo }
) | %RMAN% target %TARGET_CONNECT_STR% nocatalog msglog '%RMAN_LOG_FILE%' append
@set ERRLEVEL=%ERRORLEVEL%
@REM ---------------------------------------------------------------------------
@REM NetBackup (bphdb) stores the name of a file in an environment variable,
@REM called STATUS_FILE. This file is used by an automatic schedule to
@REM communicate status information with NetBackup's job monitor. It is up to
@REM the script to write a 0 (passed) or 1 (failure) to the status file.
@REM modified by Ram Akella 05/24/06.
@REM ---------------------------------------------------------------------------
@if %ERRLEVEL% NEQ 0 @goto err
@set LOGMSG=ended successfully
@if "%STATUS_FILE%" EQU "" goto end
@echo 0 > "%STATUS_FILE%"
@goto end
:err
@set LOGMSG=ended in error
@if "%STATUS_FILE%" EQU "" @goto end
@echo 1 > "%STATUS_FILE%"
:end
@REM ---------------------------------------------------------------------------
@REM Log the completion of this script. modified by Ram Akella 03/24/05
@REM ---------------------------------------------------------------------------
@for /F "tokens=1*" %%p in ('date /T') do @set DATE=%%p %%q
@for /F %%p in ('time /T') do @set DATE=%DATE% %%p
@echo # >> %RMAN_LOG_FILE%
@echo %==== %LOGMSG% on %DATE% ==== >> %RMAN_LOG_FILE%
@endlocal
@REM End of Main Program -----------------------------------------------------
ASKER
I have tried D2D backup & I can achieve ~10MB/s, which is still super slow. When I restored the system at DR, I was restoring to a fresh build of the DB @ 15MB/s using a SDLT (much faster) drive & an original DLT2 tape. I'm thinking that it is an Oracle settings, But not sure what that setting might be. Both backup & Restore are single channel.
hi,
you can increase the channels(streams) of RMAN provided you are running the backup when the DATABASE is not in much use.
How much time does it take for you when you run a RMAN script backup to local disk. Does it finish of faster. When i say RMAN backup i am referring to disk based backup using RMAN and not any backup solution.
i am not sure how big is your oracle DB but 15 MB also sounds very very low to me.
try running perfmon on the disksubsystem when you are running backup to verify if you have any memory/disk/processor bottlenecks.
bhanu
you can increase the channels(streams) of RMAN provided you are running the backup when the DATABASE is not in much use.
How much time does it take for you when you run a RMAN script backup to local disk. Does it finish of faster. When i say RMAN backup i am referring to disk based backup using RMAN and not any backup solution.
i am not sure how big is your oracle DB but 15 MB also sounds very very low to me.
try running perfmon on the disksubsystem when you are running backup to verify if you have any memory/disk/processor bottlenecks.
bhanu
ASKER
When I increase the channels to 2, it requires 2 tapes. Is this normal?
ASKER
I have decided to run backups as compressed, this is bringing speeds up to target. I will report back once we convert to LTO4, this week.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
if your file system backup itself is giving you 5 MB no wonder that your application specific data backup would be half the amount.
The reason is that the application processes the data and sends it to the backup up application which in turn sends it to the tape drive.
one bottle neck i would say is your DLT2 tape drive.
Try to run a file system backup and the oracle RMAN backup to a D2D folder and verify if you are getting a better throughput. You should be getting a better throughput when you run a backup to the disk as the data is written randomly which is faster when compared to a SCSI device which is done in a sequential manner.
if you intend to finish the backups faster then you can possible run the backups to disk and migrate the same to tape.
this would certainly bring down your backup window.
bhanu