Solved

RMAN Backup on Oracle 9i RAC

Posted on 2004-09-29
17
4,565 Views
Last Modified: 2013-11-15
1) We are on Oracle 9i RAC on Windows 2000 Advanced Server with SP3
2) We have 2 nodes in the Cluster
3) The Database resides in the SAN Shared Storage Area with OCFS and we are running in Archivelog mode
4) The Archivelog's are stored on both the Servers running the 2 instances and they are located in 'F:\Archive' with the "Non-CFS Local Archiving" Mode
5) We have recently installed VERITAS NetBackup 5.1 and enabled the RMAN
6) When we performed a back up of the Database and Archivelogs by connecting from one of the instance / node, we were successful in backing up the Database but when it came to backing up the Archivelogs -- it failed showing "RMAN-06059: expected archived log not found, lost of archived log compromises recoverability".
7) Then i performed a test by connecting to both the nodes (through the NET8 Alias strings) and was successful in backing-up the Archive logs from both the nodes to Disk.

Kindly advise a strategy to perform a successful backup of Database and the Archive logs from both the Nodes ?

Thanks in Advance.

Kind Regards,
Sudhakar
0
Comment
Question by:sudhakarpv
  • 7
  • 5
  • 2
  • +1
17 Comments
 
LVL 47

Accepted Solution

by:
schwertner earned 168 total points
ID: 12177548
1. Verify the databases are in archivelog mode and archive destination.
 
 a. NODE 1: thread 1
 
SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u02/app/oracle/product/9.2.0/dbs/arch
Oldest online log sequence     20
Next log sequence to archive   21
Current log sequence           21
 
 b. NODE 2: thread 2
 
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u02/app/oracle/product/9.2.0/dbs/arch
Oldest online log sequence     8
Next log sequence to archive   9
Current log sequence           9
 
 
2. Verify connectivity to the target nodes and catalog if used.
 
 a. % setenv TNS_ADMIN $ORACLE_HOME/network/admin
 b. % sqlplus /nolog
 c. SQL> connect sys/pwd@node1 as sysdba
 d. SQL> connect sys/pwd@node2 as sysdba
 e. SQL> connect rman/rman@rcat
 
3. Set your testing areas.
 
Testing HOME for logs:  /u02/home/usupport/rman
 
Backups HOME Location:  /rman/V920
 
 
4. Connect using RMAN to verify and set the controlfile persistent configuration.
   The controlfiles are shared between the instances so configuring the control-
   file on node 1 also sets it for all nodes in the RAC cluster.
 
 a. Alway note the target DBID
     connected to target database: V920 (DBID=228033884)
 
 b. Default Configuration
 
RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u02/app/oracle/product/9.2.0/dbs/snapcf_V9201.f'; # default
 
 
 c. Make changes to the default that fit your business requirements.
    Note the retention policy can be set "TO REDUNDANCY x" or it can
    be set "TO RECOVERY WINDOW OF x DAYS", this is new in Oracle9i.
 
CONFIGURE RETENTION POLICY TO REDUNDANCY 3;
CONFIGURE BACKUP OPTIMIZATION OFF;  
CONFIGURE DEFAULT DEVICE TYPE TO DISK;  
CONFIGURE CONTROLFILE AUTOBACKUP ON;  
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/rman/V920/%F';  
CONFIGURE DEVICE TYPE DISK PARALLELISM 2;  
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;  
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;  
CONFIGURE MAXSETSIZE TO UNLIMITED;  
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/rman/V920/snapcf_V92321.f';
CONFIGURE CHANNEL 1 DEVICE TYPE DISK connect 'SYS/rac@node1';
CONFIGURE CHANNEL 2 DEVICE TYPE DISK connect 'SYS/rac@node2';
 
 
 d. Review/Verify your new configuration.
 
RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 3;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/rman/V920/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 2;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT  'SYS/rac@node1';
CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT  'SYS/rac@node2';
CONFIGURE MAXSETSIZE TO UNLIMITED;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/rman/V920/snapcf_V92321.f';
 
 
5. Make a backup using the new persistent configuration parameters.
 
 a. Backup database with differential incremental 0 and then archived logs
    using the delete input option.
 
 
 backup incremental level 0
 format '/rman/V920/%d_LVL0_%T_%u_s%s_p%p'
 database;
 backup archivelog all format '/rman/V920/%d_AL_%T_%u_s%s_p%p'
 delete input;
 
 
 b. Backup again using differential inremental level 1
 
 
 backup incremental level 1
 format '/rman/V920/%d_LVL1_%T_%u_s%s_p%p'
 database;
 backup archivelog all format '/rman/V920/%d_AL_%T_%u_s%s_p%p'
 delete input;
 
 
 c. To simplify this in Oracle9i we can also use PLUS ARCHIVELOG
    Note: This uses a different alorithm then backup database and
    backup archivelog in separate commands.
 
 BACKUP incremental level 0
 format '/rman/V920/%d_LVL0_%T_%u_s%s_p%p'
 database PLUS ARCHIVELOG format '/rman/V920/%d_AL_%T_%u_s%s_p%p'
 delete input;
 
  Algorithm for PLUS ARCHIVELOG:
 
    1. Archive log current
    2. Backup archived logs
    3. Backup database level 0
    4. Archive log current
    5. Backup any remaining archived log created during backup
 
 
6. Backupset Maintenance using the configured retention policy
 
    RMAN> list backup summary;
          list backup by datafile;
          list backup of database;
          list backup of archivelog all;
          list backup of controlfile;
 
 Note: these above can be enhanced with the "until time" clause as well as
       the archivelog backups using "not backed up x times" to cut down on
       many copies of a log in several backupsets. Then continuing with SMR
       Server Managed Recovery use the change archivelog from...until...delete
       to remove old logs no longer needed on disk.
 
    RMAN> report obsolete;
 
    RMAN> delete obsolete;
          or
          delete noprompt obsolete;
 
    RMAN> report schema;
 
 
7. Restore and Recover
 
Complete Recovery
 
 a. With the database mounted on the node1 and nomount on node2 connect
    to the target and catalog using RMAN.
 
      rman target / catalog rman/rman@rcat
 
    This script will restore and recover the database completely and open.
    All previous backup will still be available for use because there was
    not RESETLOGS command given.
 
 run {
     restore database;
     recover database;
     alter database open;
     }
 
Incomplete Recovery
 
Note: If you are using instance registration the database must be mounted to
      register with the listener. This means you must use the current control
      file for restore and recovery or setup a dedicated listener if not
      already done. RMAN requires a dedicated server connection and does not
      work with using instance registration before mounting the controlfile.
      Using the autobackup controlfile feature requires the DBID of the
      TARGET database. It must be set when the database is not mounted and
      only the controlfile and spfile (in 9.2>) can be restored this way.
 
 
 a. shutdown node1 and node2
 
 b. startup nomount node2 and node1
 
 c. start rman
 
     > rman trace reco1.log
 
     RMAN> connect catalog rman/rman@rcat
 
     RMAN> set dbid=228033884;
 
     RMAN> connect target
 
 d. Restore the controlfile from autobackup
 
     % rman trace recocf.log
 
     RMAN> SET DBID=228033884;
 
     RMAN> CONNECT TARGET
 
 RUN  
    {
     SET CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE disk TO '/rman/V920/%F';
      ALLOCATE CHANNEL d1 DEVICE TYPE disk;  
       RESTORE CONTROLFILE FROM AUTOBACKUP
         MAXSEQ 5           # start at sequence 5 and count down (optional)
         MAXDAYS 5;         # start at UNTIL TIME and search back 5 days (optional)
     MOUNT DATABASE;
    }
 
 
 e. Verify what is available for incomplete recovery. We will recover to the
    highest scn log sequence and thread. We will use the log sequence in this
    case. Your options are "until time", "until scn", or "until sequence".
 
 
    SQL> select max(sequence#) from v$archived_log
      2  where thread#=1;
 
 MAX(SEQUENCE#)
 --------------
             25
 
    SQL> select max(sequence#) from v$archived_log
      2  where thread#=2;
 
 MAX(SEQUENCE#)
 --------------
             13
 
Note: In this case the scn is greater in thread 2 sequence# 13 then in
      sequence 25 from thread 1. So we will set the seqeunce to 14 for
      rman recovery because log recovery is always sequence+1 to end
      at +1 after applying the prior sequence.
 
 SQL> select sequence#, thread#, first_change#, next_change#
   2  from v$archived_log
   3  where sequence# in (13,25);
 
  SEQUENCE#    THREAD# FIRST_CHANGE# NEXT_CHANGE#
 ---------- ---------- ------------- ------------
         25          1       1744432      1744802
         13          2       1744429      1744805
 
 SQL> select sequence#, thread#, first_change#, next_change#
   2  from v$backup_redolog
   3  where sequence# in (13,25);
 
  SEQUENCE#    THREAD# FIRST_CHANGE# NEXT_CHANGE#
 ---------- ---------- ------------- ------------
         25          1       1744432      1744802
         13          2       1744429      1744805
 
 f. If using LMT Temporary tablespace the controlfile will have the syntax
    to add the tempfile after recovery is complete.
 
    SQL> alter database backup controlfile to trace;
 
 Example:
 # Commands to add tempfiles to temporary tablespaces.
 # Online tempfiles have complete space information.
 # Other tempfiles may require adjustment.
 ALTER TABLESPACE TEMP ADD TEMPFILE '/dev/usupport_vg/rV92B_temp_01.dbf'
      SIZE 41943040  REUSE AUTOEXTEND OFF;
 # End of tempfile additions.
 #
 
 g. Since log sequence 13 thread 2 next_change# is 3 changes ahead of thread 1
    sequence 25 we are using dequence 14 to stop recovery. This will restore
    the datafiles and recover them completely using the online logs.
 
 run {
  set until sequence 14 thread 2;
   restore database;
   recover database;
  alter database open resetlogs;
   }
 
 
8. Review and understand the impact of resetlogs on the catalog.
 
 RMAN> list incarnation of database V920;
 
Note: After resetlogs there are 2 incarnations in the recovery catalog. Only
      one incarnation can be current at one time for a given dbid. The Inc Key
      keeps track of the database incarnations.
 
List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            CUR Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
2656    2657    V920     228033884        NO  1          29-MAY-03
2656    3132    V920     228033884        YES 1744806    13-JUN-03
 
 
9. RMAN Sample Commands
 
 a. With a dedicated listener (not using instance registration)
   restoring the controlfile.
 
 run {
 allocate channel d1 type disk connect 'sys/rac@node1';
 allocate channel d2 type disk connect 'sys/rac@node2';
  set until sequence 14 thread 2;
   restore controlfile;
  alter database mount;
 release channel d1;
 release channel d2;
 }
 
 b. Backup Archivelog
 
 backup archivelog all not backed up 3 times;
 backup archivelog until time 'sysdate-2' not backed up 2 times;

0
 

Author Comment

by:sudhakarpv
ID: 12177719
Hi,
Thank you for your reply and patience.
But, i think the whole point is missed there.
First, our platform is NOT UNIX but WINDOWS.
Second, i am / was successful in doing a backup of the whole database, as well as, the Archive logs from RMAN on to a Disk.
Third and the most important one is, i wanted to know a strategy by which i can successfully perform the backup of the Database and Archivelogs from the nodes, as we are backing up only from one node?
Again, we are performing the backup ONLY from one node and thus, the NetBackup is unable to backup the archivelogs generated in the other node.
Therefore, i am looking for a probable answer / solution for the above -- that is, performing a backup of all the archive logs without the database ?
Hope i am clear in my explanation of the scenarion.

Thanks in Advance.

Kind Regards,
Sudhakar
0
 
LVL 1

Expert Comment

by:jclee_sg
ID: 12177963
Hi Sudhakar,

Upon my experience, your solution to set the archivelog dest to 'f:\archivelog' on the both servers is not workable when you do a crash recovery or even when you try to hot backup the whole database with rman, because both servers cann't see each other's F drive or, on the other word, both servers can't find all needed archivelogs in their local archivelog dest.

My solution: set the archivelog_dest seperately on the servers, following i'll give a sample
1. set server1's archivelog_dest to 'e:\archivelog'
2. set server2's archivelog_dest to 'f:\archivelog'
3. map the server1's drive E as server2's drive E, and set access rights properly
4. map the server2's drive F as server1's drive F, set access rights
5. done!

And you only need to backup from any one node of the 2 with RMAN and all archivelogs will be backed up in both E & F drives

Try this, hope this helps!
JC.L
0
 

Author Comment

by:sudhakarpv
ID: 12178019
Hi JC,
What you said is correct, but the concept of RAC itself means that each node will be able to read as well as write to the other node irrespective of any kind of files.
I tried mapping the server2's drive f to server1 but it didn't work.
Maybe what you said will work, by specifying different partition's for the archive log destination --- this is one probable solution.
But i need couple of days time to test and give a confirmation about the same.
Thanks & Regards,
Sudhakar
0
 
LVL 10

Assisted Solution

by:SDutta
SDutta earned 166 total points
ID: 12182649
Hi Sudhakar,

This problem happens on Windows installations of RAC which is different from Unix and is not easy to mount/dismount NFS points. If you want to run the archivelog backup from a single instance then the only solution is to write the archivelogs to a (CFS) cluster file system disk. If you put them on local disks then you have to run the archivelog jobs separately from the database backup, once from each node - this is the method we follow.

There may be an alternate way which we have not tested yet is to define a second destination with LOG_ARCHIVE_DEST_2 or LOG_ARCHIVE_DUPLEX_DEST which points to a Windows mapped NTFS disk on the second node. Also set LOG_ARCHIVE_MIN_SUCCEED_DEST = 1. The issue is that both locations must be accessible from the same machine to run the archivelog backup from that node.

0
 
LVL 1

Expert Comment

by:jclee_sg
ID: 12186194
Hi Sudhakar,

Just make clear 2 points:

1. It doesn't matter if it's drive E or F, just make sure that server1 sees server2's real local drive X as server1's mapped local X, so works have to be done to rename local conflict drive letters, and vice versa.

2. for each server the only parameter has to be set is log_archive_dest_1. For example in your spfile:

    server1.log_archive_dest_1 = 'x:\archivelog'
    server2.log_archive_dest_1 = 'y:\archivelog'

    DO NOT also set such as server1.log_archive_dest_2 = 'y:\archivelog', we just simply let both servers write to their own drives and read from both. Even though in some cases a node may do archiving jobs for other nodes, just let it be.

And finally, just not like what SDutta has said, you CAN backup the whole database from any one node as long as the other node's archivelog drive is available. And this is just what we are doing, and we've done restores and recoveries many times to test it so I can promise this to you.

Regards

JC.L
0
 
LVL 10

Expert Comment

by:SDutta
ID: 12190488
@JCL
Don't quote me wrong, this is what I said in the last sentence - "both locations must be accessible from the same machine to run the archivelog backup from that node".
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:sudhakarpv
ID: 12211089
Hi JC,
I tried the point which you suggested above as:
Server1: log_archive_dest_1='f:\archive'
Server2: log_archive_dest_2='o:\archive'
Then mapped the Server2's 'O:\archive' to Server1's 'O:\' drive and tried the backup, but it is still failing as "RMAN-06726: could not locate archivelog F:\ARCHIVE\TNDC_132.ARC" because this particular archivelog is of the Server1 which got created in the Server2's location.
Is that i did any mistake while mapping the drive's or is it another mistake ? Or is it that i need to map the Server2's archive destination as '\\server2\o$' or as above as '\\server2\archive' ?
Kindly clarify how the mapping is to be done (i tried both ways but still it is erroring out) ?
Thanks & Regards,
Sudhakar
0
 

Author Comment

by:sudhakarpv
ID: 12211093
Hi JC,
One more point after giving the sharing access, i have given full access permissions in the Server2 for the 'archive' directory.
Regards,
Sudhakar
0
 
LVL 1

Expert Comment

by:jclee_sg
ID: 12214224
Hi Sudhakar,

First, you should correct your parameters to:

Server1: log_archive_dest_1='f:\archive'
Server2: log_archive_dest_1='o:\archive'

Second, Just map the server2's drive 'O:' to server1's drive 'O:', so that server1 sees server2's 'o:\archive\' as if it's local 'o:\archive'. And again you must also map server1's drive 'F:' to server2 in the same way. The result is both servers see the same directory structure like:

    f:\archive
    o:\archive

Regards!

JC.L
0
 

Author Comment

by:sudhakarpv
ID: 12224526
Hi JC,
I have done as suggested by you - changed the Server2's archive log destination from 'f:\archive' to 'g:\archive' and mapped it to Server1 as 'g:\' and in the same way mapped Server1's 'f:\archive' to Server2 as 'f:\' and again tried to perform a backup.
This time, i again got an error as "ORA-19625: error identifying file F:\ARCHIVE\PNDC_22437.ARC", because RMAN is still looking at the old archive path of Server2 as 'f:\' and this archive was generated before i changed the location from 'f:\' to 'g:\'
Therefore, what do i need to do to tell RMAN that the location is changed and that the particular archive log is still present to be backed up.
Another point which i observed is: in 9.2 the moment RMAN performs a backup of the archivelogs, it updates the status as 'X' in the AL table of its schema and hence even if we try to perform a backup of the files again, it doesn't because of the 'X' status. How can we overcome this problem ?
Kind Regards,
Sudhakar
0
 
LVL 1

Expert Comment

by:jclee_sg
ID: 12224613
Hi Sudhakar,

You have misread my comments. Just map the drives from 'o:' to 'o:', 'p:' to 'p:' etc., instead of mapping 'o:\folder' to 'o:'.

For your 2nd question, so far I didn't touch that table yet. Maybe others experts can answer it.

Regards

JC.L
0
 

Author Comment

by:sudhakarpv
ID: 12224929
Hi JC,
Sorry, i wrote the statement wrong -- i did the mapping like what you said but while replying i wrote it the other way.
But, apart from the table, the other point i observed was regarding the message i got from the NetBackup utility -- ORA-19625: error identifying file F:\ARCHIVE\PNDC_22437.ARC -- this archivelog got created at a previous time before i changed the locations and RMAN is still looking at the old location. This archivelog was generated in the Second Server and it got created in f:\archive (the old location) -- so how can i point this change to RMAN, in turn to perform a backup of all the archivelogs generated in the Second Server.
Thanks & Regards,
Sudhakar
0
 
LVL 1

Assisted Solution

by:jclee_sg
jclee_sg earned 166 total points
ID: 12225068
Hi Sudhakar,

Simply copy the required previous arc files to the new F:\archive\ folder so that rman can find them. remember to always make backup of those files if you are doing any overwriting.

Regards

JC.L
0
 

Author Comment

by:sudhakarpv
ID: 12269183
Hi JC,
Thank you for all the support and patience.
One last request, could you briefly summarise any other points to be taken care of in case any problems arise, which i may overlook.
Thanks & Warm Regards,
Sudhakar
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

How to fix error ""Failed to validate the vCentre certificate. Either install or verify the certificate by using the vSphere Data Protection Configuration utility" when you are trying to connect to VDP instance from Vcenter.
How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now