Solved

ORA-15205 when starting RAC database (asm disks shown as dropped)

Posted on 2011-09-07
6
2,918 Views
Last Modified: 2012-05-12
ORA-15205: requested mirror side unavailable
cause the RAC database to fail at startup.

In ASM, two disks (without any redundancy) are shown as "_DROPPED_". They belong to DATA and FRA main diskgroups.

[root@bidegi01 ~]# fdisk -l | grep sd
Disco /dev/sda: 21.4 GB, 21474836480 bytes
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        2610    20860402+  8e  Linux LVM
Disco /dev/sdb: 32.2 GB, 32212254720 bytes
/dev/sdb1   *           1        3916    31455238+  8e  Linux LVM
Disco /dev/sdc: 1073 MB, 1073741824 bytes
/dev/sdc1               2        1886     1047552   83  Linux
Disco /dev/sdd: 1073 MB, 1073741824 bytes
/dev/sdd1               2        1886     1047552   83  Linux
Disco /dev/sde: 1073 MB, 1073741824 bytes
/dev/sde1               2        1886     1047552   83  Linux
Disco /dev/sdf: 21.4 GB, 21474836480 bytes
/dev/sdf1               1        2611    20970496   83  Linux
Disco /dev/sdg: 21.4 GB, 21474836480 bytes
/dev/sdg1               1        2611    20970496   83  Linux
Disco /dev/sdh: 21.4 GB, 21474836480 bytes
/dev/sdh1               1        2611    20970496   83  Linux
Disco /dev/sdi: 21.4 GB, 21474836480 bytes
/dev/sdi1               1        2611    20970496   83  Linux


SQL> select path, name from v$asm_disk;

PATH                           NAME
------------------------------ ------------------------------
ORCL:ASM_DATA02
ORCL:ASM_DATA03
ORCL:OCR_VOTE01
                               _DROPPED_0001_DATOS
                               _DROPPED_0000_FRA
ORCL:ASM_DATA01                ASM_DATA01
ORCL:ASM_DATA04                ASM_DATA04
ORCL:OCR_VOTE02                OCR_VOTE02
ORCL:OCR_VOTE03                OCR_VOTE03

9 rows selected.

Open in new window




[grid@bidegi02 bin]$ ./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATOS.dg   ora....up.type ONLINE    ONLINE    bidegi01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    bidegi01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    bidegi01
ora....N1.lsnr ora....er.type ONLINE    ONLINE    bidegi02
ora....N2.lsnr ora....er.type ONLINE    OFFLINE
ora....N3.lsnr ora....er.type ONLINE    ONLINE    bidegi01
ora....VOTE.dg ora....up.type ONLINE    ONLINE    bidegi01
ora.asm        ora.asm.type   ONLINE    ONLINE    bidegi01
ora....SM1.asm application    ONLINE    ONLINE    bidegi01
ora....01.lsnr application    ONLINE    ONLINE    bidegi01
ora....i01.gsd application    OFFLINE   OFFLINE
ora....i01.ons application    ONLINE    ONLINE    bidegi01
ora....i01.vip ora....t1.type ONLINE    ONLINE    bidegi01
ora....SM2.asm application    ONLINE    ONLINE    bidegi02
ora....02.lsnr application    ONLINE    ONLINE    bidegi02
ora....i02.gsd application    OFFLINE   OFFLINE
ora....i02.ons application    ONLINE    ONLINE    bidegi02
ora....i02.vip ora....t1.type ONLINE    ONLINE    bidegi02
ora.bret.db    ora....se.type ONLINE    OFFLINE
ora.cvu        ora.cvu.type   ONLINE    ONLINE    bidegi01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    bidegi01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    bidegi02
ora.ons        ora.ons.type   ONLINE    ONLINE    bidegi01
ora....ry.acfs ora....fs.type ONLINE    ONLINE    bidegi01
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    bidegi02
ora.scan2.vip  ora....ip.type ONLINE    OFFLINE
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    bidegi01

Open in new window


Starting database manually causes the following error:

SQL> startup
ORACLE instance started.

Total System Global Area  839282688 bytes
Fixed Size                  2231128 bytes
Variable Size             633341096 bytes
Database Buffers          197132288 bytes
Redo Buffers                6578176 bytes
Database mounted.
ORA-15205: requested mirror side unavailable

Open in new window



I've tried to shutdown all CRS and startup all the services from scratch, but no success.

[root@bidegi02 ~]#  /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@bidegi02 ~]#  /etc/init.d/oracleasm listdisks
ASM_DATA01
ASM_DATA02
ASM_DATA03
ASM_DATA04
OCR_VOTE01
OCR_VOTE02
OCR_VOTE03

Open in new window


Any solution to startup the database?
0
Comment
Question by:Javier Morales
  • 4
6 Comments
 
LVL 6

Author Comment

by:Javier Morales
ID: 36495310
Also, when I try to restart the asm, I get the following error:


ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 31682)
CRS-2675: Fallo al parar ''ora.asm'' en ''bidegi02''

La parada de `ora.OCR_VOTE.dg` en el miembro `bidegi01` se ha realizado correctamente.
Intentando parar `ora.asm` en el miembro `bidegi01`
CRS-2673: Intentando parar ''ora.asm'' en ''bidegi01''
CRS-5017: La acción de recurso "ora.asm stop" ha detectado el siguiente error:
ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 11592)
CRS-2675: Fallo al parar ''ora.asm'' en ''bidegi01''

CRS-0216: Could not stop resource 'ora.asm bidegi01 1'.

CRS-0216: Could not stop resource 'ora.asm bidegi02 1'.

CRS-0216: Could not stop resource 'ora.gsd'.

CRS-0216: Could not stop resource 'ora.gsd bidegi01 1'.

CRS-0216: Could not stop resource 'ora.gsd bidegi02 1'.

CRS-0216: Could not stop resource 'ora.ons bidegi02 1'.

Open in new window

0
 
LVL 12

Assisted Solution

by:praveencpk
praveencpk earned 250 total points
ID: 36501080
did you dropped disk groups?

if it is not droped completly you can try undrop the diskgroup.

ALTER DISKGROUP dgroup1 UNDROP disks;

It clear states that there are active client connected to the ASM instance, plz query V$ASM_CLIENT or use ASMCMD LSOF to list active clients. kill that process and restart the asm instance

you can also use shutdown abort for your ASM instance.
0
 
LVL 6

Author Comment

by:Javier Morales
ID: 36501123
Hi Praveencpk, Thanks a lot for your answer!

I didn't dropped the diskgroups, It seems that ASM dropped the disks by itself.
Anyway, the dropping isn't finish yet, and disks can't be dropped.

SQL> alter diskgroup datos drop disk '_DROPPED_0001_DATOS';
alter diskgroup datos drop disk '_DROPPED_0001_DATOS'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15071: ASM disk "_DROPPED_0001_DATOS" is already being dropped

Open in new window


I followed metalink's note "
ALTER DISKGROUP DROP DISK FAILS WITH ORA-15071 ASM DISK IS ALREADY BEING DROPPED [ID 1066335.1]" that also recommend to undrop disks and retry dropping, and get the same error ORA-15071

lsof shows no active clients and there's now only one ASM instance started (to avoid blocking processes).

I shutted down ASM instance, but when I started it again, the issue remains the same.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 16

Assisted Solution

by:rolutola
rolutola earned 250 total points
ID: 36512099
You cannot stop ASM with 'ordinary' methods because OCR and voting disks are highly likely held on diskgroups on the ASM. Hence you cannot stop ASM without stopping the clusterware stack.

To stop the clusterware stack use the following:

crsctl stop cluster

R.
0
 
LVL 6

Accepted Solution

by:
Javier Morales earned 0 total points
ID: 36522024
Hi,

@rolutola, I finally could stop the cluster, but ASM instance couldn't start because corruption on the OCR files. Once recovered from backup, I'm still facing the same issues with dg:

SQL>  select name, path from v$asm_disk;

NAME                           PATH
------------------------------ ------------------------------
                               ORCL:ASM_DATA02
                               ORCL:ASM_DATA03
                               ORCL:OCR_VOTE01
                               ORCL:OCR_VOTE02
                               ORCL:ASM_DATA04
_DROPPED_0001_OCR_VOTE
_DROPPED_0001_DATOS
ASM_DATA01                     ORCL:ASM_DATA01
OCR_VOTE03                     ORCL:OCR_VOTE03

9 rows selected.

Open in new window

The point is that actually, I finally could open the database, but I can't manage the _DROPPED_ disks and can't explain what happened with disk labeling.

My database is now open, but the ASM issue is still there...

@praveencpk, I did shutdown abort the ASM database, and it caused me the OCR corruption.
[root@bidegi02 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2900
         Available space (kbytes) :     259220
         ID                       : 1164751242
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check failed
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check failed
         Logical corruption check bypassed due to insufficient quorum

Open in new window


As I previous mention, I could restore the OCR and start the clusterware.

[root@bidegi02 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2932
         Available space (kbytes) :     259188
         ID                       : 1164751242
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded

Open in new window


I guess shutdown abort ASM instance when OCR or other clients are locking dg's is quite dangerous...

The point is ... how may I solve the ASM disks labeling and offline state?
0
 
LVL 6

Author Closing Comment

by:Javier Morales
ID: 37055454
Shutdown abort of the ASM instance in 11.2.0.2 causes the OCR may get corrupted. I had to restore OCR and reboot nodes.

With old OCR, rebooting the nodes, seemed to solve the issue with no additional tasks.

I split the points just to thank you the effort to answer my question. Thanks a lot.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Oracle SQL Char delimited error 5 30
sql for Oracle views 8 38
EXECUTE IMMEDIATE 5 36
Oracle -- identify blocking session 24 22
This article started out as an Experts-Exchange question, which then grew into a quick tip to go along with an IOUG presentation for the Collaborate confernce and then later grew again into a full blown article with expanded functionality and legacy…
How to Unravel a Tricky Query Introduction If you browse through the Oracle zones or any of the other database-related zones you'll come across some complicated solutions and sometimes you'll just have to wonder how anyone came up with them.  …
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Via a live example, show how to restore a database from backup after a simulated disk failure using RMAN.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now