[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Oracle 10gR2 RAC struggling ...

Posted on 2009-12-20
20
Medium Priority
?
639 Views
Last Modified: 2013-12-18
Hi gurus,

I've installed/configured and it was working too, but I've to move severs to another location, and now struggling to bring it back to work.

Help, assistance & guidance will highly be appreciated.

Getting this error:

[root@rac1 bin]# ./crsctl query css votedisk
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
[root@rac1 bin]# ./crsctl query css votedisk
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
[root@rac1 bin]#

I've tried to work step by step, but still couple of things are giving error:
Following these links:
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_11.shtml

Thanks in advance.
0
Comment
Question by:el123
  • 12
  • 8
20 Comments
 
LVL 3

Expert Comment

by:logictank
ID: 26092096
What storage are you using for OCR & Voting (Block, OCFS2, Raw)?  Have you verified that the ocr/voting storage locations are available and the permissions are correct, for example:
OCR Disks:
The OCR disk must be owned by root, must be in the oinstall group, and must have permissions set to 640.  Run the chown & chmod on both RAC Nodes.

sudo chown root:oinstall /dev/mpath/mpath_ocr1
sudo chown root:oinstall /dev/mpath/mpath_ocr2
sudo chmod 640 /dev/mpath/mpath_ocr1
sudo chmod 640 /dev/mpath/mpath_ocr2

sudo /oracle/app/oracle/product/10.2.0/crs/bin/ocrcheck

Voting Disks:
The voting disk must be owned by the oracle user, must be in the oinstall group, and must have permissions set to 644.  Run the chown & chmod on both RAC Nodes.

sudo chown oracle:oinstall /dev/mpath/mpath_vote1
sudo chown oracle:oinstall /dev/mpath/mpath_vote2
sudo chown oracle:oinstall /dev/mpath/mpath_vote3
sudo chmod 644 /dev/mpath/mpath_vote1
sudo chmod 644 /dev/mpath/mpath_vote2
sudo chmod 644 /dev/mpath/mpath_vote3
 
sudo /oracle/app/oracle/product/10.2.0/crs/bin/crsctl query css votedisk

You may have lost the permissions on your storage locations during the reboot if you're using block or raw devices.
 
0
 

Author Comment

by:el123
ID: 26092738
first of all thanks logictank for your reply.

I'm using ISCSI, as mentioned in the doc, openfiler for the shared storage, can't say where the problem is, but getting this error msg:

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

Please suggest ... how to fix ... thanks.
0
 

Author Comment

by:el123
ID: 26092772
problem is same as mentioned in this article ... but only one difference he is using raw devices and I'm using iscsi ... :-( how to fix ... thanks in advance
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:el123
ID: 26092834
[root@rac1 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
[root@rac1 bin]#
0
 
LVL 3

Expert Comment

by:logictank
ID: 26093369
Sorry for the delay, a few questions:
- Did you retain the same hostname, vip, and priv ip addresses after the move or are you using new addresses?
- Have you verified that your iSCSI devices (Where OCR and Voting Disks reside) are available and accessible after the server move/reboot, can you see the device paths under /dev and are the permissions set as mentioned above.
- Please post the device paths for your OCR & Voting disks
- Are you seeing any errors in your $CRS_HOME/log directories or in the /tmp/crsctl logs as mentioned in the note you posted above?
- Verified that /etc/oracle/ocr.loc is pointing to the correct OCR & OCR Mirror location
0
 

Author Comment

by:el123
ID: 26095931
thanks again a lot ... logictank, here are the answers:

1. haven't touched any thing, just move the servers from place A to place and then plug them in
2. this is the question that I need your help ... how to identify, couldn't do this yet, struggling how to find this?
and I think the rest of the qs are similar, as I'm not able to answer the 2nd one, they all interlinked ... right

please suggest how to find this, I've run some of these ... it might help:

[root@rac2 ~]# ps -aef | grep "init\."
root      4812     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run
root      4863     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root      4864     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run
root      5286  4812  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
root      5436  4863  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
root      5773  4864  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
[root@rac2 ~]# ps -ef | grep iscsi
root      3060     1  0 Dec20 ?        00:00:00 /sbin/iscsid
root      3062  3060  0 Dec20 ?        00:00:00 /sbin/iscsid
root      3080     6  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3081     6  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3082     6  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3083     6  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3086     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3087     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3088     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3089     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3116     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3117     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      9071  7907  0 10:17 pts/1    00:00:00 grep iscsi
[root@rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       15G  8.9G  5.1G  64% /
/dev/sda1              99M   16M   78M  17% /boot
none                  505M     0  505M   0% /dev/shm
[root@rac2 ~]#


0
 
LVL 3

Expert Comment

by:logictank
ID: 26098144
You should find the location to the OCR disks in your ocr.loc file, send over the contents of '/etc/oracle/ocr.loc'.  We need to verify the location of your OCR and Voting disks first and then we can do further investigation on how to make them available.
0
 

Author Comment

by:el123
ID: 26098241
[root@rac1 ~]# cat /etc/oracle/ocr.loc
ocrconfig_loc=/u02/oradata/orcl/OCRFile
ocrmirrorconfig_loc=/u02/oradata/orcl/OCRFile_mirror
local_only=FALSE
[root@rac1 ~]#
0
 
LVL 3

Expert Comment

by:logictank
ID: 26098463
Looking at your df command above it appears that the following file system/mount point is missing on your host which contains your OCR file and mirror:
/u02/oradata/orcl
Without this file system/mount point available the CRS services are unable to access the OCR registry information.  You can try a mount -a as root to see if it brings back the mount otherwise you'll need to determine why this mount point is no longer available.
0
 

Author Comment

by:el123
ID: 26098536
no luck ... still showing the same,

[root@rac1 oracle]# mount -a
[root@rac1 oracle]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       15G   13G  1.7G  88% /
/dev/sda1              99M   16M   78M  17% /boot
none                  505M     0  505M   0% /dev/shm
[root@rac1 oracle]#

how to bring these mount points back, if you can direct to any link or any steps, how to troubleshoot this scenario ... that'll be very helpful, thanks.
0
 
LVL 3

Expert Comment

by:logictank
ID: 26098588
To troubleshoot the storage please send over /etc/fstab, also wanted to verify that the following path is not available under '/':
/u02/oradata/orcl/
That this is in fact a separate mount point, we can get this from /etc/fstab.
0
 

Author Comment

by:el123
ID: 26098694
[root@rac1 oracle]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
/dev/hda                /media/cdrom            auto    pamconsole,exec,noauto,managed 0 0
/dev/fd0                /media/floppy           auto    pamconsole,exec,noauto,managed 0 0
[root@rac1 oracle]#
0
 
LVL 3

Accepted Solution

by:
logictank earned 2000 total points
ID: 26098838
I don't see an entry in /etc/fstab for '/u02/oradata/orcl/', if this is the shared iSCSI volume where the OCR and Voting disks reside I don't see an entry for it in /etc/fstab.  I would expect to see something like (just an example):

/dev/sdd1 /u02/oradata/orcl ext3 _netdev 0 0

Also, you've verified that these locations are not available correct?

ocrconfig_loc=/u02/oradata/orcl/OCRFile
ocrmirrorconfig_loc=/u02/oradata/orcl/OCRFile_mirror

ls -al  /u02/oradata/orcl/OCRFile
ls -al /u02/oradata/orcl/OCRFile_mirror
0
 

Author Comment

by:el123
ID: 26098851
[root@rac1 oracle]# ls -al  /u02/oradata/orcl/OCRFile
ls: /u02/oradata/orcl/OCRFile: No such file or directory
[root@rac1 oracle]# ls -al /u02/oradata/orcl/OCRFile_mirror
ls: /u02/oradata/orcl/OCRFile_mirror: No such file or directory
[root@rac1 oracle]#

the env is 2 nodes and shared storage using openfiler, even at openfiler, I don't see any other mount points ... what to do, how to fix this problem, is there any possible, that we chat or talk or call, to fix this ...?
0
 

Author Comment

by:el123
ID: 26098884
if you don't mind ... :-(
0
 
LVL 3

Expert Comment

by:logictank
ID: 26099007
Sure, email me at logictank@logictank.biz and we can exchange contact information.
0
 

Author Comment

by:el123
ID: 26099060
thanks ... you've got it
0
 
LVL 3

Expert Comment

by:logictank
ID: 26099089
Just found a note on an openfiler issue after reboot losing the logical volume/volume group information.  This might be your fix.
http://opensourceexperiments.wordpress.com/2008/05/01/iscsi-no-luns-detected-for-session-when-using-openfiler/ 
0
 

Author Closing Comment

by:el123
ID: 31668259
Outstanding ... unbelievably outrageous ... never met an Oracle guru like logictank ...

Thumbs Up for him, please have all your Oracle questions and toughest questions answered by ... logictank

Thanks Logictank ... for your assistance, help and guidance.

Very best regards.
0

Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Truncate is a DDL Command where as Delete is a DML Command. Both will delete data from table, but what is the difference between these below statements truncate table <table_name> ?? delete from <table_name> ?? The first command cannot be …
Working with Network Access Control Lists in Oracle 11g (part 2) Part 1: http://www.e-e.com/A_8429.html Previously, I introduced the basics of network ACL's including how to create, delete and modify entries to allow and deny access.  For many…
Via a live example, show how to take different types of Oracle backups using RMAN.
This video shows how to Export data from an Oracle database using the Datapump Export Utility.  The corresponding Datapump Import utility is also discussed and demonstrated.
Suggested Courses

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question