Oracle 10gR2 RAC struggling ...

What storage are you using for OCR & Voting (Block, OCFS2, Raw)?  Have you verified that the ocr/voting storage locations are available and the permissions are correct, for example:
OCR Disks:
The OCR disk must be owned by root, must be in the oinstall group, and must have permissions set to 640.  Run the chown & chmod on both RAC Nodes.

sudo chown root:oinstall /dev/mpath/mpath_ocr1
sudo chown root:oinstall /dev/mpath/mpath_ocr2
sudo chmod 640 /dev/mpath/mpath_ocr1
sudo chmod 640 /dev/mpath/mpath_ocr2

sudo /oracle/app/oracle/product/10.2.0/crs/bin/ocrcheck

Voting Disks:
The voting disk must be owned by the oracle user, must be in the oinstall group, and must have permissions set to 644.  Run the chown & chmod on both RAC Nodes.

sudo chown oracle:oinstall /dev/mpath/mpath_vote1
sudo chown oracle:oinstall /dev/mpath/mpath_vote2
sudo chown oracle:oinstall /dev/mpath/mpath_vote3
sudo chmod 644 /dev/mpath/mpath_vote1
sudo chmod 644 /dev/mpath/mpath_vote2
sudo chmod 644 /dev/mpath/mpath_vote3
 
sudo /oracle/app/oracle/product/10.2.0/crs/bin/crsctl query css votedisk

You may have lost the permissions on your storage locations during the reboot if you're using block or raw devices.
 

first of all thanks logictank for your reply.

I'm using ISCSI, as mentioned in the doc, openfiler for the shared storage, can't say where the problem is, but getting this error msg:

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

Please suggest ... how to fix ... thanks.

problem is same as mentioned in this article ... but only one difference he is using raw devices and I'm using iscsi ... :-( how to fix ... thanks in advance

article:
http://surachartopun.com/2009/04/why-my-oracle-cluster-could-not-start.html

[root@rac1 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
[root@rac1 bin]#

Sorry for the delay, a few questions:
- Did you retain the same hostname, vip, and priv ip addresses after the move or are you using new addresses?
- Have you verified that your iSCSI devices (Where OCR and Voting Disks reside) are available and accessible after the server move/reboot, can you see the device paths under /dev and are the permissions set as mentioned above.
- Please post the device paths for your OCR & Voting disks
- Are you seeing any errors in your $CRS_HOME/log directories or in the /tmp/crsctl logs as mentioned in the note you posted above?
- Verified that /etc/oracle/ocr.loc is pointing to the correct OCR & OCR Mirror location

thanks again a lot ... logictank, here are the answers:

1. haven't touched any thing, just move the servers from place A to place and then plug them in
2. this is the question that I need your help ... how to identify, couldn't do this yet, struggling how to find this?
and I think the rest of the qs are similar, as I'm not able to answer the 2nd one, they all interlinked ... right

please suggest how to find this, I've run some of these ... it might help:

[root@rac2 ~]# ps -aef | grep "init\."
root      4812     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run
root      4863     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root      4864     1  0 Dec20 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run
root      5286  4812  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
root      5436  4863  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
root      5773  4864  0 Dec20 ?        00:00:02 /bin/sh /etc/init.d/init.cssd startcheck
[root@rac2 ~]# ps -ef | grep iscsi
root      3060     1  0 Dec20 ?        00:00:00 /sbin/iscsid
root      3062  3060  0 Dec20 ?        00:00:00 /sbin/iscsid
root      3080     6  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3081     6  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3082     6  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3083     6  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3086     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3087     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3088     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3089     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      3116     7  0 Dec20 ?        00:00:00 [iscsi-tx]
root      3117     7  0 Dec20 ?        00:00:00 [iscsi-rx]
root      9071  7907  0 10:17 pts/1    00:00:00 grep iscsi
[root@rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       15G  8.9G  5.1G  64% /
/dev/sda1              99M   16M   78M  17% /boot
none                  505M     0  505M   0% /dev/shm
[root@rac2 ~]#

You should find the location to the OCR disks in your ocr.loc file, send over the contents of '/etc/oracle/ocr.loc'.  We need to verify the location of your OCR and Voting disks first and then we can do further investigation on how to make them available.

[root@rac1 ~]# cat /etc/oracle/ocr.loc
ocrconfig_loc=/u02/oradata/orcl/OCRFile
ocrmirrorconfig_loc=/u02/oradata/orcl/OCRFile_mirror
local_only=FALSE
[root@rac1 ~]#

Looking at your df command above it appears that the following file system/mount point is missing on your host which contains your OCR file and mirror:
/u02/oradata/orcl
Without this file system/mount point available the CRS services are unable to access the OCR registry information.  You can try a mount -a as root to see if it brings back the mount otherwise you'll need to determine why this mount point is no longer available.

no luck ... still showing the same,

[root@rac1 oracle]# mount -a
[root@rac1 oracle]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       15G   13G  1.7G  88% /
/dev/sda1              99M   16M   78M  17% /boot
none                  505M     0  505M   0% /dev/shm
[root@rac1 oracle]#

how to bring these mount points back, if you can direct to any link or any steps, how to troubleshoot this scenario ... that'll be very helpful, thanks.

To troubleshoot the storage please send over /etc/fstab, also wanted to verify that the following path is not available under '/':
/u02/oradata/orcl/
That this is in fact a separate mount point, we can get this from /etc/fstab.

[root@rac1 oracle]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
/dev/hda                /media/cdrom            auto    pamconsole,exec,noauto,managed 0 0
/dev/fd0                /media/floppy           auto    pamconsole,exec,noauto,managed 0 0
[root@rac1 oracle]#

[root@rac1 oracle]# ls -al  /u02/oradata/orcl/OCRFile
ls: /u02/oradata/orcl/OCRFile: No such file or directory
[root@rac1 oracle]# ls -al /u02/oradata/orcl/OCRFile_mirror
ls: /u02/oradata/orcl/OCRFile_mirror: No such file or directory
[root@rac1 oracle]#

the env is 2 nodes and shared storage using openfiler, even at openfiler, I don't see any other mount points ... what to do, how to fix this problem, is there any possible, that we chat or talk or call, to fix this ...?

if you don't mind ... :-(

Sure, email me at logictank@logictank.biz and we can exchange contact information.

thanks ... you've got it

Just found a note on an openfiler issue after reboot losing the logical volume/volume group information.  This might be your fix.
http://opensourceexperiments.wordpress.com/2008/05/01/iscsi-no-luns-detected-for-session-when-using-openfiler/ 

Outstanding ... unbelievably outrageous ... never met an Oracle guru like logictank ...

Thumbs Up for him, please have all your Oracle questions and toughest questions answered by ... logictank

Thanks Logictank ... for your assistance, help and guidance.

Very best regards.