Recently, we upgraded our Oracle database server Hardware and OS.
Our Oracle database was on a 2-node 12g RAC.
We went from RHEL 5.9 to 7.1
The plan is as below
The Issue I faced
All was fine until removal part.
But, while trying to add the node, I had the trouble at the cluster verify stage, while running clufy with -pre-flag as it sensed the bondings were not working.
When researched, finally the network team found out and told the Sys Admin that there is a change in the behavior of the setting for the parameter net.ipv4.conf.default.rp_filter.
Summary of the finding
RHEL7, RHEL6, and RHEL5 ship with a default /etc/sysctl.conf that sets this sysctl to 1, but the meaning of this value is different between the RHEL6 and the RHEL5 kernel.
So once this is set to 0 in RHEL 7, our network between the old hardware and new hardware could communicate with each other and the addnode.sh could successfully copy the Cluster Software from the surviving node to the newly added server and made it part of the Oracle cluster.
I hope you found this helpful.