Setting up Two Three NICs with One on a Different Subnet

Why not configure the /etc/sysconfig/network
with the IPs you want brought up which will translate into the routing table when they are brought back.
Or are you using the NetworkManager to manage your network/nic.

A netmask mismatch is is what you have node1 has a /8 while note2 has a /32

10.200.5.61/8 and 10.200.5.62/32 have a relationship such that node1 that has 10.200.5.61/8 treats the IP 10.200.5.62 as local to its segment which means it sends a request directly to the IP.
10.200.5.62/32 on the other hand has no one that it sees as local such that all its requests will be transmitted to the node's default gateway.

The other IPs are of the same but reverse nature.
To facilitate communication, they have to be on the same network segment or connected to a router that can differentiate between/among the segments.
.
This is a classic netmask issue.

Often, it is not advisable to have different segments on the same nic, but if you do, use index to bring up a second IP on the same nic.
ifconfig eno33557248:1 IP2 netmask ip2_netmask up

You can add it within the /etc/rc.local

look into bonded interfaces if you are trying to setup a single IP on multiple interfaces that will either double up the bandwidth, or provide for a failover in the event on of the network connections/devices fails.

It seems that I really have sort of a mess here.

This is what  I was given by the network administrator regarding the 10.200.5.0 network:

Two IPs: 10.200.5.61 and 10.200.5.62
Gateway: 10.200.1.1
Netmask: 255.255.0.0

That is what I used to try to get these two Linux servers to use one of their NICS in a different subnet than the other two NICs.
Two NICs should be on VLAN 17, and the other two NICs should be on VLAN 10 (the ones in 10.200.5.0).

I am not using the Network Manger to manage the networks.

How do I configure this via the /etc/sysconfig/network?

I just want to get this to work and that it is reliable the way it is set up.  This is going to be in production at some point.

Thanks.

The question/information is making it difficult to understand what the situation. Setup is to avoid

Look in /etc/sysconfig/networking
Run find /etc/sysconfig -name "ifcfg*"
There should be a list of ifcfg-<interface>
Each configured to bring an ip.
As to your IPs, if 10.200.5.61 and 10.200.5.62 are solely  for their own communication, each one has to have the same netmask.
/24 or /25 or /26 or /27 or /28 or /29 or /30 either one of those will include the two IPs in a single segment.

arnold:

Please see attached word file.  It might help you understand my setup and what I am looking to achieve.

Thanks.routing_table.docx

Ref you ifcfg-
You can set Gateway in the 10.200.5.x/16 on each node while removing the entry from the 172.16.y.y/16
Or, you could add Metric/weight to preserve the external access of the system should one link fail....

Your doc reflects that the segments/IPs have the same netmask /16

You're presenting data and restating what you posted.
I when dealing with an existing environment have to figure out how, why and what the interactions are and the possible impact of the change might be. It is possible that node01/02 can access resources via e 172.16.y.y/16 path while would not be able to reach those resources via the 10.200.x.x/16 path.

What exists on the 10.200.x.x/16 network beyond .....

I tried adding and removing the gateway as you suggested, but it did not work.

When you say:
"Your doc reflects that the segments/IPs have the same netmask /16"

Do you mean that the 10.200.x.x network is also in the same netmask/16 than the 172.16x.x nework?

I do not understand this question:  
"What exists on the 10.200.x.x/16 network beyond ..... "

I am going here with the IPs, gateway and any other information given to my by the network administrator.  As far as he told me, both the 172.16.x.x and the 10.200.x.x are in different subnets.

Based on the steps I have taken to set this second static route, am I following the proper steps to do so?

Thanks for all your input.

Dan Craciun and arnold:

I will run your comments by the network administrator tomorrow.  I am both the Linux and database administrator in the organization.  I am trying to set these two Linux servers to build an Oracle RAC environment.  The interesting part is that I thought setting this network configuration would be much more easier, and the I was going to find real difficult problems at the RAC level of things, and not at the Linux level of things.

Thank you guys.

Simply removing the entry gateway from the file  will be reflected after a reboot. The ifcfg- us the file used on bootup to setup the network.


You want to switch gateway from 172.16. To 10.200.
For the switch if the gateway to continue to function as the 172.16 you Have to ve certain their access us comparable.
I.e. The network topology.....

If anything, where do you see this in this network setup:

"f your network admin really uses a /16 broadcast domain"

I would like to point that out to the network administrator.

Thanks.

arnold:

I do not want to switch gateways from 172.16. To 10.200.  I want to have the 172.16.x.x addresses route via the default gateway 172.16.x.x, and the 10.200.x.x addresses route via the 10.200.1.1 gateway.

Thanks.

OK. I will relay all of this information to the network administrator, and we will go from there.  I will update you as soon as possible.

Thanks.

"A /30 255.255.255.252 network is enough..."

Does the above mean a network with only two hosts; namely 10.200.5.61 and 10.200.5.62 with a broadcast of 10.200.5.63?

"pri"

Is this the same as private internet?

"I.e. To establish an oracle RAC you need a Nic that connects the nodes (is your current setup an oracle cluster?)"

My current set up is an Oracle cluster of two nodes.

"Usually, the oracle dedicated network would not be a routeable, limited internal to the application so using/setting up gateway suggests that this is not a dedicated network for.....
A single network that would be used to access/query the db...."

Does this apply to all of the IPs in the RAC:  public, private, scan, and VIP?  


The 10.200.5.0 network has a range of IPs going from 10.200.5.1 to 10.200.5.71.  

Do the private addresses 10.200.5.61 and 10.200.5.62 need to be on their own VLAN all by themselves?  This means without any other IP such as 10.200.5.70, 10.200.5.15, 10.200.5.82, and so on that are currently part of the 10.200.5.x network?

Some of my questions here my be off topic, but I am asking them since they dictate the network design that I am trying to achieve.  Plus, knowing these answers will help me coordinate with the network administrator as to what I need from him.

Thanks.

In a cluster of two, commonly, the two nodes are directly connected by a single cross-cable (no switch us needed/used) which is where the /30'limiting of segment to minimize impact in the event network was not consulted in the choice of the network segment.
Commonly 192.168 based networks are used for oracle and other private segments since the larger 10. And even 172.16/12 are used widely in enterprise ....

If you are setting up multi head oracle RAC, presumably you have a dedicated connection to storage via FC or will you maintain the existing storage resource (DAS via SCSI/sas/esata)?
The data connection 10.200.5 should not be on the same Nic as the one externally....

What is the relation between 172.16.16 and 172.16.18?

Since one of those nics is the one you piggy back the 10.200.5

"In a cluster of two, commonly, the two nodes are directly connected by a single cross-cable (no switch us needed/used)..."
I have read the a cross-cable should not be used for the private interconnect.


"If you are setting up multi head oracle RAC, presumably you have a dedicated connection to storage via FC or will you maintain the existing storage resource (DAS via SCSI/sas/esata)?"
Yes, we do.  I have storage configured.  It uses SCSI/sas/sata according to the network admin.


"The data connection 10.200.5 should not be on the same Nic as the one externally...."
The 10.200.5.61 and 10.200.5.62 IPs are on their own NICs.


"What is the relation between 172.16.16 and 172.16.18?"
They are on the same subnet using the default gateway 172.16.16.1.

"This is a virtualized environment, or the configs you are posting are a test representation?"
This is a virtualized environment, and everything here is intended for production.


"The other issue I see with your tie up if two nics using separate IPs but the same gateway confusing
IMHO 172.16.18.x should be a sub-interface/sevondary ip to the interface where 172.16.16 us"
The two NICs with IPs 172.16.16.51 and 172.16.18.21 on node01 and 172.16.16.52 and 172.16.18.22 on node02 are for different purposes.  

Public IPs = 172.16.16.51 and  172.16.16.52.
IPs for me to log into the server (management IPs):  172.16.18.21 and 172.16.18.22.  The problem I see here is that they are in the same subnet than the public IPs.



"If you want to dedicate two nics/double bandwidth/fault tolerance, consider bonded interfaces
Bond0 with enoxxxxxxx and enozzzzzzz as subordinates."
No, I do not want to dedicate two nics like that.


I think I am getting a better understanding of the network requirement here: I need a set of IPs in the same subnet/VLAN for the public, scan, and VIP IPs.  I need two IPs on their own VLAN for private interconnect.  The VLAN for the private interconnect should be the VLAN with the shortest amount of hosts possible.

Let me know if my understanding is OK.


Thanks.
Thanks

Thanks.