We help IT Professionals succeed at work.

Need help in routing table on Solaris 10 non-global zone

Dip Sh
Dip Sh asked
on
Medium Priority
72 Views
Last Modified: 2020-02-25
Hi folks,

I am working on a Solaris-10 global server, which is hosting 7 non-global zones. There are two kind of network interfaces on our servers, admin and prod. There is one route, which is going via admin interface, I want to delete that and have that from prod interface.
e1000g0 is admin interface
e1000g3 is prod interface

 I want to delete this -
# netstat -nrv | grep 224.0.0.0
224.0.0.0            240.0.0.0       192.168.244.22       e1000g0:4  1500*    0   1 U        0      0
#

Open in new window

Since this is NGZ, I need to delete it from global zone. But I don't see this entry in route table on global server, what should I delete here? -
# netstat -nrv

IRE Table: IPv4
  Destination             Mask           Gateway          Device Mxfrg Rtt   Ref Flg  Out  In/Fwd
-------------------- --------------- -------------------- ------ ----- ----- --- --- ----- ------
default              0.0.0.0         192.168.241.1                1500*    0   1 UG    1072      0
default              0.0.0.0         216.221.133.193      e1000g3  1500*    0   1 UG   13020      0
10.0.0.0             255.0.0.0       192.168.244.1                1500*    0   1 UG       0      0
172.16.0.0           255.255.0.0     192.168.244.1                1500*    0   1 UG       0      0
192.168.64.0         255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0
192.168.69.0         255.255.255.0   192.168.244.1                1500*    0   1 UG      54      0
192.168.78.0         255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0
192.168.110.0        255.255.255.0   192.168.244.1                1500*    0   1 UG      43      0
192.168.201.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0
192.168.202.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0
192.168.241.0        255.255.255.128 192.168.241.20       e1000g1  1500*    0   1 U        4      0
192.168.244.0        255.255.255.128 192.168.244.20       e1000g0  1500*    0   1 U        7      0
192.168.244.0        255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0
192.168.244.22       255.255.255.255 216.221.133.193              1500*    0   1 UGH      0      0
192.168.246.64       255.255.255.224 192.168.244.1                1500*    0   1 UG       0      0
216.221.133.192      255.255.255.192 216.221.133.250      e1000g3  1500*    0   1 U        7      0
224.0.0.0            240.0.0.0       192.168.244.20       e1000g0  1500*    0   1 U        0      0
127.0.0.1            255.255.255.255 127.0.0.1            lo0     8232*    0   2 UH       4      0

Open in new window

I assumed that route is coming from entries in /etc/inet/static_routes. I commented (see below) a line, rebooted server, but still no luck.
# cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.
default 192.168.241.1
-net 192.168.110.0/24 192.168.244.1
-net 192.168.201.0/24 192.168.244.1
-net 192.168.202.0/24 192.168.244.1
-net 192.168.246.64 -netmask 255.255.255.224 192.168.244.1
-net 172.16.0.0/16 192.168.244.1
10.0.0.0/8 192.168.244.1
192.168.64.0/22 192.168.244.1
192.168.78.0/24 192.168.244.1
192.168.69.0/24 192.168.244.1
net 192.168.244.0 -netmask 255.255.252.0 192.168.244.1
224.0.0.0/4 216.221.133.251 -ifp e1000g3:5
#224.0.0.0/4 192.168.244.20 -ifp e1000g0
host 192.168.244.22 216.221.133.193
#

Open in new window

Then I want it to be routed thorugh prod interface (216.221.133.193). This output is from NGZ -
# netstat -nrv | grep default
default              0.0.0.0         216.221.133.193      e1000g3  1500*    0   1 UG   13031      0
#

Open in new window

Any pointers and advice please?

Thanks in advance.
Comment
Watch Question

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Do you know which roure you want to delete?

Your /etc/hostname.e1000g0 make sure you are not defining gateway parameter.
This is the source of the default gateway.

You can also user route delete net 224.0.0.0 mask 240.0.0

You are bringing up this network when you are creating the index:4 and bringing the IP up.

Author

Commented:

I don't see a gateway here on these files :

# ls -ltr /etc/hostna* 
-rw-r--r--   1 root     root          19 Aug  7  2012 /etc/hostname.e1000g1 
-rw-r--r--   1 root     root          19 Feb 26  2013 /etc/hostname.e1000g3 
-rw-r--r--   1 root     root          20 May 11  2015 /etc/hostname.e1000g0 
# cat /etc/hostna* 
ddd-zonemgr1-admin 
ddd-zonemgr1-prod 
ddd-zonemgr1-data 
# cat /etc/hosts | grep ddd-zonemgr1 
192.168.241.20  ddd-zonemgr1-prod ddd-zonemgr1-prod.xypoint.com 
216.221.133.250 ddd-zonemgr1-data ddd-zonemgr1-data.xypoint.com 
192.168.244.20  ddd-zonemgr1-admin ddd-zonemgr1-admin.xypoint.com \ 
#

I want to delete "224.0.0.0" route from below and want to add it through e1000g3

# netstat -nrv 
 
IRE Table: IPv4 
  Destination             Mask           Gateway          Device Mxfrg Rtt   Ref Flg  Out  In/Fwd 
-------------------- --------------- -------------------- ------ ----- ----- --- --- ----- ------ 
default              0.0.0.0         216.221.133.193      e1000g3  1500*    0   1 UG   13612      0 
10.0.0.0             255.0.0.0       192.168.244.1                1500*    0   1 UG       0      0 
172.16.0.0           255.255.0.0     192.168.244.1                1500*    0   1 UG       0      0 
192.168.64.0         255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.69.0         255.255.255.0   192.168.244.1                1500*    0   1 UG      55      0 
192.168.78.0         255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.110.0        255.255.255.0   192.168.244.1                1500*    0   1 UG      45      0 
192.168.201.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.202.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.244.0        255.255.255.128 192.168.244.22       e1000g0:4  1500*    0   1 U        7      0 
192.168.244.0        255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.244.22       255.255.255.255 216.221.133.193              1500*    0   1 UGH      0      0 
192.168.246.64       255.255.255.224 192.168.244.1                1500*    0   1 UG       0      0 
216.221.133.192      255.255.255.192 216.221.133.251      e1000g3:3  1500*    0   1 U       83      0 
224.0.0.0            240.0.0.0       192.168.244.22       e1000g0:4  1500*    0   1 U        0      0 
127.0.0.1            255.255.255.255 127.0.0.1            lo0:4   8232*    0  40 UH   54733      0 
#

But the issue is, the above output is from NGZ and I can't delete/add anything on NGZ.

# route -p delete 224.0.0.0 240.0.0.0 -netmask 192.168.244.22 
delete net 224.0.0.0: gateway 240.0.0.0: insufficient privileges 
#

Route delete/add should be done from Global server. But I don't see above route on that.

# netstat -nrv | grep 224.0.0.0 
224.0.0.0            240.0.0.0       192.168.244.20       e1000g0  1500*    0   1 U        0      0 
#
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
240.0.0.0 is not a gateway, it is the mask
Trying to locate the source that sets it.

The clue is that the interface binding points to e1000g0:4 where are you bringing up ip, indexes?

Author

Commented:

e1000g0 is a physical interface on global server. There are multiple NGZ running on this. Each NGZ, which requires to have IP from this interface will be defined in zonecfg like below. 

net: 
        address: zone-admin 
        physical: e1000g0 
        defrouter not specified 
net: 
        address: zone-data 
        physical: e1000g3 
        defrouter not specified 
device 
        match: /dev/e1000g0 
device 
        match: /dev/e1000g3

Other than that, I can't find it.

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
More the record you want removed is e1000g0:4
The routing seems to be added from within the container, zone.

Author

Commented:

On global server e1000g0:4 is admin network interface for zone-123, which I don't want to remove.

# ifconfig e1000g0:4 
e1000g0:4: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
        zone zone-123 
        inet 192.168.244.22 netmask ffffff80 broadcast 192.168.244.127 
#

Here is output from within zone, if this gives any clue

# ifconfig -a 
lo0:4: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
        inet 127.0.0.1 netmask ff000000 
e1000g0:4: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
        inet 192.168.244.22 netmask ffffff80 broadcast 192.168.244.127 
e1000g3:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
        inet 216.221.133.251 netmask ffffffc0 broadcast 216.221.133.255 
# netstat -nrv 
 
IRE Table: IPv4 
  Destination             Mask           Gateway          Device Mxfrg Rtt   Ref Flg  Out  In/Fwd 
-------------------- --------------- -------------------- ------ ----- ----- --- --- ----- ------ 
default              0.0.0.0         216.221.133.193      e1000g3  1500*    0   1 UG   14008      0 
10.0.0.0             255.0.0.0       192.168.244.1                1500*    0   1 UG       0      0 
172.16.0.0           255.255.0.0     192.168.244.1                1500*    0   1 UG       0      0 
192.168.64.0         255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.69.0         255.255.255.0   192.168.244.1                1500*    0   1 UG      57      0 
192.168.78.0         255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.110.0        255.255.255.0   192.168.244.1                1500*    0   1 UG      45      0 
192.168.201.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.202.0        255.255.255.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.244.0        255.255.255.128 192.168.244.22       e1000g0:4  1500*    0   1 U        7      0 
192.168.244.0        255.255.252.0   192.168.244.1                1500*    0   1 UG       0      0 
192.168.244.22       255.255.255.255 216.221.133.193              1500*    0   1 UGH      0      0 
192.168.246.64       255.255.255.224 192.168.244.1                1500*    0   1 UG       0      0 
216.221.133.192      255.255.255.192 216.221.133.251      e1000g3:3  1500*    0   1 U       85      0 
224.0.0.0            240.0.0.0       192.168.244.22       e1000g0:4  1500*    0   1 U        0      0 
127.0.0.1            255.255.255.255 127.0.0.1            lo0:4   8232*    0  40 UH   56271      0 
#

Background ::: Our network team has a problem with this. Per their email request -

I need just one static route for multicast on Prod side like this:

224.0.0.0            216.221.133.193      UG        1          0

And remove this one on Admin side:

224.0.0.0            192.168.244.22       U         1          0 e1000g0:5
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
The default route is through the 216.221.133.193,

not sure why you are trying to remove the reference is it interfering with anything?

To clarify ,see https://www.thegeekdiary.com/solaris-netstat-understanding-the-output-when-displaying-routing-table/

Flags, U means up, G is the one designates which path it will take to get to this network.

Author

Commented:

I don't have full picture, but I know there are some issues on a software application and network side and they suggested that this should go via prod gateway.

In our environment, admin interface is not used for applications, ideally.

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Are you sure the issue relates to the Multi-cast network?
if you need to have access to Multi-cast on the admin network, you would need to

The issue is it that you can not get multi-cast packes on the admin network, or you are getting them and do not need/want them?

Author

Commented:

Yes, thats what I got from network guy.

They want multi-cast packets should go via prod gw (216.221.133.193), instead of admin gw (192.168.244.22).

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Based on your netstat report,
224.0.0.0            240.0.0.0       192.168.244.22       e1000g0:4  1500*    0   1 U        0      0
0 packets in and out.

The other, your static route, binds the multicast to the e1000g0 versus an instant

224.0.0.0/4 216.221.133.251 -ifp e1000g3:5
#224.0.0.0/4 192.168.244.20 -ifp e1000g0

ifconfig -a look at which interface is defined as MULTICAST.

Author

Commented:

But e1000g0 is already commented (and then rebooted server) in /etc/inet/static_routes. Any other way to break this binding with   e1000g0 ?

I see MULTICAST flag on all interfaces on global server


# ifconfig -a | grep MULTICAST 
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:2: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:3: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:4: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:5: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:6: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
lo0:7: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:4: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:5: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:6: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g0:7: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 
e1000g1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 
e1000g3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:4: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:5: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
e1000g3:6: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 
#
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
That is common, but only the one with the default gateway will actually receive and send.
you could as part of the config of the interface disable the multicast

you still may need within segment multicast.
Are your network folks complain about multicast floods, ?

Author

Commented:

Not flooding, but they are trying to send multicast packets. It is going via Admin and they want to sent it through Prod (I guess)

Apps guys is using this command to send multicast 

/export/home/trmc/mcSend -a 239.93.97.0 -p 9700 -t 6

From Network guy : The PCAP shows you trying to send multicast with the admin interface instead of the prod. This is a mess and not ideal. Setup it up in way to use the prod (data) interface 216.221.133.251 , not the admin interface 192.168.244.32. I have already allowed multicast for that VLAN.

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Much depends on how much flexibility you have to try
one option is to change the index of the interface to make the desired e1000g3 as the lower index by increasing the index on the e1000g0:4

see if mcsend have an option to specify an interface or
ifconfig e1000g0 index 6

note sub-interfaces would downshift as well.

The mcsend might be picking the first non lo interface to attach.

Author

Commented:

mcsend is being used to test it. But ultimately their application should use prod for multicast packets. So tweaking that would not be ideal here.
I made index 5 for e1000g0, but looks like it didn't helped.


How I tested : I took two sessions of zone-1. On one session I issued mcSend command

# mcSend -a 239.93.97.0 -p 9700 -t 6 
multicastIP: 239.93.97.0 
multicastIF: 0.0.0.0 
multicastPort: 9700 
TTL:  6 
loopback enabled: 0 

On another session, I ran snoop command on e1000g0 and then e1000g3. It still doesn't show packets on e1000g3

# snoop -d e1000g3 -x0 host 239.93.97.0 
Using device e1000g3 (promiscuous mode) 
^C 
# 
# snoop -d e1000g0 -x0 host 239.93.97.0 
Using device e1000g0 (promiscuous mode) 
zone-1 -> 239.93.97.0  UDP D=9700 S=51392 LEN=136 
 
           0: 0100 5e5d 6100 0014 4f78 f884 0800 4500    ..^]a...Ox....E. 
          16: 009c 3153 0000 0611 0000 c0a8 f416 ef5d    ..1S...........] 
          32: 6100 c8c0 25e4 0088 05b7 4865 6c6c 6f2c    a...%.....Hello, 
          48: 2057 6f72 6c64 2100 0000 0000 0000 0000     World!......... 
          64: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
          80: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
          96: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         112: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         128: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         144: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         160: 0000 0000 0000 0000 0000                   .......... 
 
zone-1 -> 239.93.97.0  UDP D=9700 S=51392 LEN=136 
 
           0: 0100 5e5d 6100 0014 4f78 f884 0800 4500    ..^]a...Ox....E. 
          16: 009c 3154 0000 0611 0000 c0a8 f416 ef5d    ..1T...........] 
          32: 6100 c8c0 25e4 0088 05b7 4865 6c6c 6f2c    a...%.....Hello, 
          48: 2057 6f72 6c64 2100 0000 0000 0000 0000     World!......... 
          64: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
          80: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
          96: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         112: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         128: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         144: 0000 0000 0000 0000 0000 0000 0000 0000    ................ 
         160: 0000 0000 0000 0000 0000                   .......... 
 
^C 
#
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
The other option, is to switch use of interfaces.
Make e10000g0 connected to the production network while e10000g3 will be on the admin

Author

Commented:

That can be an alternate, but that will change a lot of things on the server. Trying to get the idea, if I can do something on OS end.

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
you have to adjust all the zones to use the other interface for network connection.

I think the issue is that it binds to all (0.0.0.0) and multicasts across all of them. or alternatively it binds to the first interface in this case it is the admin network.
Commented:

I found the issue, it was tricky.

Multicast packets are going through admin interface because it is managed by /lib/svc/method/net-svc configuration. One of its parameter says 

if [ "$_INIT_NET_STRATEGY" = "dhcp" ]; then 
        mcastif=`/sbin/dhcpinfo Yiaddr` || mcastif=$_INIT_UTS_NODENAME 
else 
        mcastif=$_INIT_UTS_NODENAME 
fi

It says multicast should go via NODENAME. That means, whatever is hostname and uname -n returns. By default hostname is set to admin interface. Two changes I made :

-Changed hostname and zonename in /etc/hosts, so at zonemanager level, it look to pubic IP
-In zonecfg, I moved up the public interface, so it goes FIRST in zone description file.


Thanks for your efforts and help