• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 6982
  • Last Modified:

Network routing doesn't come up: RTNETLINK: Network is unreachable

Dear experts,

I have a CentOS 4.6 with two NICs:
eth0      Link encap:Ethernet  HWaddr 00:50:DA:B6:EF:E6
          inet addr:192.168.140.5  Bcast:192.168.140.255  Mask:255.255.255.0
          inet6 addr: fe80::250:daff:feb6:efe6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1659 errors:0 dropped:0 overruns:0 frame:0
          TX packets:161 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:124890 (121.9 KiB)  TX bytes:16949 (16.5 KiB)
          Interrupt:169 Base address:0x4400

eth1      Link encap:Ethernet  HWaddr 00:14:5E:1F:38:D1
          inet addr:192.168.100.110  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:85647 errors:0 dropped:0 overruns:0 frame:0
          TX packets:120177 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15686518 (14.9 MiB)  TX bytes:113475875 (108.2 MiB)
          Interrupt:169

Open in new window


But the routing table doesn't show eth0 network:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   *               255.255.255.0   U     0      0        0 eth1
172.16.146.0    *               255.255.255.0   U     0      0        0 vmnet8
172.16.153.0    *               255.255.255.0   U     0      0        0 vmnet1

Open in new window


Executing route add -net 192.168.140.0 netmask 255.255.255.0 eth0 doesn't return any errors, route stays on for half a second and then dissapears (again without any error).

The only error I get is, when doing ifup eth0:
RTNETLINK answers: Network is unreachable

I also received a report (it is a remote production server), that if take down eth1, eth0 network becomes available.

What is the answer, to get the second network available?
0
zeldi
Asked:
zeldi
1 Solution
 
zeldiAuthor Commented:
OK, I thing which I have found out: in /etc/sysconfig/network-scripts I had two files route-eth0 (referencing eth1 network) and route-eth1 (referencing eth0 network), which should be inversed. If i reverse the filenames I get the error
RTNETLINK answers: File exists
0
 
zeldiAuthor Commented:
Another one: if I remove files route-eth0 and route-eth1 the ifup eth0 doesn't return any error, I get a route to network:
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0 (it's 169!!!)
for a couple of seconds, and then it is removed
0
 
JelcinCommented:
Hi, not sure if this helps but on my Ubuntu ifup and ifdown does not work. Try "ifconfig eth0 up".
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 
giltjrCommented:
Do you see any thing in /var/messages?

Do you see anything in the output of dmesg related to eth0.

It sounds as if eth0 is having physical problems and although it shows up, it really is not.
0
 
zeldiAuthor Commented:
Hi,

as mentioned before, I get no messages either in dmesg nor in /var/log/messages which would show any errors.

But the eth0 stays up (as can be seen from ifconfig), just the route gets dropped, while the route is up, the ping to remote network works, and then the route is dropped and ping stops working.
0
 
giltjrCommented:
Without any overriding parameters if you do a ifup eth0 (or any other interface) it will attempt to do dhcp.  If there are no dhcp servers then it will use the "auto config" IP address of 169.254.0.0/16.

Try issuing:

     ip route list table all

or

     ip route list table cache

0
 
zeldiAuthor Commented:
As you have noticed in my first post, IPs are configured on the NICs (hence ifconfig shows the configured IPs).
Please find below attached the outputs from the ip route command

--------------------------------------------------------
ip route list table all
--------------------------------------------------------
192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.110
172.16.146.0/24 dev vmnet8 proto kernel scope link src 172.16.146.1
172.16.153.0/24 dev vmnet1 proto kernel scope link src 172.16.153.1
default via 192.168.100.1 dev eth1
broadcast 192.168.100.0 dev eth1 table local proto kernel scope link src 192.168.100.110
broadcast 172.16.146.0 dev vmnet8 table local proto kernel scope link src 172.16.146.1
local 172.16.146.1 dev vmnet8 table local proto kernel scope host src 172.16.146.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 172.16.153.255 dev vmnet1 table local proto kernel scope link src 172.16.153.1
broadcast 192.168.100.255 dev eth1 table local proto kernel scope link src 192.168.100.110
broadcast 172.16.146.255 dev vmnet8 table local proto kernel scope link src 172.16.146.1
broadcast 172.16.153.0 dev vmnet1 table local proto kernel scope link src 172.16.153.1
local 172.16.153.1 dev vmnet1 table local proto kernel scope host src 172.16.153.1
local 192.168.100.110 dev eth1 table local proto kernel scope host src 192.168.100.110
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local ::1 via :: dev lo proto none metric 0 mtu 16436 rtt 1ms rttvar 1ms cwnd 2 advmss 16376 hoplimit 64
local fe80::250:56ff:fec0:1 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64
local fe80::250:56ff:fec0:8 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64
fe80::/64 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64
fe80::/64 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64
ff00::/8 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1
ff00::/8 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1
unreachable default dev lo proto none metric -1 error -101 hoplimit 25

--------------------------------------------------------
ip route list table cache
--------------------------------------------------------
broadcast 192.168.100.255 from 192.168.100.37 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.112 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.118 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
192.168.100.118 from 192.168.100.110 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.44 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.34 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.114 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.19 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.10 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 from 127.0.0.1 dev lo
    cache <local>  mtu 16436 rtt 5ms rttvar 8ms cwnd 3 advmss 16396 hoplimit 64
broadcast 192.168.100.255 dev eth1  src 192.168.100.110
    cache <local,brd>  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.114 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.61 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 dev lo  src 127.0.0.1
    cache <local>  mtu 16436 advmss 16396 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.118 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.114 from 192.168.100.110 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.112 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.10 dev eth1  src 192.168.100.110
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.60 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.36 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 dev lo  src 127.0.0.1
    cache <local>  mtu 16436 advmss 16396 hoplimit 64
local 192.168.100.110 from 192.168.100.10 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
local 192.168.100.110 from 192.168.8.55 dev lo  src 192.168.100.110
    cache <local>  iif eth1
broadcast 192.168.100.255 from 192.168.100.16 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.8.55 from 192.168.100.110 tos 0x10 via 192.168.100.1 dev eth1
    cache  mtu 1500 rtt 105ms rttvar 105ms cwnd 3 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.91 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.8.55 from 192.168.100.110 via 192.168.100.1 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.140 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.38 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.10 from 192.168.100.110 dev eth1
    cache  mtu 1500 rtt 26ms rttvar 42ms cwnd 2 advmss 1460 hoplimit 64
local 192.168.100.110 from 192.168.110.6 dev lo  src 192.168.100.110
    cache <local>  iif eth1
192.168.100.112 from 192.168.100.110 dev eth1
    cache  mtu 1500 rtt 15ms rttvar 28ms cwnd 3 advmss 1460 hoplimit 64

Open in new window

0
 
giltjrCommented:
It appears that there is a script that is monitoring the interface and is issuing something like:

     ip route flush dev eth0

This will get rid of any/all route table entries that use the dev eth0.
0
 
zeldiAuthor Commented:
Just checking it right now..., I've just found process NetworkManager, which might be doing the dirty job...
0
 
memoo_Commented:
zeldi ,  i need to check this file /etc/sysconfig/networks-scripts/ifcfg-eth0 and ifcfg-eth1 too, i think that i can fix the error



 
0
 
mccrackyCommented:
If it's a server, get rid of network manager and use the network-scripts.  fix the two route-ethX files.  The "file exists" error, if I remember correctly, doesn't really matter and things should be fine.  If not you can manually set the routing in rc.local.
0
 
zeldiAuthor Commented:
yup, it was a NetworkManager, now it is removed from init.

@giltjr: I wasn't really thinking, that any daemon would be doing such a stupid thing, but after your hint did a thorough check of all the processes and there it was.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Tackle projects and never again get stuck behind a technical roadblock.
Join Now