Solved

Network routing doesn't come up: RTNETLINK: Network is unreachable

Posted on 2010-09-08
12
6,181 Views
Last Modified: 2012-08-13
Dear experts,

I have a CentOS 4.6 with two NICs:
eth0      Link encap:Ethernet  HWaddr 00:50:DA:B6:EF:E6
          inet addr:192.168.140.5  Bcast:192.168.140.255  Mask:255.255.255.0
          inet6 addr: fe80::250:daff:feb6:efe6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1659 errors:0 dropped:0 overruns:0 frame:0
          TX packets:161 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:124890 (121.9 KiB)  TX bytes:16949 (16.5 KiB)
          Interrupt:169 Base address:0x4400

eth1      Link encap:Ethernet  HWaddr 00:14:5E:1F:38:D1
          inet addr:192.168.100.110  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:85647 errors:0 dropped:0 overruns:0 frame:0
          TX packets:120177 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15686518 (14.9 MiB)  TX bytes:113475875 (108.2 MiB)
          Interrupt:169

Open in new window


But the routing table doesn't show eth0 network:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   *               255.255.255.0   U     0      0        0 eth1
172.16.146.0    *               255.255.255.0   U     0      0        0 vmnet8
172.16.153.0    *               255.255.255.0   U     0      0        0 vmnet1

Open in new window


Executing route add -net 192.168.140.0 netmask 255.255.255.0 eth0 doesn't return any errors, route stays on for half a second and then dissapears (again without any error).

The only error I get is, when doing ifup eth0:
RTNETLINK answers: Network is unreachable

I also received a report (it is a remote production server), that if take down eth1, eth0 network becomes available.

What is the answer, to get the second network available?
0
Comment
Question by:zeldi
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
12 Comments
 

Author Comment

by:zeldi
ID: 33625727
OK, I thing which I have found out: in /etc/sysconfig/network-scripts I had two files route-eth0 (referencing eth1 network) and route-eth1 (referencing eth0 network), which should be inversed. If i reverse the filenames I get the error
RTNETLINK answers: File exists
0
 

Author Comment

by:zeldi
ID: 33625751
Another one: if I remove files route-eth0 and route-eth1 the ifup eth0 doesn't return any error, I get a route to network:
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0 (it's 169!!!)
for a couple of seconds, and then it is removed
0
 
LVL 6

Expert Comment

by:Jelcin
ID: 33625824
Hi, not sure if this helps but on my Ubuntu ifup and ifdown does not work. Try "ifconfig eth0 up".
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 57

Expert Comment

by:giltjr
ID: 33625836
Do you see any thing in /var/messages?

Do you see anything in the output of dmesg related to eth0.

It sounds as if eth0 is having physical problems and although it shows up, it really is not.
0
 

Author Comment

by:zeldi
ID: 33625889
Hi,

as mentioned before, I get no messages either in dmesg nor in /var/log/messages which would show any errors.

But the eth0 stays up (as can be seen from ifconfig), just the route gets dropped, while the route is up, the ping to remote network works, and then the route is dropped and ping stops working.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33627673
Without any overriding parameters if you do a ifup eth0 (or any other interface) it will attempt to do dhcp.  If there are no dhcp servers then it will use the "auto config" IP address of 169.254.0.0/16.

Try issuing:

     ip route list table all

or

     ip route list table cache

0
 

Author Comment

by:zeldi
ID: 33627854
As you have noticed in my first post, IPs are configured on the NICs (hence ifconfig shows the configured IPs).
Please find below attached the outputs from the ip route command

--------------------------------------------------------
ip route list table all
--------------------------------------------------------
192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.110
172.16.146.0/24 dev vmnet8 proto kernel scope link src 172.16.146.1
172.16.153.0/24 dev vmnet1 proto kernel scope link src 172.16.153.1
default via 192.168.100.1 dev eth1
broadcast 192.168.100.0 dev eth1 table local proto kernel scope link src 192.168.100.110
broadcast 172.16.146.0 dev vmnet8 table local proto kernel scope link src 172.16.146.1
local 172.16.146.1 dev vmnet8 table local proto kernel scope host src 172.16.146.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 172.16.153.255 dev vmnet1 table local proto kernel scope link src 172.16.153.1
broadcast 192.168.100.255 dev eth1 table local proto kernel scope link src 192.168.100.110
broadcast 172.16.146.255 dev vmnet8 table local proto kernel scope link src 172.16.146.1
broadcast 172.16.153.0 dev vmnet1 table local proto kernel scope link src 172.16.153.1
local 172.16.153.1 dev vmnet1 table local proto kernel scope host src 172.16.153.1
local 192.168.100.110 dev eth1 table local proto kernel scope host src 192.168.100.110
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local ::1 via :: dev lo proto none metric 0 mtu 16436 rtt 1ms rttvar 1ms cwnd 2 advmss 16376 hoplimit 64
local fe80::250:56ff:fec0:1 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64
local fe80::250:56ff:fec0:8 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64
fe80::/64 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64
fe80::/64 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64
ff00::/8 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1
ff00::/8 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1
unreachable default dev lo proto none metric -1 error -101 hoplimit 25

--------------------------------------------------------
ip route list table cache
--------------------------------------------------------
broadcast 192.168.100.255 from 192.168.100.37 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.112 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.118 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
192.168.100.118 from 192.168.100.110 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.44 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.34 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 192.168.100.110 from 192.168.100.114 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.19 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.10 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 from 127.0.0.1 dev lo
    cache <local>  mtu 16436 rtt 5ms rttvar 8ms cwnd 3 advmss 16396 hoplimit 64
broadcast 192.168.100.255 dev eth1  src 192.168.100.110
    cache <local,brd>  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.114 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.61 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 dev lo  src 127.0.0.1
    cache <local>  mtu 16436 advmss 16396 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.118 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.114 from 192.168.100.110 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.112 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.10 dev eth1  src 192.168.100.110
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.60 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.36 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
local 127.0.0.1 dev lo  src 127.0.0.1
    cache <local>  mtu 16436 advmss 16396 hoplimit 64
local 192.168.100.110 from 192.168.100.10 dev lo  src 192.168.100.110
    cache <local,src-direct>  iif eth1
local 192.168.100.110 from 192.168.8.55 dev lo  src 192.168.100.110
    cache <local>  iif eth1
broadcast 192.168.100.255 from 192.168.100.16 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.8.55 from 192.168.100.110 tos 0x10 via 192.168.100.1 dev eth1
    cache  mtu 1500 rtt 105ms rttvar 105ms cwnd 3 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.91 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.8.55 from 192.168.100.110 via 192.168.100.1 dev eth1
    cache  mtu 1500 advmss 1460 hoplimit 64
broadcast 192.168.100.255 from 192.168.100.140 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
broadcast 192.168.100.255 from 192.168.100.38 dev lo  src 192.168.100.110
    cache <local,brd,src-direct>  iif eth1
192.168.100.10 from 192.168.100.110 dev eth1
    cache  mtu 1500 rtt 26ms rttvar 42ms cwnd 2 advmss 1460 hoplimit 64
local 192.168.100.110 from 192.168.110.6 dev lo  src 192.168.100.110
    cache <local>  iif eth1
192.168.100.112 from 192.168.100.110 dev eth1
    cache  mtu 1500 rtt 15ms rttvar 28ms cwnd 3 advmss 1460 hoplimit 64

Open in new window

0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 33628025
It appears that there is a script that is monitoring the interface and is issuing something like:

     ip route flush dev eth0

This will get rid of any/all route table entries that use the dev eth0.
0
 

Author Comment

by:zeldi
ID: 33628185
Just checking it right now..., I've just found process NetworkManager, which might be doing the dirty job...
0
 

Expert Comment

by:memoo_
ID: 33629069
zeldi ,  i need to check this file /etc/sysconfig/networks-scripts/ifcfg-eth0 and ifcfg-eth1 too, i think that i can fix the error



 
0
 
LVL 12

Expert Comment

by:mccracky
ID: 33631500
If it's a server, get rid of network manager and use the network-scripts.  fix the two route-ethX files.  The "file exists" error, if I remember correctly, doesn't really matter and things should be fine.  If not you can manually set the routing in rc.local.
0
 

Author Comment

by:zeldi
ID: 33657816
yup, it was a NetworkManager, now it is removed from init.

@giltjr: I wasn't really thinking, that any daemon would be doing such a stupid thing, but after your hint did a thorough check of all the processes and there it was.
0

Featured Post

Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Redhat upgrade 1 48
Port to open for RDP connection to VM in DMZ ? 5 65
Advice on ESXi 5.1 Health / Storage 1 43
SMTP log file for IMSVA 5 29
Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
PRTG Network Monitor lets you monitor your bandwidth usage, so you know who is using up your bandwidth, and what they're using it for.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

756 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question