Solved

Network routing doesn't come up: RTNETLINK: Network is unreachable

Posted on 2010-09-08
12
5,606 Views
Last Modified: 2012-08-13
Dear experts,

I have a CentOS 4.6 with two NICs:
eth0      Link encap:Ethernet  HWaddr 00:50:DA:B6:EF:E6
          inet addr:192.168.140.5  Bcast:192.168.140.255  Mask:255.255.255.0
          inet6 addr: fe80::250:daff:feb6:efe6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1659 errors:0 dropped:0 overruns:0 frame:0
          TX packets:161 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:124890 (121.9 KiB)  TX bytes:16949 (16.5 KiB)
          Interrupt:169 Base address:0x4400

eth1      Link encap:Ethernet  HWaddr 00:14:5E:1F:38:D1
          inet addr:192.168.100.110  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:85647 errors:0 dropped:0 overruns:0 frame:0
          TX packets:120177 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15686518 (14.9 MiB)  TX bytes:113475875 (108.2 MiB)
          Interrupt:169

Open in new window


But the routing table doesn't show eth0 network:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   *               255.255.255.0   U     0      0        0 eth1
172.16.146.0    *               255.255.255.0   U     0      0        0 vmnet8
172.16.153.0    *               255.255.255.0   U     0      0        0 vmnet1

Open in new window


Executing route add -net 192.168.140.0 netmask 255.255.255.0 eth0 doesn't return any errors, route stays on for half a second and then dissapears (again without any error).

The only error I get is, when doing ifup eth0:
RTNETLINK answers: Network is unreachable

I also received a report (it is a remote production server), that if take down eth1, eth0 network becomes available.

What is the answer, to get the second network available?
0
Comment
Question by:zeldi
12 Comments
 

Author Comment

by:zeldi
Comment Utility
OK, I thing which I have found out: in /etc/sysconfig/network-scripts I had two files route-eth0 (referencing eth1 network) and route-eth1 (referencing eth0 network), which should be inversed. If i reverse the filenames I get the error
RTNETLINK answers: File exists
0
 

Author Comment

by:zeldi
Comment Utility
Another one: if I remove files route-eth0 and route-eth1 the ifup eth0 doesn't return any error, I get a route to network:
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0 (it's 169!!!)
for a couple of seconds, and then it is removed
0
 
LVL 6

Expert Comment

by:Jelcin
Comment Utility
Hi, not sure if this helps but on my Ubuntu ifup and ifdown does not work. Try "ifconfig eth0 up".
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Do you see any thing in /var/messages?

Do you see anything in the output of dmesg related to eth0.

It sounds as if eth0 is having physical problems and although it shows up, it really is not.
0
 

Author Comment

by:zeldi
Comment Utility
Hi,

as mentioned before, I get no messages either in dmesg nor in /var/log/messages which would show any errors.

But the eth0 stays up (as can be seen from ifconfig), just the route gets dropped, while the route is up, the ping to remote network works, and then the route is dropped and ping stops working.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Without any overriding parameters if you do a ifup eth0 (or any other interface) it will attempt to do dhcp.  If there are no dhcp servers then it will use the "auto config" IP address of 169.254.0.0/16.

Try issuing:

     ip route list table all

or

     ip route list table cache

0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:zeldi
Comment Utility
As you have noticed in my first post, IPs are configured on the NICs (hence ifconfig shows the configured IPs).
Please find below attached the outputs from the ip route command

--------------------------------------------------------

ip route list table all

--------------------------------------------------------

192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.110

172.16.146.0/24 dev vmnet8 proto kernel scope link src 172.16.146.1

172.16.153.0/24 dev vmnet1 proto kernel scope link src 172.16.153.1

default via 192.168.100.1 dev eth1

broadcast 192.168.100.0 dev eth1 table local proto kernel scope link src 192.168.100.110

broadcast 172.16.146.0 dev vmnet8 table local proto kernel scope link src 172.16.146.1

local 172.16.146.1 dev vmnet8 table local proto kernel scope host src 172.16.146.1

broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1

broadcast 172.16.153.255 dev vmnet1 table local proto kernel scope link src 172.16.153.1

broadcast 192.168.100.255 dev eth1 table local proto kernel scope link src 192.168.100.110

broadcast 172.16.146.255 dev vmnet8 table local proto kernel scope link src 172.16.146.1

broadcast 172.16.153.0 dev vmnet1 table local proto kernel scope link src 172.16.153.1

local 172.16.153.1 dev vmnet1 table local proto kernel scope host src 172.16.153.1

local 192.168.100.110 dev eth1 table local proto kernel scope host src 192.168.100.110

broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1

local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1

local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1

local ::1 via :: dev lo proto none metric 0 mtu 16436 rtt 1ms rttvar 1ms cwnd 2 advmss 16376 hoplimit 64

local fe80::250:56ff:fec0:1 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64

local fe80::250:56ff:fec0:8 via :: dev lo proto none metric 0 mtu 16436 advmss 16376 hoplimit 64

fe80::/64 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64

fe80::/64 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 64

ff00::/8 dev vmnet8 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1

ff00::/8 dev vmnet1 metric 256 expires 2132910sec mtu 1500 advmss 1440 hoplimit 1

unreachable default dev lo proto none metric -1 error -101 hoplimit 25



--------------------------------------------------------

ip route list table cache

--------------------------------------------------------

broadcast 192.168.100.255 from 192.168.100.37 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

local 192.168.100.110 from 192.168.100.112 dev lo  src 192.168.100.110

    cache <local,src-direct>  iif eth1

local 192.168.100.110 from 192.168.100.118 dev lo  src 192.168.100.110

    cache <local,src-direct>  iif eth1

192.168.100.118 from 192.168.100.110 dev eth1

    cache  mtu 1500 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.44 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.34 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

local 192.168.100.110 from 192.168.100.114 dev lo  src 192.168.100.110

    cache <local,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.19 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.10 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

local 127.0.0.1 from 127.0.0.1 dev lo

    cache <local>  mtu 16436 rtt 5ms rttvar 8ms cwnd 3 advmss 16396 hoplimit 64

broadcast 192.168.100.255 dev eth1  src 192.168.100.110

    cache <local,brd>  mtu 1500 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.114 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.61 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

local 127.0.0.1 dev lo  src 127.0.0.1

    cache <local>  mtu 16436 advmss 16396 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.118 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

192.168.100.114 from 192.168.100.110 dev eth1

    cache  mtu 1500 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.112 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

192.168.100.10 dev eth1  src 192.168.100.110

    cache  mtu 1500 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.60 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.36 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

local 127.0.0.1 dev lo  src 127.0.0.1

    cache <local>  mtu 16436 advmss 16396 hoplimit 64

local 192.168.100.110 from 192.168.100.10 dev lo  src 192.168.100.110

    cache <local,src-direct>  iif eth1

local 192.168.100.110 from 192.168.8.55 dev lo  src 192.168.100.110

    cache <local>  iif eth1

broadcast 192.168.100.255 from 192.168.100.16 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

192.168.8.55 from 192.168.100.110 tos 0x10 via 192.168.100.1 dev eth1

    cache  mtu 1500 rtt 105ms rttvar 105ms cwnd 3 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.91 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

192.168.8.55 from 192.168.100.110 via 192.168.100.1 dev eth1

    cache  mtu 1500 advmss 1460 hoplimit 64

broadcast 192.168.100.255 from 192.168.100.140 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

broadcast 192.168.100.255 from 192.168.100.38 dev lo  src 192.168.100.110

    cache <local,brd,src-direct>  iif eth1

192.168.100.10 from 192.168.100.110 dev eth1

    cache  mtu 1500 rtt 26ms rttvar 42ms cwnd 2 advmss 1460 hoplimit 64

local 192.168.100.110 from 192.168.110.6 dev lo  src 192.168.100.110

    cache <local>  iif eth1

192.168.100.112 from 192.168.100.110 dev eth1

    cache  mtu 1500 rtt 15ms rttvar 28ms cwnd 3 advmss 1460 hoplimit 64

Open in new window

0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
Comment Utility
It appears that there is a script that is monitoring the interface and is issuing something like:

     ip route flush dev eth0

This will get rid of any/all route table entries that use the dev eth0.
0
 

Author Comment

by:zeldi
Comment Utility
Just checking it right now..., I've just found process NetworkManager, which might be doing the dirty job...
0
 

Expert Comment

by:memoo_
Comment Utility
zeldi ,  i need to check this file /etc/sysconfig/networks-scripts/ifcfg-eth0 and ifcfg-eth1 too, i think that i can fix the error



 
0
 
LVL 12

Expert Comment

by:mccracky
Comment Utility
If it's a server, get rid of network manager and use the network-scripts.  fix the two route-ethX files.  The "file exists" error, if I remember correctly, doesn't really matter and things should be fine.  If not you can manually set the routing in rc.local.
0
 

Author Comment

by:zeldi
Comment Utility
yup, it was a NetworkManager, now it is removed from init.

@giltjr: I wasn't really thinking, that any daemon would be doing such a stupid thing, but after your hint did a thorough check of all the processes and there it was.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
How to scan rdp  ''only'' open port 3333? 5 64
sync conflicts 1 20
Trying to make SNMP connection work 7 40
linux ssh 4 26
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now