Link to home
Start Free TrialLog in
Avatar of joaotelles
joaotellesFlag for United States of America

asked on

RHEL6 + dockers - No route to host

Hi,
I have an env with RHEL6 and dockers installed. And I have the following network interfaces:

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0

eno16780032: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.216.209.75  netmask 255.255.255.0  broadcast 10.216.209.255

eno33559296: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.75  netmask 255.255.255.0  broadcast 192.168.1.255

Open in new window



For the dockers I have the following networks:

# docker network ls
NETWORK ID          NAME                DRIVER
c06f507f0280        bridge              bridge              
24d1020d4fc5        none                null                
371ea25cf335        host                host    

And inspecting the bridge one: (which matches with the docker0 network interface)

        "Name": "bridge",
        "Id": "c06f507f0280c900863f50270de22215727a2f6a7b9bb75f3c54c6275db0be6a",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": 
            "Driver": "default",
            "Config": 
                {
                    "Subnet": "172.17.0.1/16",
                    "Gateway": "172.17.0.1"
                }

Open in new window


For the other interfaces one is for external connections with the IP address of the box where everything is running and the other in just an internal IP.

===

With this setup I tried to start an application that needs to fetch some info from the http address below: ("http://10.216.209.75:14180/xxxx/default")

Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://10.216.209.75:14180/xxxx/default": No route to host; nested exception is java.net.NoRouteToHostException: No route to host 

Open in new window


But as you can see Im getting no route to the host, even though Im starting the application and this IP is the IP of the box where everything is running.

This is the TCPdump of the time I started the appl and the http GET went in place:

#  tcpdump -nn -r /tmp/dump
reading from file /tmp/dump, link-type LINUX_SLL (Linux cooked)
15:07:15.916005 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858209917 ecr 0,nop,wscale 7], length 0
15:07:16.918374 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858210920 ecr 0,nop,wscale 7], length 0
15:07:16.918374 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858210920 ecr 0,nop,wscale 7], length 0

Open in new window



As you can see the HTTP GET goes from the docker0 network to the eno16780032 and thats is what is not working....

PS:If I try 'curl --interface docker0 http://10.216.209.75:14180/xxxx/default" and get the proper return because it does not switch networks...

On the tcpdump I get this:

13:24:52.920075 IP 172.17.0.1.59548 > 172.17.0.4.14180: Flags [S], seq 2169422352, win 14600, options [mss 1460,sackOK,TS val 2197666921 ecr 0,nop,wscale 7], length 0
13:24:52.920106 IP 172.17.0.4.14180 > 172.17.0.1.59548: Flags [S.], seq 3110907639, ack 2169422353, win 14480, options [mss 1460,sackOK,TS val 2197666921 ecr 2197666921,nop,wscale 7], length 0

Open in new window


If I try:

# ping -I docker0 172.17.0.1 (same network)
PING 172.17.0.1 (172.17.0.1) from 172.17.0.1 docker0: 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.081 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.072 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=0.091 ms

Open in new window


But if I try:
# ping -I docker0 192.168.1.75 (dif. networks)
PING 192.168.1.75 (192.168.1.75) from 172.17.0.1 docker0: 56(84) bytes of data.
From 172.17.0.1 icmp_seq=1 Destination Host Unreachable
From 172.17.0.1 icmp_seq=2 Destination Host Unreachable
From 172.17.0.1 icmp_seq=3 Destination Host Unreachable
From 172.17.0.1 icmp_seq=4 Destination Host Unreachable

Open in new window


=====

The obvious thing here would be a routing table issue...

So if I issue the command 'route' it hangs: (not showing any data).

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

But if I try I get the routing table:

# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.216.209.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
0.0.0.0         10.216.209.1    0.0.0.0         UG        0 0          0 eth0

====

So my questions are:

1- Why does the route doesnt display anything and the netstat -nr does?
2- How/What should I add to the routing table (if this is the issue), to solve this issue?

Tks,
Joao
Avatar of Joshua Hopkins
Joshua Hopkins
Flag of United States of America image

Try the following to see the route table:
route -n

Open in new window

or
ip route list

Open in new window


If you still dont see anything try and manually add the route:

route add default gw 172.16.0.1 docker0

Open in new window


You might need to specify the routes between your interfaces.  I would also look at your iptables config and make sure there is nothing there to cause issues.

Also have a look at:

vi /etc/sysconfig/network

Open in new window


To make sure that your default route is permanent.
Avatar of giltjr
"1- Why does the route doesnt display anything and the netstat -nr does?"

What happens if you issue the command "route -n"?
ASKER CERTIFIED SOLUTION
Avatar of Joshua Hopkins
Joshua Hopkins
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Yes, I know that "-n" says to show numerical.  I am assuming that the "route" command is hanging because it is attempting to do reverse lookup on the IP address and is having a problem for some reason.  I noticed when he issued the netstat command he uses "-n"  and it worked without any issues.
Avatar of joaotelles

ASKER

Installed a missing package and now the route is working. -> net-tools

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.216.209.1    0.0.0.0         UG    100    0        0 eno16780032
10.216.209.0    0.0.0.0         255.255.255.0   U     100    0        0 eno16780032
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 eno33559296

Open in new window


Or the ip route list:

# ip route list
default via 10.216.209.1 dev eno16780032  proto static  metric 100 
10.216.209.0/24 dev eno16780032  proto kernel  scope link  src 10.216.209.75  metric 100 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.1.0/24 dev eno33559296  proto kernel  scope link  src 192.168.1.75  metric 100 

Open in new window


# cat /etc/sysconfig/network
# Created by anaconda
NETWORKING_IPV6=no
IPV6INIT=no
NETWORKING=yes

Open in new window


==============

So, what should I add?

Should I add the route, seems to be missing...

route add default gw 172.16.0.1 docker0

Open in new window


This is my iptables (I havent configured anything there...)

]# iptables -L -n 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:11210
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:8093
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:8092
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:8091
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.3           tcp dpt:6379
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.4           tcp dpt:14180

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (3 references)
target     prot opt source               destination         
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (3 references)
target     prot opt source               destination         
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (3 references)
target     prot opt source               destination         
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination         

Open in new window


=====

For the vi /etc/sysconfig/network, should I add the route add line?

Sry for so many questions but Im a little confused on how all these place interact and/or interfere with each other...

Tks,
Joao
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
But still does not work...

I havent added this one: maybe I should and try?

route add default gw 172.16.0.1 docker0

Open in new window