joaotelles
asked on
RHEL6 + dockers - No route to host
Hi,
I have an env with RHEL6 and dockers installed. And I have the following network interfaces:
For the dockers I have the following networks:
# docker network ls
NETWORK ID NAME DRIVER
c06f507f0280 bridge bridge
24d1020d4fc5 none null
371ea25cf335 host host
And inspecting the bridge one: (which matches with the docker0 network interface)
For the other interfaces one is for external connections with the IP address of the box where everything is running and the other in just an internal IP.
===
With this setup I tried to start an application that needs to fetch some info from the http address below: ("http://10.216.209.75:14180/xxxx/default")
But as you can see Im getting no route to the host, even though Im starting the application and this IP is the IP of the box where everything is running.
This is the TCPdump of the time I started the appl and the http GET went in place:
As you can see the HTTP GET goes from the docker0 network to the eno16780032 and thats is what is not working....
PS:If I try 'curl --interface docker0 http://10.216.209.75:14180/xxxx/default" and get the proper return because it does not switch networks...
On the tcpdump I get this:
If I try:
But if I try:
=====
The obvious thing here would be a routing table issue...
So if I issue the command 'route' it hangs: (not showing any data).
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
But if I try I get the routing table:
# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.216.209.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 10.216.209.1 0.0.0.0 UG 0 0 0 eth0
====
So my questions are:
1- Why does the route doesnt display anything and the netstat -nr does?
2- How/What should I add to the routing table (if this is the issue), to solve this issue?
Tks,
Joao
I have an env with RHEL6 and dockers installed. And I have the following network interfaces:
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
eno16780032: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.216.209.75 netmask 255.255.255.0 broadcast 10.216.209.255
eno33559296: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.75 netmask 255.255.255.0 broadcast 192.168.1.255
For the dockers I have the following networks:
# docker network ls
NETWORK ID NAME DRIVER
c06f507f0280 bridge bridge
24d1020d4fc5 none null
371ea25cf335 host host
And inspecting the bridge one: (which matches with the docker0 network interface)
"Name": "bridge",
"Id": "c06f507f0280c900863f50270de22215727a2f6a7b9bb75f3c54c6275db0be6a",
"Scope": "local",
"Driver": "bridge",
"IPAM":
"Driver": "default",
"Config":
{
"Subnet": "172.17.0.1/16",
"Gateway": "172.17.0.1"
}
For the other interfaces one is for external connections with the IP address of the box where everything is running and the other in just an internal IP.
===
With this setup I tried to start an application that needs to fetch some info from the http address below: ("http://10.216.209.75:14180/xxxx/default")
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://10.216.209.75:14180/xxxx/default": No route to host; nested exception is java.net.NoRouteToHostException: No route to host
But as you can see Im getting no route to the host, even though Im starting the application and this IP is the IP of the box where everything is running.
This is the TCPdump of the time I started the appl and the http GET went in place:
# tcpdump -nn -r /tmp/dump
reading from file /tmp/dump, link-type LINUX_SLL (Linux cooked)
15:07:15.916005 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858209917 ecr 0,nop,wscale 7], length 0
15:07:16.918374 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858210920 ecr 0,nop,wscale 7], length 0
15:07:16.918374 IP 172.17.0.5.47100 > 10.216.209.75.14180: Flags [S], seq 1354517853, win 14600, options [mss 1460,sackOK,TS val 1858210920 ecr 0,nop,wscale 7], length 0
As you can see the HTTP GET goes from the docker0 network to the eno16780032 and thats is what is not working....
PS:If I try 'curl --interface docker0 http://10.216.209.75:14180/xxxx/default" and get the proper return because it does not switch networks...
On the tcpdump I get this:
13:24:52.920075 IP 172.17.0.1.59548 > 172.17.0.4.14180: Flags [S], seq 2169422352, win 14600, options [mss 1460,sackOK,TS val 2197666921 ecr 0,nop,wscale 7], length 0
13:24:52.920106 IP 172.17.0.4.14180 > 172.17.0.1.59548: Flags [S.], seq 3110907639, ack 2169422353, win 14480, options [mss 1460,sackOK,TS val 2197666921 ecr 2197666921,nop,wscale 7], length 0
If I try:
# ping -I docker0 172.17.0.1 (same network)
PING 172.17.0.1 (172.17.0.1) from 172.17.0.1 docker0: 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.081 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.072 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=0.091 ms
But if I try:
# ping -I docker0 192.168.1.75 (dif. networks)
PING 192.168.1.75 (192.168.1.75) from 172.17.0.1 docker0: 56(84) bytes of data.
From 172.17.0.1 icmp_seq=1 Destination Host Unreachable
From 172.17.0.1 icmp_seq=2 Destination Host Unreachable
From 172.17.0.1 icmp_seq=3 Destination Host Unreachable
From 172.17.0.1 icmp_seq=4 Destination Host Unreachable
=====
The obvious thing here would be a routing table issue...
So if I issue the command 'route' it hangs: (not showing any data).
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
But if I try I get the routing table:
# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.216.209.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 10.216.209.1 0.0.0.0 UG 0 0 0 eth0
====
So my questions are:
1- Why does the route doesnt display anything and the netstat -nr does?
2- How/What should I add to the routing table (if this is the issue), to solve this issue?
Tks,
Joao
"1- Why does the route doesnt display anything and the netstat -nr does?"
What happens if you issue the command "route -n"?
What happens if you issue the command "route -n"?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Yes, I know that "-n" says to show numerical. I am assuming that the "route" command is hanging because it is attempting to do reverse lookup on the IP address and is having a problem for some reason. I noticed when he issued the netstat command he uses "-n" and it worked without any issues.
ASKER
Installed a missing package and now the route is working. -> net-tools
Or the ip route list:
==============
So, what should I add?
Should I add the route, seems to be missing...
This is my iptables (I havent configured anything there...)
=====
For the vi /etc/sysconfig/network, should I add the route add line?
Sry for so many questions but Im a little confused on how all these place interact and/or interfere with each other...
Tks,
Joao
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.216.209.1 0.0.0.0 UG 100 0 0 eno16780032
10.216.209.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16780032
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eno33559296
Or the ip route list:
# ip route list
default via 10.216.209.1 dev eno16780032 proto static metric 100
10.216.209.0/24 dev eno16780032 proto kernel scope link src 10.216.209.75 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.1.0/24 dev eno33559296 proto kernel scope link src 192.168.1.75 metric 100
# cat /etc/sysconfig/network
# Created by anaconda
NETWORKING_IPV6=no
IPV6INIT=no
NETWORKING=yes
==============
So, what should I add?
Should I add the route, seems to be missing...
route add default gw 172.16.0.1 docker0
This is my iptables (I havent configured anything there...)
]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INPUT_direct all -- 0.0.0.0/0 0.0.0.0/0
INPUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
INPUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_direct all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_IN_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_IN_ZONES all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_OUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
OUTPUT_direct all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:11210
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8093
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8092
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8091
ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:6379
ACCEPT tcp -- 0.0.0.0/0 172.17.0.4 tcp dpt:14180
Chain FORWARD_IN_ZONES (1 references)
target prot opt source destination
FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_OUT_ZONES (1 references)
target prot opt source destination
FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_direct (1 references)
target prot opt source destination
Chain FWDI_public (3 references)
target prot opt source destination
FWDI_public_log all -- 0.0.0.0/0 0.0.0.0/0
FWDI_public_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDI_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain FWDI_public_allow (1 references)
target prot opt source destination
Chain FWDI_public_deny (1 references)
target prot opt source destination
Chain FWDI_public_log (1 references)
target prot opt source destination
Chain FWDO_public (3 references)
target prot opt source destination
FWDO_public_log all -- 0.0.0.0/0 0.0.0.0/0
FWDO_public_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDO_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain FWDO_public_allow (1 references)
target prot opt source destination
Chain FWDO_public_deny (1 references)
target prot opt source destination
Chain FWDO_public_log (1 references)
target prot opt source destination
Chain INPUT_ZONES (1 references)
target prot opt source destination
IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain INPUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain INPUT_direct (1 references)
target prot opt source destination
Chain IN_public (3 references)
target prot opt source destination
IN_public_log all -- 0.0.0.0/0 0.0.0.0/0
IN_public_deny all -- 0.0.0.0/0 0.0.0.0/0
IN_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW
Chain IN_public_deny (1 references)
target prot opt source destination
Chain IN_public_log (1 references)
target prot opt source destination
Chain OUTPUT_direct (1 references)
target prot opt source destination
=====
For the vi /etc/sysconfig/network, should I add the route add line?
Sry for so many questions but Im a little confused on how all these place interact and/or interfere with each other...
Tks,
Joao
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
But still does not work...
I havent added this one: maybe I should and try?
I havent added this one: maybe I should and try?
route add default gw 172.16.0.1 docker0
Open in new window
orOpen in new window
If you still dont see anything try and manually add the route:
Open in new window
You might need to specify the routes between your interfaces. I would also look at your iptables config and make sure there is nothing there to cause issues.
Also have a look at:
Open in new window
To make sure that your default route is permanent.