asked on
Two External networks, inbound DNAT problems and route Cache
Afternoon
For a good number of years I have been using a script taken from the nano.txt howto with Julian's patches applied. Just recently I started to see a problem on a few in place installations of a multiple internet connection firewall and a hosted mail server behind. The setups seem rather common but here's the obligatory diagram:
---- 10.0.0.2 10.0.0.5
|Lan Servers
---Eth0 10.0.0.0/24
|
|
Firewall
|
---eth1 192.168.0.2 -> GW 192.168.0.1
|
|
---eth2 172.16.0.2 -> GW 172,16,0,1
Servers at 10.0.0.0/24 have round robin DNS setup. The problem creeps up most often with SMTP so lets use that an an example. FTP, TELNET, RSYNC, HTTP all have the same issue.
Lets say I'm hosting a mail server at 10.0.0.5. The client has 3 workstations on their own network somewhere in the internet.
The 1st workstation resolves an ip to IFACE1 and enjoys a successful connection. Seconds later Workstation 2 resolves the other interface eth2, no connection at all. That workstation gets a timeout error
and the client gets unhappy. Wait a few seconds, it connects fine but cannot connect on eth1.
I manually test for the problem by doing the following:
1. telnet 192.168.0.2 25 (from a client machine on another network)
I can get my SMTP session.
2. telnet 172.16.0.2 25 from the same machine. Connection times out.
I can reverse that order with the same result.
3. on the firewall ip route flush cache
4. telnet 172.16.0.2 25 Connection accepts and the session works fine
5. telnet 182.168.0.2 25 Connection times out
Eventually the cache expires (about 30 seconds) and either interface will pick denying the other.
I have been through a good deal of blog posts and what not about marking packets inbound on a given ip and then adding ip rule add fwmark 1 table 201 but that doesn't seem to work out for me. I assume I'm doing it wrong or not placing the various additions to the right places in the below script template.
First things first, the script I am using:
some of the interfaces eth1 or eth2 might have extra virtual IPs associated to them. So far that doesn't seem to make any difference in the problem, same thing if I have other IPs or don't so I'll include them in the script.
Thoughts on what I can do to try and fix this?
$IFI =eth0
$IPI = 10.0.0.1
$NWI = 10.0.0.0
$NMI = 24
$IFE1 =eth1
$IPE1 =192.168.0.2
$NME1 =29
$NWE1 = 192.168.0.0
$GWE1 = 192.168.0.1
$IFE2 = eth2
$IPE2 = 172.16.0.2
$NME2 = 29
$NEW2 = 172.16.0.0
$GWE2 = 172.16.0.1
echo "Clear old rules"
#Clear out old rules
ip rule show | grep -Ev '^(0|32766|32767):|iif lo' \
| while read PRIO NATRULE; do
ip rule del prio ${PRIO%%:*} $( echo $NATRULE | sed 's|all|0/0|' )
done
echo "Setting Loopback route"
ip link set lo up
ip addr add 127.0.0.1/8 brd + dev lo
echo "Setting ETH0/Internal route"
ip link set $IFI up
ip addr add $IPI/$NMI brd + dev $IFI
ip rule add prio 50 table main
ip route del default table main
echo "Setting External Interfaces without gateway or Route"
ip link set $IFE1 up
ip addr flush dev $IFE1
ip addr add $IPE1/$NME1 brd $BRD1 dev $IFE1
ip addr add 192.168.0.3/29 brd $BRD1 dev $IFE1 label eth1:2
ip addr add 192.168.0.4/29 brd $BRD1 dev $IFE1 label eth1:5
ip link set $IFE2 up
ip addr flush dev $IFE2
ip addr add $IPE2/$NME2 brd $BRD2 dev $IFE2
ip addr add 172.16.0.2/29 brd $BRD2 dev $IFE2 label eth2:2
echo "Set Established traffic to use same IF"
ip rule add prio 201 from $NWE1/$NME1 table 201
ip route add default via $GWE1 dev $IFE1 src 192.168.0.3 proto static table 201
ip route add default via $GWE1 dev $IFE1 src 192.168.0.4 proto static table 201
ip route add default via $GWE1 dev $IFE1 src $IPE1 proto static table 201
ip route append prohibit default table 201 metric 1 proto static
ip rule add prio 202 from $NWE2/$NME2 table 202
ip route add default via $GWE2 dev $IFE2 src 172.16.0.3 proto static table 202
ip route add default via $GWE2 dev $IFE2 src $IPE2 proto static table 202
ip route add default via $GWE2 dev $IFE2 table 202
ip route append prohibit default table 202 metric 1 proto static
echo "Default Multipath Route and setting weights!"
ip rule add prio 222 table 222
#ETH1 is a lot faster
ip route add default table 222 proto static \
nexthop via $GWE1 dev $IFE1 weight 3 \
nexthop via $GWE2 dev $IFE2 weight 1
Next up is the iptables part of it all with source natting.
echo "Setting the Chains on Filter and Nat"
$IPTABLES -t filter -N keep_state
$IPTABLES -t filter -A keep_state -m state --state RELATED,ESTABLISHED \
-j ACCEPT
$IPTABLES -t filter -A keep_state -j RETURN
$IPTABLES -t nat -N keep_state
$IPTABLES -t nat -A keep_state -m state --state RELATED,ESTABLISHED \
-j ACCEPT
$IPTABLES -t nat -A keep_state -j RETURN
echo "Setting Jumps for PRE POST OUT IN FORWARD OUT for NAT and FILTER"
$IPTABLES -t nat -A PREROUTING -j keep_state
$IPTABLES -t nat -A POSTROUTING -j keep_state
$IPTABLES -t nat -A OUTPUT -j keep_state
$IPTABLES -t filter -A INPUT -j keep_state
$IPTABLES -t filter -A FORWARD -j keep_state
$IPTABLES -t filter -A OUTPUT -j keep_state
echo "IPSEC Policy"
$IPTABLES -t nat -A POSTROUTING -m policy --dir out \
--pol ipsec --mode tunnel -j ACCEPT
$IPTABLES -t nat -A POSTROUTING -m policy --dir in \
--pol ipsec --mode tunnel -j ACCEPT
echo " SNAT for servers on virt IPs"
$IPTABLES -t nat -A POSTROUTING -o $IFE1 -s 10.0.0.5 \
-j SNAT --to 192.168.0.3
$IPTABLES -t nat -A POSTROUTING -o $IFE2 -s 10.0.0.5 \
-j SNAT --to 172.168.0.3
echo " SNAT-Default"
$IPTABLES -t nat -A POSTROUTING -o $IFE1 -s $NWI/$NMI -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o $IFE2 -s $NWI/$NMI -j SNAT --to $IPE2
echo "Protocol forwards"
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE1 -d $IPE1 --dport 25 \
-j DNAT --to 10.0.0.2
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE1 -d 192.168.0.3 --dport 25 \ -j DNAT --to 10.0.0.5
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE2 -d $IPE2 --dport 25 \
-j DNAT --to 10.0.0.2
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE2 -d 172.16.0.3 --dport 25 \ -j DNAT --to 10.0.0.5
echo "Dropping Everything else in INPUT chain"
$IPTABLES -A INPUT -i $IFE1 -j DROP
$IPTABLES -A INPUT -i $IFE2 -j DROP
For a good number of years I have been using a script taken from the nano.txt howto with Julian's patches applied. Just recently I started to see a problem on a few in place installations of a multiple internet connection firewall and a hosted mail server behind. The setups seem rather common but here's the obligatory diagram:
---- 10.0.0.2 10.0.0.5
|Lan Servers
---Eth0 10.0.0.0/24
|
|
Firewall
|
---eth1 192.168.0.2 -> GW 192.168.0.1
|
|
---eth2 172.16.0.2 -> GW 172,16,0,1
Servers at 10.0.0.0/24 have round robin DNS setup. The problem creeps up most often with SMTP so lets use that an an example. FTP, TELNET, RSYNC, HTTP all have the same issue.
Lets say I'm hosting a mail server at 10.0.0.5. The client has 3 workstations on their own network somewhere in the internet.
The 1st workstation resolves an ip to IFACE1 and enjoys a successful connection. Seconds later Workstation 2 resolves the other interface eth2, no connection at all. That workstation gets a timeout error
and the client gets unhappy. Wait a few seconds, it connects fine but cannot connect on eth1.
I manually test for the problem by doing the following:
1. telnet 192.168.0.2 25 (from a client machine on another network)
I can get my SMTP session.
2. telnet 172.16.0.2 25 from the same machine. Connection times out.
I can reverse that order with the same result.
3. on the firewall ip route flush cache
4. telnet 172.16.0.2 25 Connection accepts and the session works fine
5. telnet 182.168.0.2 25 Connection times out
Eventually the cache expires (about 30 seconds) and either interface will pick denying the other.
I have been through a good deal of blog posts and what not about marking packets inbound on a given ip and then adding ip rule add fwmark 1 table 201 but that doesn't seem to work out for me. I assume I'm doing it wrong or not placing the various additions to the right places in the below script template.
First things first, the script I am using:
some of the interfaces eth1 or eth2 might have extra virtual IPs associated to them. So far that doesn't seem to make any difference in the problem, same thing if I have other IPs or don't so I'll include them in the script.
Thoughts on what I can do to try and fix this?
$IFI =eth0
$IPI = 10.0.0.1
$NWI = 10.0.0.0
$NMI = 24
$IFE1 =eth1
$IPE1 =192.168.0.2
$NME1 =29
$NWE1 = 192.168.0.0
$GWE1 = 192.168.0.1
$IFE2 = eth2
$IPE2 = 172.16.0.2
$NME2 = 29
$NEW2 = 172.16.0.0
$GWE2 = 172.16.0.1
echo "Clear old rules"
#Clear out old rules
ip rule show | grep -Ev '^(0|32766|32767):|iif lo' \
| while read PRIO NATRULE; do
ip rule del prio ${PRIO%%:*} $( echo $NATRULE | sed 's|all|0/0|' )
done
echo "Setting Loopback route"
ip link set lo up
ip addr add 127.0.0.1/8 brd + dev lo
echo "Setting ETH0/Internal route"
ip link set $IFI up
ip addr add $IPI/$NMI brd + dev $IFI
ip rule add prio 50 table main
ip route del default table main
echo "Setting External Interfaces without gateway or Route"
ip link set $IFE1 up
ip addr flush dev $IFE1
ip addr add $IPE1/$NME1 brd $BRD1 dev $IFE1
ip addr add 192.168.0.3/29 brd $BRD1 dev $IFE1 label eth1:2
ip addr add 192.168.0.4/29 brd $BRD1 dev $IFE1 label eth1:5
ip link set $IFE2 up
ip addr flush dev $IFE2
ip addr add $IPE2/$NME2 brd $BRD2 dev $IFE2
ip addr add 172.16.0.2/29 brd $BRD2 dev $IFE2 label eth2:2
echo "Set Established traffic to use same IF"
ip rule add prio 201 from $NWE1/$NME1 table 201
ip route add default via $GWE1 dev $IFE1 src 192.168.0.3 proto static table 201
ip route add default via $GWE1 dev $IFE1 src 192.168.0.4 proto static table 201
ip route add default via $GWE1 dev $IFE1 src $IPE1 proto static table 201
ip route append prohibit default table 201 metric 1 proto static
ip rule add prio 202 from $NWE2/$NME2 table 202
ip route add default via $GWE2 dev $IFE2 src 172.16.0.3 proto static table 202
ip route add default via $GWE2 dev $IFE2 src $IPE2 proto static table 202
ip route add default via $GWE2 dev $IFE2 table 202
ip route append prohibit default table 202 metric 1 proto static
echo "Default Multipath Route and setting weights!"
ip rule add prio 222 table 222
#ETH1 is a lot faster
ip route add default table 222 proto static \
nexthop via $GWE1 dev $IFE1 weight 3 \
nexthop via $GWE2 dev $IFE2 weight 1
Next up is the iptables part of it all with source natting.
echo "Setting the Chains on Filter and Nat"
$IPTABLES -t filter -N keep_state
$IPTABLES -t filter -A keep_state -m state --state RELATED,ESTABLISHED \
-j ACCEPT
$IPTABLES -t filter -A keep_state -j RETURN
$IPTABLES -t nat -N keep_state
$IPTABLES -t nat -A keep_state -m state --state RELATED,ESTABLISHED \
-j ACCEPT
$IPTABLES -t nat -A keep_state -j RETURN
echo "Setting Jumps for PRE POST OUT IN FORWARD OUT for NAT and FILTER"
$IPTABLES -t nat -A PREROUTING -j keep_state
$IPTABLES -t nat -A POSTROUTING -j keep_state
$IPTABLES -t nat -A OUTPUT -j keep_state
$IPTABLES -t filter -A INPUT -j keep_state
$IPTABLES -t filter -A FORWARD -j keep_state
$IPTABLES -t filter -A OUTPUT -j keep_state
echo "IPSEC Policy"
$IPTABLES -t nat -A POSTROUTING -m policy --dir out \
--pol ipsec --mode tunnel -j ACCEPT
$IPTABLES -t nat -A POSTROUTING -m policy --dir in \
--pol ipsec --mode tunnel -j ACCEPT
echo " SNAT for servers on virt IPs"
$IPTABLES -t nat -A POSTROUTING -o $IFE1 -s 10.0.0.5 \
-j SNAT --to 192.168.0.3
$IPTABLES -t nat -A POSTROUTING -o $IFE2 -s 10.0.0.5 \
-j SNAT --to 172.168.0.3
echo " SNAT-Default"
$IPTABLES -t nat -A POSTROUTING -o $IFE1 -s $NWI/$NMI -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o $IFE2 -s $NWI/$NMI -j SNAT --to $IPE2
echo "Protocol forwards"
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE1 -d $IPE1 --dport 25 \
-j DNAT --to 10.0.0.2
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE1 -d 192.168.0.3 --dport 25 \ -j DNAT --to 10.0.0.5
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE2 -d $IPE2 --dport 25 \
-j DNAT --to 10.0.0.2
$IPTABLES -t nat -A PREROUTING -p TCP -i $IFE2 -d 172.16.0.3 --dport 25 \ -j DNAT --to 10.0.0.5
echo "Dropping Everything else in INPUT chain"
$IPTABLES -A INPUT -i $IFE1 -j DROP
$IPTABLES -A INPUT -i $IFE2 -j DROP
NetworkingLinux Networking
Last Comment
gheist