Why is Iptables kicking my backside? IP Forwarding rule not working as expected
I am running a small subnet which has several servers performing their varied tasks, one of which is a a PostGres SQL box. This small group has a management server which has two nics - one on the primary net and the other on the "internal".
The two servers in question we will call "Jump" and "PG"
Jump has an ip address of 10.10.10.10 for the primary net and an address of 192.168.20.10 for the internal net.
BG has an ip address of 192.168.20.40
My iptables rules are as follows:
iptables -A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 5432 -j ACCEPT
iptables -A FORWARD -d 192.168.20.40/32 -p tcp -m tcp --dport 5432 -m comment --comment "Forward inbound PGAdmin traffic to postgres server" -j ACCEPT
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j DNAT --to-destination 192.168.20.40:543
iptables -t nat -A POSTROUTING -s 192.168.26.44 -o eth0 -j MASQUERADE
PGAdmin is configured to address the Jump server on port 5432 and when attempting contact, the traffic arrives to the PG server and returns to my workstation but with a hitch.
Watching tcpdump on the linux boxes and Wireshark on the Windows box, I am able to see the tcp stream initiate at my workstation travel through the jump box and arrive at the PG box. When the packet has arrived at the PG box, the source is my workstation.
IP 10.10.10.11.54564 > 192.168.20.40.5432: Flags [S]
The PG box sends a reply to my workstation:
IP 192.168.20.40.5432 > 10.10.10.11.49561 FLAGS [S] ack
On my workstation here is how Wireshark sees the transaction: (truncated for clarity)
Source Dest Protocol Info
10.10.10.11 10.10.10.10 TCP 66 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
10.10.10.11 10.10.10.10 TCP 66 [TCP Retransmission] 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
10.10.10.11 10.10.10.10 TCP 62 [TCP Retransmission] 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
After which the connection dies and PGAdmin reports a connection timeout.
Since 10.10.10.11 is getting replies directly from 192.168.20.40, I am pretty sure this is the reason for the failed connection. The original traffic was sent from 10.10.10.11 to 10.10.10.10 which successfully forwarded the packets to 192.168.20.40. This is the most success I have had.
If I attempt to use the iptables rule:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j REDIRECT --to-port 5432
and
iptables -t nat -A POSTROUTING -j MASQUERADE
I receive a connection refused on PGAdmin. I also see no traffic on either linux system. Wireshark shows immediate RST
every other rule I have tried, gives me the same result of connection refused.
Not sure what I am not doing right here.
The two servers in question we will call "Jump" and "PG"
Jump has an ip address of 10.10.10.10 for the primary net and an address of 192.168.20.10 for the internal net.
BG has an ip address of 192.168.20.40
My iptables rules are as follows:
iptables -A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 5432 -j ACCEPT
iptables -A FORWARD -d 192.168.20.40/32 -p tcp -m tcp --dport 5432 -m comment --comment "Forward inbound PGAdmin traffic to postgres server" -j ACCEPT
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j DNAT --to-destination 192.168.20.40:543
iptables -t nat -A POSTROUTING -s 192.168.26.44 -o eth0 -j MASQUERADE
PGAdmin is configured to address the Jump server on port 5432 and when attempting contact, the traffic arrives to the PG server and returns to my workstation but with a hitch.
Watching tcpdump on the linux boxes and Wireshark on the Windows box, I am able to see the tcp stream initiate at my workstation travel through the jump box and arrive at the PG box. When the packet has arrived at the PG box, the source is my workstation.
IP 10.10.10.11.54564 > 192.168.20.40.5432: Flags [S]
The PG box sends a reply to my workstation:
IP 192.168.20.40.5432 > 10.10.10.11.49561 FLAGS [S] ack
On my workstation here is how Wireshark sees the transaction: (truncated for clarity)
Source Dest Protocol Info
10.10.10.11 10.10.10.10 TCP 66 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
10.10.10.11 10.10.10.10 TCP 66 [TCP Retransmission] 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
10.10.10.11 10.10.10.10 TCP 62 [TCP Retransmission] 54564 → 5432 [SYN]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
192.168.22.40 10.10.10.11 TCP 66 [TCP Retransmission] 5432 → 54564 [SYN, ACK]
After which the connection dies and PGAdmin reports a connection timeout.
Since 10.10.10.11 is getting replies directly from 192.168.20.40, I am pretty sure this is the reason for the failed connection. The original traffic was sent from 10.10.10.11 to 10.10.10.10 which successfully forwarded the packets to 192.168.20.40. This is the most success I have had.
If I attempt to use the iptables rule:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j REDIRECT --to-port 5432
and
iptables -t nat -A POSTROUTING -j MASQUERADE
I receive a connection refused on PGAdmin. I also see no traffic on either linux system. Wireshark shows immediate RST
every other rule I have tried, gives me the same result of connection refused.
Not sure what I am not doing right here.
ASKER
Arnold -
Chain INPUT (policy DROP)
num target prot opt source destination
1 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
2 ACCEPT icmp -- anywhere anywhere
3 ACCEPT all -- anywhere anywhere
4 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
5 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:591 /* HBSS */
6 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:1002 /* HPSA */
7 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:3001 /* HPSA */
8 ACCEPT icmp -- anywhere anywhere icmp echo-reply
9 ACCEPT icmp -- anywhere anywhere icmp destination-unreachable
10 ACCEPT icmp -- anywhere anywhere icmp time-exceeded
11 ACCEPT icmp -- anywhere anywhere icmp echo-request
12 ACCEPT tcp -- anywhere anywhere state NEW tcp multiport dports https,pcsync-https
13 ACCEPT tcp -- anywhere anywhere state NEW,ESTABLISHED tcp dpt:postgres
Chain FORWARD (policy DROP)
num target prot opt source destination
1 ACCEPT tcp -- anywhere postgres.box tcp dpt:postgres /* Forward inbound PGAdmin traffic to postgres server */
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 REDIRECT tcp -- anywhere anywhere tcp dpt:https redir ports 8443
2 DNAT tcp -- anywhere anywhere tcp dpt:postgres to:192.168.20.40:5432
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 192.168.20.40 anywhere
I don't have a DNAT table or Chain...is that where I am wrong?
TCP forwarding is set to 1 in sysctl.conf
The 192.168.20.0 net is not natted as far as I know.
Chain INPUT (policy DROP)
num target prot opt source destination
1 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
2 ACCEPT icmp -- anywhere anywhere
3 ACCEPT all -- anywhere anywhere
4 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
5 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:591 /* HBSS */
6 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:1002 /* HPSA */
7 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:3001 /* HPSA */
8 ACCEPT icmp -- anywhere anywhere icmp echo-reply
9 ACCEPT icmp -- anywhere anywhere icmp destination-unreachable
10 ACCEPT icmp -- anywhere anywhere icmp time-exceeded
11 ACCEPT icmp -- anywhere anywhere icmp echo-request
12 ACCEPT tcp -- anywhere anywhere state NEW tcp multiport dports https,pcsync-https
13 ACCEPT tcp -- anywhere anywhere state NEW,ESTABLISHED tcp dpt:postgres
Chain FORWARD (policy DROP)
num target prot opt source destination
1 ACCEPT tcp -- anywhere postgres.box tcp dpt:postgres /* Forward inbound PGAdmin traffic to postgres server */
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 REDIRECT tcp -- anywhere anywhere tcp dpt:https redir ports 8443
2 DNAT tcp -- anywhere anywhere tcp dpt:postgres to:192.168.20.40:5432
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 192.168.20.40 anywhere
I don't have a DNAT table or Chain...is that where I am wrong?
TCP forwarding is set to 1 in sysctl.conf
The 192.168.20.0 net is not natted as far as I know.
ASKER
Addendum -
All the wireshark entries should say 20.40 vs. 22.40. I frakked up the find and replace when changing real IPs.
All the wireshark entries should say 20.40 vs. 22.40. I frakked up the find and replace when changing real IPs.
DNAT is the one that deals with forwarding.
Often it is simpler to add a a chain to INPUT and PREROUTING
and have the single rule there (within the new chain) that applies to both
Is your linux system functioning as a router?
eth0 in
eth1 out?
Your definition is such that eth0 NATS all traffic to eth1?
Does your HTTPs port redirect work?
Commonly even if your WAN side has a private IP space, the linux box has to be setup as though it is a router/
The idea is that a response from 192.168.20.40 has to be converted to 10.10.10.11 before it is sent out (source address change which is done through the SNAT Source NAT rule on the outgoing side OUTPUT side of the nat......
request comes in to 10.10.10.11 5432 The destination is altered to reflect 192.168.20.40:543 while the source is an external IP.
the response is then going through the postrouting to output and needs to alter the source instead of 192.168.20.40:543 to 10.10.10.11:5432 destinated to external IP.
what is your workstation's IP in this setup?
the wireshark should not be seeing that many IPs 10.10.10.10 10.10.10.11 and 192.168.20.40
In your example, the source of the response should be 10.10.10.10:5432 => 10.10.10.11:54564
src src port destination destination port
The data has to flow
IP1 port1 ip2 port2 request
ip2 port2 ip1 port1 response
if the response source IP or source port does not match the first, the response will be ignored by the requesting side. i.e. you say jim/jane , and harry or mary answer.....
this seems to be what is going with your traces someone else 192.168.20.40 is responding to the request 10.10.10.11 sent to 10.10.10.10 5432 and expects a response from 10.10.10.10 5432
Often it is simpler to add a a chain to INPUT and PREROUTING
and have the single rule there (within the new chain) that applies to both
Is your linux system functioning as a router?
eth0 in
eth1 out?
Your definition is such that eth0 NATS all traffic to eth1?
Does your HTTPs port redirect work?
Commonly even if your WAN side has a private IP space, the linux box has to be setup as though it is a router/
The idea is that a response from 192.168.20.40 has to be converted to 10.10.10.11 before it is sent out (source address change which is done through the SNAT Source NAT rule on the outgoing side OUTPUT side of the nat......
request comes in to 10.10.10.11 5432 The destination is altered to reflect 192.168.20.40:543 while the source is an external IP.
the response is then going through the postrouting to output and needs to alter the source instead of 192.168.20.40:543 to 10.10.10.11:5432 destinated to external IP.
what is your workstation's IP in this setup?
the wireshark should not be seeing that many IPs 10.10.10.10 10.10.10.11 and 192.168.20.40
In your example, the source of the response should be 10.10.10.10:5432 => 10.10.10.11:54564
src src port destination destination port
The data has to flow
IP1 port1 ip2 port2 request
ip2 port2 ip1 port1 response
if the response source IP or source port does not match the first, the response will be ignored by the requesting side. i.e. you say jim/jane , and harry or mary answer.....
this seems to be what is going with your traces someone else 192.168.20.40 is responding to the request 10.10.10.11 sent to 10.10.10.10 5432 and expects a response from 10.10.10.10 5432
Please refrain from wasting Experts' time by mangling private IP numbers. They are all internal to your network, no-one outside can get at them.
Please run
From your Wireshark trace, I think I see your problem anyway:
IP 10.10.10.11 sends a SYN segment to IP 10.10.10.10 from port 54564 to port 5432.
It expects to get a response from IP 10.10.10.10 from port 5432 to port 54564.
It does not get one. Instead it gets a response from IP 192.168.22.40. Re-transmissions follow.
You need source NAT (SNAT) on traffic from 192.168.22.40 to 10.10.10.11. MASQUERADE may be adequate but is inappropriate. From man iptables-extensions (which I strongly encourage you to read)
Please run
{ set -x;for i in filter nat mangle raw;do iptables -t $i -L -v -n --line-numbers;done;set +x; }From your Wireshark trace, I think I see your problem anyway:
IP 10.10.10.11 sends a SYN segment to IP 10.10.10.10 from port 54564 to port 5432.
It expects to get a response from IP 10.10.10.10 from port 5432 to port 54564.
It does not get one. Instead it gets a response from IP 192.168.22.40. Re-transmissions follow.
You need source NAT (SNAT) on traffic from 192.168.22.40 to 10.10.10.11. MASQUERADE may be adequate but is inappropriate. From man iptables-extensions (which I strongly encourage you to read)
MASQUERADE
This target is only valid in the nat table, in the POSTROUTING chain. It should only be used with dynamically assigned IP (dialup) connections: if you have a static IP address, you should use the SNAT target. Masquerading is equivalent to specifying a mapping to the IP address of the interface the packet is going out, but also has the effect that connections are forgotten when the interface goes down. This is the correct behavior when the next dialup is unlikely to have the same interface address (and hence any established connections are lost anyway).
ASKER
Arnold - The HTTPS redirect is working. As far as the box being used as a router, I will have to dig in order to answer that. Still getting up to speed on this network setup. I will look at this in the am first thing.
Duncan - Point taken...too much paranoia I guess. I will run the code tomorrow am and put it in. I was thinking about SNAT as well, and had read about it in man iptables. I didn't know about the iptables-extensions file and will read that as well. I'll get back to you with what I find.
Thank you.
Duncan - Point taken...too much paranoia I guess. I will run the code tomorrow am and put it in. I was thinking about SNAT as well, and had read about it in man iptables. I didn't know about the iptables-extensions file and will read that as well. I'll get back to you with what I find.
Thank you.
Run
Iptables -t filter -L --line-numbers
iptables -t nat -L --line-numbers
Look at the port 443 rule and make sure you have it match.
Line 12 input is where your httpd is accepted, there is no input rule for 5432
Better still, look at /etc/iptables/ or is it /etc/systemconf/iptables/.
Port 443 is on the box, centos it is not being forwarded to a system on the 192.168.20.0/24 you have to be natting the 192.168.20.0/24 for the SNAT and DNAT...
Iptables -t filter -L --line-numbers
iptables -t nat -L --line-numbers
Look at the port 443 rule and make sure you have it match.
Line 12 input is where your httpd is accepted, there is no input rule for 5432
Better still, look at /etc/iptables/ or is it /etc/systemconf/iptables/.
Port 443 is on the box, centos it is not being forwarded to a system on the 192.168.20.0/24 you have to be natting the 192.168.20.0/24 for the SNAT and DNAT...
ASKER
1st - thank you for sticking with me on this. This being my second question I think. I will try to be a little more clear. I realize that I made my question too confusing.
Arnold -
After getting back into the system, I have found that the system is not being used as a router. It merely is configured to move the web traffic as that redirect.
Line 13 above is my ACCEPT for the 5432 - postgres port.
Duncan -
After getting back into the man iptables (extensions section - I understood you to mean a different man page not a section d'oh) I did indeed read that part about SNAT but didn't really get it. I have reread it and I think I got it.
The output of the code snippet:
After reading the SNAT section of the man page, I removed the MASQUERADE rule and replaced it with the following:
The results with the new rule are the same (wireshark capture at my box):
So - I am still missing the boat with the SNAT. I expected that my setting the out interface to the 10 nework iface, and SNAT for the specific source port, that only the traffic coming into the Jump box with a source port of 5432 would trigger the SNAT rule.
I also watched a tcpdump on the postgres server and the following is the dump during a connection attempt from my workstation:
It looks to me like the DNAT isn't working? The 10.22.49.80 address is my workstation, not the Jump box.
Thanks again gents. Hope this is a bit better.
Arnold -
After getting back into the system, I have found that the system is not being used as a router. It merely is configured to move the web traffic as that redirect.
Line 13 above is my ACCEPT for the 5432 - postgres port.
Duncan -
After getting back into the man iptables (extensions section - I understood you to mean a different man page not a section d'oh) I did indeed read that part about SNAT but didn't really get it. I have reread it and I think I got it.
The output of the code snippet:
Chain INPUT (policy DROP 712 packets, 84134 bytes)
num pkts bytes target prot opt in out source destination
1 2423K 365M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
3 15 716 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 13 676 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
5 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:591 /* HBSS */
6 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:1002 /* HPSA */
7 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3001 /* HPSA */
8 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 0
9 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 3
10 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 11
11 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 8
12 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp multiport dports 443,8443 /* Jive ports */
13 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW,ESTABLISHED tcp dpt:5432
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 6 304 ACCEPT tcp -- * * 0.0.0.0/0 192.168.26.44 tcp dpt:5432 /* Forward inbound PGAdmin traffic to postgres server */
Chain OUTPUT (policy ACCEPT 1036 packets, 89702 bytes)
num pkts bytes target prot opt in out source destination
Chain LOGGING (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/min burst 5 LOG flags 0 level 7 prefix `IPTables Packet Dropped: '
2 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain PREROUTING (policy ACCEPT 5364 packets, 663K bytes)
num pkts bytes target prot opt in out source destination
1 2 96 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 redir ports 8443
2 2 104 DNAT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5432 to:192.168.26.44:5432
Chain POSTROUTING (policy ACCEPT 19 packets, 1269 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 MASQUERADE all -- * eth0 192.168.22.44 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 17 packets, 1165 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 REDIRECT tcp -- * * 0.0.0.0/0 127.0.0.1 tcp dpt:443 redir ports 8443
Chain PREROUTING (policy ACCEPT 13343 packets, 1511K bytes)
num pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 9494 packets, 1052K bytes)
num pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 6 packets, 304 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 7114 packets, 656K bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 7120 packets, 656K bytes)
num pkts bytes target prot opt in out source destination
Chain PREROUTING (policy ACCEPT 13343 packets, 1511K bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 7114 packets, 655K bytes)
num pkts bytes target prot opt in out source destinationAfter reading the SNAT section of the man page, I removed the MASQUERADE rule and replaced it with the following:
iptables -t nat -A POSTROUTING -o eth0 -j SNAT -p tcp --source-port 5432 --to-source 10.22.48.40The results with the new rule are the same (wireshark capture at my box):
No. Time Source Destination Protocol Length Info
1 0.000000 10.22.49.80 10.22.48.40 TCP 66 53480 → 5432 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 0.001321 192.168.26.44 10.22.49.80 TCP 66 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
3 1.250265 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
4 3.001106 10.22.49.80 10.22.48.40 TCP 66 [TCP Retransmission] 53480 → 5432 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
5 3.002222 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
6 3.252343 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
7 7.250455 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
8 9.001278 10.22.49.80 10.22.48.40 TCP 62 [TCP Retransmission] 53480 → 5432 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 SACK_PERM=1
9 9.002447 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
10 15.250788 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128
11 31.250912 192.168.26.44 10.22.49.80 TCP 66 [TCP Retransmission] 5432 → 53480 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 SACK_PERM=1 WS=128So - I am still missing the boat with the SNAT. I expected that my setting the out interface to the 10 nework iface, and SNAT for the specific source port, that only the traffic coming into the Jump box with a source port of 5432 would trigger the SNAT rule.
I also watched a tcpdump on the postgres server and the following is the dump during a connection attempt from my workstation:
12:19:24.620799 IP 10.22.49.80.62012 > 192.168.26.44.postgres: Flags [S], seq 499676986, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
12:19:24.620871 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:25.620430 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:27.620476 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:27.620570 IP 10.22.49.80.62012 > 192.168.26.44.postgres: Flags [S], seq 499676986, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
12:19:27.620587 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:31.620506 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:33.620712 IP 10.22.49.80.62012 > 192.168.26.44.postgres: Flags [S], seq 499676986, win 8192, options [mss 1460,nop,nop,sackOK], length 0
12:19:33.620748 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:39.620659 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:19:55.620430 IP 192.168.26.44.postgres > 10.22.49.80.62012: Flags [S.], seq 953006703, ack 499676987, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0It looks to me like the DNAT isn't working? The 10.22.49.80 address is my workstation, not the Jump box.
Thanks again gents. Hope this is a bit better.
Your SNAT rule must match the IP to which requests are sent. Based on the capture, it shoukd be 10.10.10.10
Second double check your SNAT rule, it shoukd apply to the network interface where 192.168.20.0/24 IPs are.
ifconfig -a
If you need to setup this system as a reverse proxy to shield PostgreSQL, webservers, ....
Let's possibly start from the begining, what is it you are trying to do?
What is the end result you are trying to achieve?
Second double check your SNAT rule, it shoukd apply to the network interface where 192.168.20.0/24 IPs are.
ifconfig -a
If you need to setup this system as a reverse proxy to shield PostgreSQL, webservers, ....
Let's possibly start from the begining, what is it you are trying to do?
What is the end result you are trying to achieve?
Unless you have a very old installation, the command is man iptables-extensions. What do you get from iptables --version? - I get iptables v1.6.0.
That will not make any difference to the resolution of your current problem. I have to go right now - will look at this more carefully but maybe the day after tomorrow.
That will not make any difference to the resolution of your current problem. I have to go right now - will look at this more carefully but maybe the day after tomorrow.
ASKER
Arnold - I have several small instances where I have a postgres database on a small sub-network sitting in the 192 address space. There is a single server which has two NICs which we use to jump into the rest of the instance. Currently, we have to ssh into the jump box and then run the PGAdmin tool from there in order to manage the database. My preference is to use the Jump box to forward the PGAdmin traffic to the DB Server so that we are able to use the PGAdmin tool from our own workstations. I just didn't think that I could miss the boat so badly.
Duncan - Our IPTables version is 1.4.7. I am not sure at this point about any updates. I recently inherited this series of systems, so I am still on the uptake with configs and versioning etc.
Duncan - Our IPTables version is 1.4.7. I am not sure at this point about any updates. I recently inherited this series of systems, so I am still on the uptake with configs and versioning etc.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
There may be more to it than this but:
Your MASQUERADE rule specifies a different address from your DNAT rule (192.168.26.44 differs from 192.168.22.44 in the 3rd octet).
I cannot easily tell which tables the rules are in because you failed to run the complete command I specified in https:#a42226314 - it has set -x and set +x so you can see which table has what. Please run it properly before asking for more help
Your MASQUERADE rule specifies a different address from your DNAT rule (192.168.26.44 differs from 192.168.22.44 in the 3rd octet).
I cannot easily tell which tables the rules are in because you failed to run the complete command I specified in https:#a42226314 - it has set -x and set +x so you can see which table has what. Please run it properly before asking for more help
ASKER
Duncan - after verifying in my history file, I am able to assure you that I ran the code exactly as you put it in above.
I have run it again and have included the line you sent in the snip below. My rules currently have an SNAT in the POSTROUTING rather than the MASQEURADE.
Arnold -
I appreciate the re-direct to using tunneling. I have seen several examples which I am trying today. Thanks.
I have run it again and have included the line you sent in the snip below. My rules currently have an SNAT in the POSTROUTING rather than the MASQEURADE.
{ set -x;for i in filter nat mangle raw;do iptables -t $i -L -v -n --line-numbers;done;set +x; }
+ for i in filter nat mangle raw
+ iptables -t filter -L -v -n --line-numbers
Chain INPUT (policy DROP 4181 packets, 629K bytes)
num pkts bytes target prot opt in out source destination
1 1890K 408M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
3 663 35316 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 18 952 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
5 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:591 /* HBSS */
6 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:1002 /* HPSA */
7 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3001 /* HPSA */
8 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 0
9 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 3
10 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 11
11 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 8
12 309 16268 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp multiport dports 443,8443 /* Jive ports */
13 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW,ESTABLISHED tcp dpt:5432
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT tcp -- * * 0.0.0.0/0 192.168.26.44 tcp dpt:5432 /* Forward inbound PGAdmin traffic to postgres server */
Chain OUTPUT (policy ACCEPT 7831 packets, 754K bytes)
num pkts bytes target prot opt in out source destination
Chain LOGGING (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/min burst 5 LOG flags 0 level 7 prefix `IPTables Packet Dropped: '
2 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
+ for i in filter nat mangle raw
+ iptables -t nat -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 150 packets, 20756 bytes)
num pkts bytes target prot opt in out source destination
1 310 16292 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 redir ports 8443
2 0 0 DNAT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5432 to:192.168.26.44:5432
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 SNAT tcp -- * eth0 0.0.0.0/0 0.0.0.0/0 tcp spt:5432 to:10.22.48.40
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 REDIRECT tcp -- * * 0.0.0.0/0 127.0.0.1 tcp dpt:443 redir ports 8443
+ for i in filter nat mangle raw
+ iptables -t mangle -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 1363K packets, 206M bytes)
num pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 929K packets, 149M bytes)
num pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 723K packets, 94M bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 723K packets, 94M bytes)
num pkts bytes target prot opt in out source destination
+ for i in filter nat mangle raw
+ iptables -t raw -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 1363K packets, 206M bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 723K packets, 94M bytes)
num pkts bytes target prot opt in out source destination
+ set +xArnold -
I appreciate the re-direct to using tunneling. I have seen several examples which I am trying today. Thanks.
Looks OK to me - but there are no port 5432 packets in either direction. Are they still not working? Can you post another tcpdump if so please?
ASKER
Good morning -
Thanks for sticking with me on this.
Here is the dump from the Jump Box which is running the IP Tables config above:
Here is the dump from the Postgres box:
Also - here is a new capture of the IP Tables.
Arnold - I have been trying to get back to the SSH tunnel thing, but we had a ESXi problem and I have been getting the cluster back from being a *CLUSTER*
Thanks for sticking with me on this.
Here is the dump from the Jump Box which is running the IP Tables config above:
13:19:11.447588 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:19:14.147017 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:19:19.547173 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,nop,sackOK], length 0Here is the dump from the Postgres box:
13:18:35.029523 IP 192.168.26.44.postgres > 10.22.49.80.51163: Flags [S.], seq 3419792773, ack 1187850173, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:47.473092 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:18:47.473126 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:48.629524 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:50.172505 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:18:50.172530 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:50.629525 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:54.629522 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:18:55.572572 IP 10.22.49.80.51476 > 192.168.26.44.postgres: Flags [S], seq 3860119586, win 8192, options [mss 1460,nop,nop,sackOK], length 0
13:18:55.572599 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:19:02.641583 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:19:18.841554 IP 192.168.26.44.postgres > 10.22.49.80.51476: Flags [S.], seq 2031884984, ack 3860119587, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0Also - here is a new capture of the IP Tables.
{ set -x;for i in filter nat mangle raw;do iptables -t $i -L -v -n --line-numbers;done;set +x; }
+ for i in filter nat mangle raw
+ iptables -t filter -L -v -n --line-numbers
Chain INPUT (policy DROP 1826K packets, 283M bytes)
num pkts bytes target prot opt in out source destination
1 4421K 801M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
3 675 35796 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 22 1160 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
5 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:591 /* HBSS */
6 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:1002 /* HPSA */
7 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3001 /* HPSA */
8 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 0
9 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 3
10 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 11
11 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 8
12 388 20376 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp multiport dports 443,8443 /* Jive ports */
13 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW,ESTABLISHED tcp dpt:5432
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 9 456 ACCEPT tcp -- * * 0.0.0.0/0 192.168.26.44 tcp dpt:5432 /* Forward inbound PGAdmin traffic to postgres server */
Chain OUTPUT (policy ACCEPT 3577K packets, 411M bytes)
num pkts bytes target prot opt in out source destination
Chain LOGGING (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/min burst 5 LOG flags 0 level 7 prefix `IPTables Packet Dropped: '
2 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
+ for i in filter nat mangle raw
+ iptables -t nat -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 3772K packets, 543M bytes)
num pkts bytes target prot opt in out source destination
1 395 20688 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 redir ports 8443
2 3 156 DNAT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5432 to:192.168.26.44:5432
Chain POSTROUTING (policy ACCEPT 16164 packets, 1131K bytes)
num pkts bytes target prot opt in out source destination
1 0 0 SNAT tcp -- * eth0 0.0.0.0/0 0.0.0.0/0 tcp spt:5432 to:10.22.48.40
Chain OUTPUT (policy ACCEPT 16161 packets, 1131K bytes)
num pkts bytes target prot opt in out source destination
1 0 0 REDIRECT tcp -- * * 0.0.0.0/0 127.0.0.1 tcp dpt:443 redir ports 8443
+ for i in filter nat mangle raw
+ iptables -t mangle -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 7667K packets, 1143M bytes)
num pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 5282K packets, 825M bytes)
num pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 9 packets, 456 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 4292K packets, 504M bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 4292K packets, 504M bytes)
num pkts bytes target prot opt in out source destination
+ for i in filter nat mangle raw
+ iptables -t raw -L -v -n --line-numbers
Chain PREROUTING (policy ACCEPT 7667K packets, 1143M bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 4292K packets, 504M bytes)
num pkts bytes target prot opt in out source destination
+ set +xArnold - I have been trying to get back to the SSH tunnel thing, but we had a ESXi problem and I have been getting the cluster back from being a *CLUSTER*
Here is the issue you are facing dealing with, you are trying to convert a jumpbox to a router by using iptables.
Please post the /etc/sysconfig/iptables I.e. The setup/settings
Your destinatiion rewrite rule works, you need to at the same time rewrite the source to reflect the jump box as the source of the packet on the packet going to the other system.
IPA to jumpbox:
Jumpbox to ipc:postgre
ipc:postgres response to jumpbox
Jumpbox response to IPA
This would have been done through the nat mechanism from one interface to the other.
Commonly when used/setup jump boxes, they usually never configured as a bridge as you have it connected to two different segments, the setup usually is they are setup in a tier that has access rights to others.
If this system is externally accessible, I would suggest/recommend you abandon this pursuit as it might be countermanding policy, using iptables circumvents the need for authentication in order to access internal resources,potentially exposing .....
Please post the /etc/sysconfig/iptables I.e. The setup/settings
Your destinatiion rewrite rule works, you need to at the same time rewrite the source to reflect the jump box as the source of the packet on the packet going to the other system.
IPA to jumpbox:
Jumpbox to ipc:postgre
ipc:postgres response to jumpbox
Jumpbox response to IPA
This would have been done through the nat mechanism from one interface to the other.
Commonly when used/setup jump boxes, they usually never configured as a bridge as you have it connected to two different segments, the setup usually is they are setup in a tier that has access rights to others.
If this system is externally accessible, I would suggest/recommend you abandon this pursuit as it might be countermanding policy, using iptables circumvents the need for authentication in order to access internal resources,potentially exposing .....
ASKER
Arnold -
I think I am with you here. While I am not concerned about external access too much (this is a closed net in a lab) I am concerned with setting it up the right way rather than trying to bludgeon my way into a wonky config.
So I will follow your suggestion and work on the SSH tunnel. I will let you know how it goes.
I think I am with you here. While I am not concerned about external access too much (this is a closed net in a lab) I am concerned with setting it up the right way rather than trying to bludgeon my way into a wonky config.
So I will follow your suggestion and work on the SSH tunnel. I will let you know how it goes.
ASKER
Arnold -
I successfully connected with the SSH Tunnel via Putty. I had to dance a little with the config you gave me since I am using Putty, but this did the trick.
Thanks for the help.
Not sure which answer to call best solution.
I successfully connected with the SSH Tunnel via Putty. I had to dance a little with the config you gave me since I am using Putty, but this did the trick.
Thanks for the help.
Not sure which answer to call best solution.
Since the tunneling solution is working for you, I think would do best to stick with it.
Since the jump box is not a router, it seems to me that what is happening is that the return segments to 192.168.26.44 are not being sent to it. You would have to set this up as a route on the postgres box, and it would have to be to an IP address (you can't route dependent on destination port).
Since the jump box is not a router, it seems to me that what is happening is that the return segments to 192.168.26.44 are not being sent to it. You would have to set this up as a route on the postgres box, and it would have to be to an IP address (you can't route dependent on destination port).
IN putty you can create sessions and under the ssh, tunnel, you can add/specify several local tunnels to allow you to connect through your jumpbox to other systems using localhost:localport to connect to a system local/accessible by the jumpbox.
Saving those settings in the session..
Pick the piste/answered that you feel helped you resolve provide an alternate approach solution for your situation.
Saving those settings in the session..
Pick the piste/answered that you feel helped you resolve provide an alternate approach solution for your situation.
ASKER
Thanks much for the patience and assistance. Using Arnold's advice, I was able to set up a double tunnel in Putty which everyone is able to duplicate to run PGAdmin locally and get to the boxes behind the Jump boxes. Duncan's comments helped me to understand IPTables more though.
iptables -t filter -L INPUT --line-numbers
Repeat the DNAT and PREROUTING, forward
Is this system configured as a rputer /etc/system deals with forward rule for TCP.
Presumably other iptables work versus using firewalld....
Is the 192.168.40.0/24 natted behind the 10.10.10.x segment.