Issues with routing VPN Traffic To Local Subnet

i think first there has to be a route to the 172.20.net pointing to the remote-router.
and vice versa.

in ipsec, you can set up vpn-policies, that configure the hosts, that are allowed to communicate over the tunnel.
so you may have routes correctly set up, but packets maybe dropped by ipsec.

can you post some more parts of your config?

Alright, the route to 172.20 exists; Packets route from the remote VPN box to any 172.20 address fine. Packets from the VPN box's local network do not - they are received by the VPN box, as evidenced by tcpdump, and then the VPN box never forwards the packet over the IPSEC tunnel.

ipsec.conf:

conn uplink-store
    authby=secret
    left=173.203.49.147
    leftsubnet=10.179.49.147/16
    leftsourceip=10.179.49.147
    right=67.79.33.XXX
    rightsubnet=172.20.0.0/16
    auto=start


route:

Kernel IP routing table
Destination     Gateway              Genmask                 Flags Metric Ref      Use Iface
10.179.59.139   *                         255.255.255.255     UH    0      0        0    eth1
173.203.115.0   *                         255.255.255.0         U     0      0        0     eth0
10.179.0.0         *                         255.255.0.0             U     0      0        0     eth1
172.20.0.0         *                         255.255.0.0             U     0      0        0     eth0
default               173.203.115.1   0.0.0.0                      UG    0      0        0   eth0
10.179.0.0         10.179.49.147   255.255.0.0             UG     0       0       0   eth1  <----

The last line I added to try and get the packets to route but it didn't help. It has since been removed.            

ok, i am confused now with your terms.
give me that infos:

HQ LAN: 10.0.0.0/8
HQ Router LAN IP???
HQ WAN IP ???

BRanch LAN: 172.20..0.0/16
BR Router LAN IP
BR WAN IP

please post:
ipsec.conf and route HQ Router
ipsec.conf and route BR Router

what subnet do you want to go over the tunnel?
the leftsubnet=10.179.49.147/16 looks strange
try leftsubnet = 10.0.0.0/8

whats that 10.179.0.0/16 network?

Its the VPN Box's local network that is supposed to be the second site in the site-to-site tunnel.

All the PCs at Site A (10.179.0.0) need to virtually be on the same LAN as all PCs at site B (172.20.0.0).

Wait a minute... Do the subnets have to be the same? Not being an expert about routing I figured that was the job of the router to link the two subnets...

ok, i am confused now with your terms.
give me that infos:

HQ LAN: 10.0.0.0/8
HQ Router LAN IP??? - 10.179.49.147
HQ WAN IP ??? - 173.203.115.85

BRanch LAN: 172.20..0.0/16
BR Router LAN IP 172.20.1.1
BR WAN IP 67.79.33.XXX (Redacted)

please post:
ipsec.conf and route HQ Router
ipsec.conf and route BR Router - This is a Linksys Router, configured via GUI.

what subnet do you want to go over the tunnel?
the leftsubnet=10.179.49.147/16 looks strange   - I didnt configure this part.
try leftsubnet = 10.0.0.0/8

I changed the left subnet to 10.179.0.0/16.

The settings are perfectly converse on the BRanch.

Still cant ping from 172.20.1.21 to 10.179.6.22.

I wish i could edit posts... Ok so the error is Destination Host Unreachable from either side (from inside HQ net or BR net, I get that error pinging to anything on the subnet on the other side. However, I can ping either router by its local side IP address just fine ONLY from the other router.; i.e. 10.179.49.147)

ok, if you can ping the LAN adresses of the routers acros the tunnel, your routes may be correct.

i think you have a problem with your ipsec-policies. the ipsec.conf maybe ok now.
but check the BR router. maybe he doesnt allow the whole subnet.

try traceroute/tracert an ip in the other subnet to verify at which hop the packets are dropped.

Not getting into the details of your IPs and subnets. I get a little confused unless I see them on the router consoles. Overall I feel your linux box is not functioning as a router and hence not forwarding packets between two different networks.
Check /etc/sysctl.conf - 'the net.ipv4.ip_forward' value should be 1, else the box will not forward packets (i.e. won't behave as a router)
Restart network service if you change the value to 1. Then use NAT option of the iptables to setup IP forwarding and Masquerading so that the linux box forwards packets to and fro (use the below commands)
# iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
# iptables --append FORWARD --in-interface eth1 -j ACCEPT

This should work. Also check that a static route to your local network is configured on your remote end linksys router.

Let me know...

ipv4 forwarding was already on. Does it need to be performing NAT? I'd prefer that hosts could be reached directly without being NAT'd.

nat shouldnt be needed

can you ping the HQ Router LAN IP from the Linksys and vice versa?

then your routes should be ok.

Yes, it would be forwarding the packets received on your internal interface (eth1) to your external interface. Since your VPN box already knows the route to the remote network, it should then forward the packets ahead.

Do give it a try...

Also send me the traceroute output from both the networks...

Yes, I can.. Here's asummary:

(1) HQ Router -> BR Router (Works)
(2) BR Router -> HQ Router (Works)

And these are Destination Host Unreachable Sent by Whichever is the REMOTE router:
(3) HQ Subnet Device -> BR Router (Does not work)
(4) HQ Subnet Device -> BR Subnet Device (Does not work)
(5) BR Subnet Device -> HQ Router (Does Not Work)
(6) BR Subnet Device -> HQ Subnet Device (Does not work)

(The Iptables modification did not fix it. In fact, it disabled the ability for either router to ping eachother). In fact, even after deleting those rules, for some reason I cant even ping router to router.


Which means now the Tracert only stays on the localhost and times out.

The routes are setup on the BR router too - 10.179.0.0/255.255.0.0 -> ipsec0
172.20.0.0 -> LAN

please tracert vom a HQ lan device to a BR lan device
and vice versa.

Correction: Pinging from BR Subnet to HQ Router works. Pinging from HQ Router or HQ Subnet to ANYTHING at the BR fails (Even HQ Router to BR Router).

TRACEROUTE From HOST on BR Network (Not router):

Tracing route to 10.179.49.147 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  172.20.1.1     <--- BR Router
  2    48 ms    52 ms    49 ms  10.179.49.147  <---- HQ Router

Trace complete.

Coming back from HQ Router it just times out at localhost.
Pinging from BR Host to HQ Host gets Destination Host Unreachable From 10.179.49.147

From HQ ROuter to BR Router:

[root@Uplink ~]$ tracert 172.20.1.1
traceroute to 172.20.1.1 (172.20.1.1), 30 hops max, 40 byte packets
 1  Uplink (173.203.115.85)  3007.577 ms !H  3007.580 ms !H  3007.576 ms !H

From HQ Router to BR Host:

[root@Uplink ~]$ traceroute 172.20.1.242
traceroute to 172.20.1.242 (172.20.1.242), 30 hops max, 40 byte packets
 1  Uplink (173.203.115.85)  3006.899 ms !H  3006.886 ms !H  3006.865 ms !H
[root@Uplink ~]$


From HQ Host to BR Router, its
1: 10.179.6.22
2: 173.203.115.85 !H   !H  !H

ok, i wonder, why your 172.20.x route goes out to eth0
shouldnt that be some ipsec-pseudo-interface?

what is at 173.203.115.1?
that seems to be the standard-gw of your HQ-VPN-Router, correct?

well the ipsec0 interface that im used to seeing never existed on the box even after configing OpenSwan
And yes i believe that the 115.1 is the outside-of-our-network gateway

ok, lets summarize something.

can you ping any of the routers LAN! ips from the other subnet
or just the wan ip?

the WAN should be pingable everytime you dont filter icmp.
that doesnt say anything about your tunnel.

please try leftnexthop=173.203.115.1
and forwardcontrol=yes

can you post the output of
ipsec auto --start uplink-store
?

if you do masquerade in postrouting please exempt the packets you want to tunnel
iptables -t nat -A POSTROUTING -o ethx -s 10xxxx -d \! 172xxxx -j MASQUERADE

port 500 and 50 have to be open as well for authentication

iptables -A INPUT -p udp --sport 500 --dport 500 -j ACCEPT
iptables -A INPUT -p 50 -j ACCEPT

please check /var/log/auth.log

you can watch this log using
tail -f /var/log/auth.log
while you try to establish a tunnel

if there is no entry, i think no ipsec-packets reach your gateway
check with
tcpdump -i eth0 not port ssh

can you please post the whole .conf including config setup?

there is no longer an ipsec0 since kernel-2.6/openswan-2.2

i googled that.

what versions do you use?

I can ping the LAN ip of both router from the other.
I can ping the HQ LAN IP from BR Router OR BR Hosts.

Basically, everything works except that the HQ Router doesnt seem to forward packets between the eth1 and IPSEC tunnel.

ok, so the tunnel is established correctly, and you made sure to have the correct policies for the whole subnets.
check with auth.log or kernelmessages or ipsec auto --start uplink-store

you said, that you have basic routing enabled
check in /proc

try forwardcontrol=yes in ipsec.conf
try leftnexthop=173.203.115.1 as well
i googled, that you need this parameter, when you do not specify left with %default

do you use NAT on your HQ?

There is no NAT on the HQ - I am going to try leftnexthop and forward control.

[root@Uplink /]$ cat /proc/sys/net/ipv4/ip_forward
1

[root@Uplink /]$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.179.59.139   *               255.255.255.255 UH    0      0        0 eth1
173.203.115.0   *               255.255.255.0   U     0      0        0 eth0
10.179.0.0      *               255.255.0.0     U     0      0        0 eth1
172.20.0.0      173.203.115.1   255.255.0.0     UG    0      0        0 eth0
default         173.203.115.1   0.0.0.0         UG    0      0        0 eth0


[root@Uplink /]$ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


[root@Uplink /]$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


Ping from HQ Router out just hangs, makes me think the box rejects the output somehow...
[root@Uplink /]$ ping 172.20.1.1
PING 172.20.1.1 (172.20.1.1) 56(84) bytes of data.
^C
--- 172.20.1.1 ping statistics ---
49 packets transmitted, 0 received, 100% packet loss, time 48017ms






ipsec.conf below.

# basic configuration
config setup
        # plutodebug / klipsdebug = "all", "none" or a combation from below:
        # "raw crypt parsing emitting control klips pfkey natt x509 private"
        # eg: plutodebug="control parsing"
        #
        # ONLY enable plutodebug=all or klipsdebug=all if you are a developer !!
        #
        # NAT-TRAVERSAL support, see README.NAT-Traversal
        #nat_traversal=yes
        # virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12
        #
        # enable this if you see "failed to find any available worker"
        nhelpers=0
        forwardcontrol=yes


# Add connections here
conn store-rackspace
        authby=secret
        left=173.203.115.85
        leftsubnet=10.179.0.0/16
        leftnexthop=173.203.115.1
        leftsourceip=10.179.49.147
        right=67.79.33.XXX
        rightsubnet=172.20.0.0/16
        auto=start

#Disable Opportunistic Encryption
include /etc/ipsec.d/examples/no_oe.conf

Select allOpen in new window

Oh, and ofcourse the .XXX is actual numbers.

And heres a host at BR. Shows router VPN (not on config page) and successful ping to the HQ router.

Oops, heres the SS
Host.JPG

(And FYI, that ping is from host 172.20.1.241 on the BR net)

whats the routers lan ip once again?
10.179.59.139   or  10.179.49.147

can you post ifconfig?

49.147

59.139 is another server that SHOULD be on the VPN via the router.

[root@Uplink /]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 40:40:1f:b5:31:34
          inet addr:173.203.115.85  Bcast:173.203.115.255  Mask:255.255.255.0
          inet6 addr: fe80::4240:1fff:feb5:3134/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:107460 errors:0 dropped:0 overruns:0 frame:0
          TX packets:115443 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48861302 (46.5 MiB)  TX bytes:19391054 (18.4 MiB)

eth1      Link encap:Ethernet  HWaddr 40:40:d6:fc:64:fa
          inet addr:10.179.49.147  Bcast:10.179.255.255  Mask:255.255.0.0
          inet6 addr: fe80::4240:d6ff:fefc:64fa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:191 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5392 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15518 (15.1 KiB)  TX bytes:238536 (232.9 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:729 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:95112 (92.8 KiB)  TX bytes:95112 (92.8 KiB)

Select allOpen in new window

sure you use NAT, since you have a private subnet on your lan

exempt the packets going to the remote lan

iptables -t nat -A POSTROUTING -o ethx -s 10xxxx -d \! 172xxxx -j MASQUERADE

is the source and dest supposed to be subnet or IP of the routers?

subnets

try 10.179.0.0/16 and 172.20.0.0/16

if it dowsnt work, try this
iptables -t nat -A POSTROUTING -p ! esp -o eth0 -j MASQUERADE

^_^. This works. The HQ Router now can ping to anything on the BR Subnet.

One last thing, though: The BR Subnet/Router CANNOT ping back to the HQ Subnet, and I dont honestly understand what exactly that last iptables entry did.

I will have to check but i also believe that the HQ subnet cant ping the BR subnet, even though the HQ router can. Its like the rest of the HQ subnet outside the router is still not being forwarded right

IPSEC will go through iptables twice.  First for the IPSEC encoded
packets and finally the decoded packets.  You don't want to run the
decoded packets through the POSTROUTING NAT a second time.  So "-p !
esp" becomes your friend here.

Second, you now have private traffic coming in your firewall on the
external interface (because of this second parsing of the packets).  So
you need your rules to reflect that.  You might want to log everything
before you drop to see what might be getting caught that shouldn't be.

I actually didnt use iptables -t nat -A POSTROUTING -p ! esp -o eth0 -j MASQUERADE yet. The iptables entry before that did it so far..

Lets try the ! esp line

And the last line just made it so neither router can ping itself... That's...odd...

Flushing the table results in functionality again, once I re-add the POSTROUTING rule.

ok, the last line should tell iptables to not masquerade ANY packets with esp-protcol
no metter what source or dest network

this doesnt work???

cant believe. did you forget the ! ???

Alternatively, I could disable private interfaces and configure each server to tunnel to the VPN hub as another site. This is probably a better option, and firewalling all non-IPSEC traffic, as realistically these servers are Cloud slices, and therefore I could firewall the private end and simplify, but then there is still an "insecure" connection over a private net full of other peoples slices.

my problem there is that by adding any other connection profile to ipsec.conf at "HQ" causes the first entry to stop functioning.

How would I specify only a single machine at a site? 10.179.49.147/1 for subnet?

flush all rule and try the last line again.
so that is the only rule

set up a logging for packets that will dropped

post output

i didnt rellay understand your last posting.
if you just want to let 1 server, thats a unicast adress, to go over the tunnel, you use the mask 255.255.255.255, thats /32

Ah, my thinking was backwards. So 10.179.49.147/32 for left subnet?

But the question was why adding a second tunnel config in ipsec.conf disables the first tunnel?

no, you misunderstood that.

if you configure left= 10.179.49.147/32 the only destination, reachable over that tunnel is 10.179.49.147

this MUST work:
iptables -t nat -A POSTROUTING -p ! esp -o eth0 -j MASQUERADE

i just wonder, if you have to setup a second rule with the same but for -i eth0
the incoming traffic

Understand:

The other clients on the HQ subnet also have public IPs; They are Cloud slices.
The HQ Subnet is full of cloud slices that are not mine, and therefore, pose a security risk just the same as the public internet. Is it not just better to configure a tunnel from each "HQ Subnet client" to the HQ VPN router, thereby ensuring that one must be authenticated on the VPN to communicate at all (firewalling off all non-ipsec traffic?)

Also, flushing POSTROUTING and using only:

iptables -t nat -A POSTROUTING -p ! esp -o eth0 -j MASQUERADE

exactly, fails. Packets will go in, but not out with that. Not sure why.

And, upon flushing the tables completely it seems to work for point-to-point but thats ok. This is mostly a solution because the next question is a different matter and it may be me being stupid xD. Thank you much Deibel!

damn!
this must work.
i am sure.
i bet my wife!

your HQ LAN has private IP

what do you mean with your cloud slices?

Its a cloud server. Rackspace. Dedicated. It was never a true HQ, its just a hub for all sites to connect to and communicate with about 14 servers that provide varying functionality to our LAN. We moved some onsite services offsite in order to provide them to new locations.

(basically, all those private IPs are virtual machines on the same host)

thanks for accepting my solution.
it has something to do with masquerading in your iptables

logg the droped packets and see if it brings you further

It's ok because I went ahead and setup my second tunnel and that works. Im going to be disabling eth1 entirely on each of these servers, and firewalling anything non IPSEC.

ok, then do your homework in subnetting!!!

you cant configure a tunnel where one client can access all the other clients VM!

ok, technically you can, but you are rhight with your security considerations.

you have to devide the networks, so one customer can only access his own servers

maybe you go and get an apliance for that

Like I said, I think all interfaces will be totally firewalled off except for VPN traffic, and then just tunnel each of the VMs into the hub.

not only that!

you have to devide the customers VM into subnets.

if you only setup the vpn policy, customer A cant reach customer Bs server through the tunnel from his remote site.
but what, if customer A telenets or RDPs into his VM? the servers can access each other over the HQ LAN.
so i telnet into my server and telnet from there to customer Bs server.

got that?

I do, but its not my cloud. Again, the idea was tunnel over public IP and leave the private interfaces on netmask 255.255.255.255 with firewall rules to deny all on that port except the VPN traffic...That...should work? No?

I'm just a customer A with lots of customer Bs.

i need a detailed graphic of the infrastructure to check that. not quite sure if i understood your infrastructure correctly.

you are customer A1 in a datacenter, there are other customers  A2 A3 A4 in the datacenter
you have some kind of "subcustomers" B1 B2 B3 B4 that run VM on your server
you have a public ip, A2 A3 A4 have other public IPs
controlling the traffic between these public IPs is the datacenter-managers duty.
your duty is to control the communication between your customers B1 B2 B3 B4

if i am B1 and i can telnet to my Server S1 in your virtual Hyper-V or VMware network.
and all the virtual servers are in one big subnet, without any policies in that virtual subnet
then i can surely attack B2s server from my server

if you just set up some vpn-policies, i cant access B2s server from my pc in my office
but i dont need to. i do the following:

[root@office-pc] telnet 172.20.59.100
login and stuff
[root@172.20.59.100] telnet 172.20.69.166
or
[root@172.20.59.100] just-another-stupid-hacker-tool -attack 172.20.69.166 /force

Quite simply, there are no subcustomers.
There is no dedicated server with VMs. There is The RackSpace Cloud, and we have 14 slices of it. We are the end user for all 14 slices, each comprised of 4-16GB Ram/so much allotted disk space and CPU time.

It just so happens that other slices are subnetted with ours on the private end.
Heres an example; the lines represent secure connections.

The big cloud is all one big 10.x network of multiple customers.
All servers are VMs

Network.jpg

And the idea is that everything that is connected via a line can communicate securely as if it were all on one LAN, but is secured on the private side against other RackSpace Cloud Customers.

ok, maybe i dont get this myself.
to not going to tell you something wrong, i think it would be better to contact the rackspace support.