We help IT Professionals succeed at work.

routing between two internal subnets each having their own WAN

liam137
liam137 used Ask the Experts™
on
network layout
Hello all.  Thanks in advance for your time.

I am having issues connecting my two internal networks.

A little detail about the above image:

router0 and router1 have separate WAN. The need for two networks is so that when I tinker with things I won't affect other users in the house.

router0 runs pfsense with a gateway 192.168.2.4 specified on the LAN interface and a static route to 192.168.2.0/24 with 192.168.2.4 as its gateway.

router1 runs dd-wrt with a static route to 192.168.1.0/24 with a gateway of 192.168.2.4.

router2 runs dd-wrt with no static routes specified, gateway provided by DHCP/static lease. It is configured in router mode with nat, dhcp, spi, dnsmasq, etc. disabled.  I am unsure if I should manually configure the WAN interface without specifying a gateway.

There is also another router (ap0) that acts as an access point and has no WAN connection.

I had an entry in iptables for router2 to accept/forward all packets but I ran into problems with competing DHCP servers. So I removed that entry.  It also didn't seem to change the problem.

I have a solaris box (fs0) running samba that needs to be accessed by all - it resides on 192.168.1.0/24. Naturally, all computers from that subnet have no trouble connecting. When attempting to connect to the server from windows machines on 192.168.2.0/24 all attempts fail. I am also unable to connect to www or proxy running on the same server. After I ping into 192.168.1.0, the client is then able to access the share, www, and proxy. But nothing seems to connect until I ping.

I can ping and traceroute into 192.168.1.0 from 192.168.2.0 but am unable to ping or traceroute into 192.168.2.0 from 192.168.1.0. However, on the solaris box I am able to ping and traceroute into 192.168.2.0 without any problems. There is no static route specified on this box. It follows the route from router0 just fine.

I don't know if it will help or not but when I took the WAN if down on router0 (pfsense) I noticed what appeared to be dropped packets/connections coming from the internal network but showing they were on the WAN interface.  Also, when I traceroute out to the internet I see hops with internal IP address on the same /8 as me but not the same /24 if it makes any difference.

Any thoughts on what I may be doing wrong?  Any guidance or advice is most appreciated.

TIA
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
I don't see how Router1 can have two gateways.
I believe there has to be a default or internet gateway entered or gotten via DHCP from the cable modem such as 10.0.0.1.
Yet, you say that 192.158.2.4 is the gateway??
That's odd.  It implies a gateway on the LAN side.
The only "gateway" on the LAN side would be for DHCP service to the LAN devices.
Generally that would be 192.168.2.2 - the router's own LAN address.

But maybe I'm thrown by that line from router0 to the cable modem??  Is that a mistake?
I'm thinking that the IP address on the "WAN NIC" in  router0 would be something like 10.0.0.2 to correspond with my fictitious IP for the cable modem.

I see the LAN devices connecting to the internet source possibly two ways so let's trace that path:

192.168.1.0 subnet hits router0 192.168.1.2
and
192.168.1.0 subnet hits router2 192.168.1.4
They can't both be internet gateways without some routing implementation.
First off, what is the IP address for the gateway for the devices on this subnet?
Second, is there a higher metric default route to the other apparent gateway?
Otherwise, how do you switch the LAN from one path to the other?

It would help if you would specify which boxes have DHCP service running and which do not.

You can generally ping "upstream" through a router from one private subnet to another but you can't generally ping "downstream" through a router from one private subnet to another.  That's likely why 192.168.1.0 can "see" 192.168.2.0 and not vice versa.

I have attached a couple of papers that might help.  It's not clear to me what your objective is yet.  I would start with a simpler network topology and go from there according to one stated need and one change at a time perhaps.
2011-08-26-How-Subnets-Work-in-P.pdf
Multiple-Subnets.pdf

Author

Commented:
fmarshall, thank you for your reply and the pdfs as they were educational.  I understand your time is at a premium and appreciate you giving a little bit to me.

First let me address your question as to what is my goal.  My goal is to have two separate networks each with separate internet access.  The purpose is so that I can make changes to my own network, systems, routers, etc. and not affect the other users in the house.  I then would like to have them linked so that they can communicate between one another.

As for the gateways, let me explain:

The cable modem had 4 LAN ports.  It was originally configured as a gateway and wireless access point.  I have disabled the wireless and bridged the WAN/LAN ports.  With that, up to 4 devices can pull an IP from my ISP.  The IPs provided are routable.

router0 receives its WAN IP via DHCP through the cable modem.  A gateway is specified in pfsense that is used by the static route to 192.168.2.0.  It is not marked as the default gateway and is in-use only with the static route.  This box runs DHCP for 192.168.1.0/24.

router1 also receives its WAN via DHCP through the cable modem.  With DD-WRT, there is a static route specified to 192.168.1.0 with a gateway of 192.168.2.4 (router2).  This router runs DHCP for 192.168.2.0/24.

router2 just sits between the two networks.  It is set as just a plain router that should forward packets between the two networks.

For devices on 192.168.1.0/24 each has a default gateway of 192.168.1.2.

For devices on 192.168.2.0/24 each has a default gateway of 192.169.2.2.

router2 behaves the same if gateways are specified on the respective interface or not.  There are options to specify gateways for both the WAN and LAN interfaces.

As for the metric, I believe on router1 it is 0 and as for the pfsense box I see not place to specify the metric.

I understand what you're talking about into where one can ping.  But it's reversed of what you describe.  I can ping upstream through router2 from 192.168.2.0 but am unable to ping downstream from 192.168.1.0.  I'm not sure if it's a windows thing because tracert on windows just shows "* * * Request timed out." while traceroute on solaris shows the hops just fine.

The thing is, this setup works - kinda.  I can connect to fs0 from 192.168.2.0 after pinging or tracert to fs0 or any other device accessible on 192.168.1.0.  As soon as the first reply is received or the first hop is traversed, the share pops open and is then accessible.

I'm just confused, I guess.  To me, the setup seems simple.  I have two networks and I wish to connect them.  Perhaps I am misunderstanding the complexities involved - which is a distinct possibility to be honest.

Again, thank you and I await any further guidance anyone may have.
Can you confirm that router2 is/is not doing NAT?  The way it's described it could be either it seems.

I can ping upstream through router2 from 192.168.2.0 but am unable to ping downstream from 192.168.1.0.  I'm not sure if it's a windows thing because tracert on windows just shows "* * * Request timed out." while traceroute on solaris shows the hops just fine.

This still confuses me.
The router2 WAN is the "upstream" side and is on the 192.168.2.0 subnet.
That points to the internet which is also what I'd call "upstream".
So these are consistent with respect to that terminology.
The router2 LAN is the "downstream" side and is on the 192.168.1.0 subnet.

You say: The devices on 192.168.1.0 use 192.168.1.2 as their gateway.
So, if a device pings and address in 192.168.2.0, those packets will be forwarded to 192.168.1.2.  That router knows nothing about 192.168.2.0 and should drop the packets.

Perhaps you can re-write this quote because I have no idea where you are pinging From and where you are pinging To with those statements.

As I see it, you would not be able to ping from 192.168.1.0 to 192.168.2.0 because there appears to be no path available.  Unless .... router2 is somehow responding to packets destined for 192.168.2.0 even though they aren't destined for router2.

As I see it, you would not be able to ping from 192.168.2.0 to 192.168.1.0 because that's "downstream" through router2.  But, if router 2 is not set to NAT then that may make a difference.  If it's set to NAT then I don't think you can access the LAN from the WAN when there are private addresses.  But mileage may vary with different boxes.  With NAT at least, router2 is set up exactly as in the paper to disallow connection into 192.168.1.0.

So, there are still some unknowns for me here and it would be good to clear them up.

I hope this is helping.
Which devices are you using to route from 192.168.2.0 to 192.168.1.0?

I think you're going to have to setup some static routes on a couple of routers.

http://www.dd-wrt.com/wiki/index.php/Linking_Subnets_with_Static_Routes

Author

Commented:
@ inbox788
Thanks for your reply.  The two routers, router0 & router1 do have static routes specified with router2 as the gateway for that route.  As with the link you provided, most people have a cable_modem->router->router scenario or are bridging via wifi.  WHich would work if everything shared one WAN connection.  Thank you, though.

@ fmarshall
Thank you, too, for getting back so quickly.  And yes, you are helping.  My nipples explode with delight!  I've been trolling the interwebs for weeks and have yet to find any situations that match mine - which is about right.  Mother always said I had to do things the difficult way.

I was confused as to upstream/downstream.  So you were correct in your original statement.

NAT is not enabled on router2.  There are options for gateway, router with NAT enabled, disable respectively.

The gateway 192.168.1.0 (192.168.1.2) does forward packets to 192.168.1.4 as it has a route specified to 192.168.2.0 via 192.168.1.4.  I can ping and traceroute from that box into the other network without problems.

The way I understand it is that when a request for 192.168.2.0 comes into the gateway on 192.168.1.0 that box then forwards the request to router2 (192.168.1.4) which then sees it has a route to 192.168.2.0 via its WAN interface and then forward the packets onto that network.  Should a gateway be specified for the LAN and WAN of router2?  Such as for the WAN on 192.169.2.0 specify a gateway of 192.168.2.2 and for the LAN on 192.168.1.0 specify a gateway of 192.168.1.2.

If I may digress for a moment and ask about the metric.  Should the static routes have a metric of 0 or 1?

So here's the communication or lack thereof:

router0(192.168.1.2) - (from the console) this box can ping and traceroute into 192.168.2.0.  traceroute shows the hop at router2(192.168.1.4).

router1(192.168.2.2) - (from the shell) this router can ping and traceroute into 192.168.1.0.  traceroute shows the hop at router2(192.168.2.4).

router2(wan:192.168.2.4 - lan:192.168.1.4) - (from the shell) this router can ping and traceroute into 192.168.2.0.  It can ping into 192.168.1.0 and tracroute shows 1 hop for everything but the gateway for 192.168.1.0.  When traceroute 192.168.1.2 is executed the command just hangs.  I'm wondering if this may be the issue.

from a windows box on 192.168.2.0: trying to access a SMB share on fs0 via \\192.168.1.10 I receive "no network provider accepted the given network path".  If I attempt to connect to http://192.168.1.10 the connection hangs.  If I execute a ping or traceroute to 192.168.1.10 (both work fine) then I am able to access the SMB share via \\192.168.1.10 and via http://192.168.1.10.

from fs0(solaris)(192.168.1.10) - (from the shell) this box can ping and traceroute to everything.

Am I, perhaps, using the wrong hardware for what I need?  Should I look at replacing router2 with a different appliance?
Do you have static routes on router2? Again, which devices are you using to route from 192.168.2.0 to 192.168.1.0? It looks like you're trying to use router2, but you've got it set up as gateway, not a router. (the link I provided has an example that isn't your 2 WAN, but focus on the static link portion)

Author

Commented:
inbox788, thanks for checking-back.  I've double checked all the settings.  router0 & router1 have static routes to one another with router2 as the gateway for the respective static route.  As I understand the dd-wrt documentation setting the 'operating mode' to 'router' disables nat which is how router2 is set.

Before moving everything around, I was using the scenario presented on that page and it worked without problems.  Not so much now that I changed it up.
I forgot to mention:

When you say "has a route" it would be good to differentiate between what you have entered manually and what is showing in the routing table.  So, you might review your posts and illuminate in this regard.  That will make it easier to understand.
router2(wan:192.168.2.4 - lan:192.168.1.4) - (from the shell)
 .......snip.................
When traceroute 192.168.1.2 is executed the command just hangs.  I'm wondering if this may be the issue.

Well, let's see:  You are trying to traceroute out through the LAN interface of a router.  I've never tried to do that.  Everything I've ever seen does those tests through the WAN interface.  And there is no path via the WAN interface.  So, I'm not too surprised and not alarmed by this.

One thing I can say that's pertinent:
These commodity devices (and maybe more expensive ones as well, I don't know) do *know* the difference between the LAN side and the WAN side **even when you don't care!** as in a router application.
For example, I have found that Cisco/Linksys RV042s cascaded together as routers MUST have the WAN side pointing toward the internet or they don't work.  This is in "router" mode where I could care less which interface side is which.  But they do.
This observation just enforces my comment above about where pings and traceroutes go out of the box.
I've also seen cases where manually entered routes do nothing when it seems logical (like your traceroute) that they would "do the logical thing".  Nope.  Your logic and the programmers logic have no common ground necessarily.  And, it's possible that either one of us could be wrong in our logic from time to time.
The gateway 192.168.1.0 (192.168.1.2) does forward packets to 192.168.1.4 as it has a route specified to 192.168.2.0 via 192.168.1.4.  I can ping and traceroute from that box into the other network without problems

OK.  Thanks. Understood.

The way I understand it is that when a request for 192.168.2.0 comes into the gateway on 192.168.1.0 that box then forwards the request to router2 (192.168.1.4) which then sees it has a route to 192.168.2.0 via its WAN interface and then forward the packets onto that network.

Yes.  I would say "packet" instead of "request" as you did at the end.  But then, many packets *are* requests of one type or another.  So you have the right idea.

Should a gateway be specified for the LAN and WAN of router2?  Such as for the WAN on 192.169.2.0 specify a gateway of 192.168.2.2 and for the LAN on 192.168.1.0 specify a gateway of 192.168.1.2.

This gets a little involved with the interface design maybe.
Normally, the WAN side will want to have a gateway entered because it's pointing "upstream" and needs to know where to send packets.
Normally, the LAN side *is* its own gateway / i.e. the gateway for the LAN.  So often there is no entry for a gateway address here.  The expectation is that packets coming "down" will just get dumped on the wire or will follow static manually-entered routes on the LAN which may forward to other subnets, etc.  In that sense, the manual routes can establish "gateways" for particular address ranges.
Well, except there will be a gateway address entry in the LAN-side DHCP setup which you aren't using anyway.  
So, I don't think there is a place to enter a gateway address on the LAN side in this case.

It would be reasonable for router2 WAN to have 192.168..2.2 as the gateway.  But maybe this is moot or redundant in some sense as no packets should arrive at its LAN 192.168.1.4 that are destined for any other place than 192.168.2.0, right?  Those would all be dealt with in principle by being destined to the 192.168.1.0 gateway 192.168.1.2.  I would say: It can't hurt and it may help in some strange way that we needn't understand - so enter 192.168.2.2 as the WAN gateway for router2.

from a windows box on 192.168.2.0: trying to access a SMB share on fs0 via \\192.168.1.10 I receive "no network provider accepted the given network path".  If I attempt to connect to http://192.168.1.10 the connection hangs.  If I execute a ping or traceroute to 192.168.1.10 (both work fine) then I am able to access the SMB share via \\192.168.1.10 and via http://192.168.1.10
.

This sounds like a credentials sort of issue.  I have no idea.

Earlier you said:

....but am unable to ping or traceroute into 192.168.2.0 from 192.168.1.0. However, on the solaris box I am able to ping and traceroute into 192.168.2.0 without any problems.

This isn't consistent. Either you can ping or traceroute in general from 192.168.1.0 to 192.168.2.0 or you can't.  The Solaris box is part of 192.168.1.0 ... so.....
Perhaps you mean that you can't ping from anything other than the Solaris box into 192.168.2.0?
If so, this would imply that the Solaris box has its own route to 192.168.2.0 pointing at 192.168.1.4 i.e. router2.  AND router0 isn't doing its job of forwarding the same to the same.
You could always add a route to a PC on that subnet to see if it makes a difference.

Author

Commented:
Thanks for getting back to me.

Everything from 192.168.1.0 can send packets into 192.168.2.0.  The problems lies with packets going from 192.168.2.0 into 192.168.1.0.

I'm changing-out a blade server tomorrow and will have a spare box for testing.  I'm going to throw pfsense on it and see if I can get it to route between the two networks.  If that doesn't work, I'll try a linux distro.  If that doesn't work, I'll dig-up another router to use and swap that in for the pfsense box at router0 to eliminate a pfsense problem.

Thanks for your time, gentlemen.  I'll check-in if I make any progress.

If I may, how would you go about connecting these networks?  Is there a simpler, better solution?

Author

Commented:
Well, I've made it all work.  I think fmarshall hit the nail on the head regarding commodity (see cheap) hardware still knowing the wan/lan sides.

I installed linux on a box, enabled ip forwarding and communication between subnets looked okay from a command-line stand point.  Shares were browsable but with horrible lag.

I then threw another NIC into the pfsense box, set firewall rules and now everything works perfectly.  Though not the ideal situation, it does five me fine control over what transmits between the networks.

Thanks again for your guidance it is most appreciated.

Commented:
Using a LINKSYS WRT54GL loaded with DD-WRT I was able to get a route between two subnets working, however I ran into the same issue as you.  I'm able ping one way, but not the other, but can view shares from both networks both ways through it....

I'm sure there's got to be a way to get it working with DD-WRT being it's linux based, I'm just not knowledgeable enough with Linux networking to do it.

I'm going to link my thread here for others to find possible useful info with regards to this type of situation.
Here is the tread.