Dual ISP, Dual Firewalls Dual NICs on a Webserver - Routing Headache

We have two ISPs each providing a separate 27bit address allocation. We then subnet the 27bit network into 2 separate 28bit networks. For each ISP, one 28bit network is bound to our DMZ network, whilst the other 28bit network is for WAN connectivity.  We have two Nokia (Checkpoint) firewalls, each configured with a WAN interface in one 28bit network , another  interface configured in the other 28bit network, and a third interface configured for our LAN.

In our DMZ we have a webserver  with dual NICs. Each NIC one is bound to one of the 28bit networks on each firewall.

The problem is that currently all of our DNS records point to the ip addresses supplied by ISP2 (there is no DNS RoundRobin Load Balancing taking place) If someone visits the server via the DNS name (ISP2) everything works perfectly, however if you try to access the server via an ip address from ISP1 you get no response.
This is because the default GW of the servers is FW2, so all public internet traffic is routed via FW2 even if it came in via FW1. If I change the routing on the webserver so that FW1 is the GW everything works as you expect, except it takes the server out of service for users coming in from ISP2.

My question is does anyone know if there is a way to make a windows machine respond ONLY via the interface on which it receives a request. This would allow me to configure identical (mirrored) routing configuration on both servers, so that the system would decide which route to choose based on the interface that it received the packet.

I know this is not an ideal situation in the first place,  but I need to maintain 2 isps  and two public IP addresses (one from each isp) on each server  other suggestions on how to achive this would also be welcomed. I have not ruled out the idea of throwing extra kit at this to make it fly (proxy boxes/load balancers etc)
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

I would think your best way to solve this would be to use a device like f5. you can then have both IP aggragate to 1 box to control inbound and out bound this can also control multi connections on the inside and keep them seperate
nickhillsAuthor Commented:
budget is limited, so i would need to replicate any functionality like that on a linux box.

As i understand it, the f5 will act as a proxy to relay requests for both external ip addresses to the server that is 'behind' the f5. Also limited is available IP addresses (in reality there is more than 1 webserver, but to illustrate the issue i only mentioned one)
There is a further complication, in that on at least some of the webservers proxying ip addressed is not possible. (technical limitations)

I am starting to suspect that what i am asking can not be achieved, but if anyone knows if there is a way to influence the routing table on a per interface basis, i may have to accept that there is no way to do as i have been asked (told)

thanks for your ideas, but the requirements are (currently) set in stone.
It can be done with linux  + policy routing.

The web servers will have a 10.x.x.x ip, and the linux router/proxy will handle the sessions to and from the box for both external ip's, Natting or reverse-proxying them

Using the state full iptables firewall, and packet marking to select the between 2 routing tables to use the correct GW/ISP for packets flowing out to WWW.

The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

nickhillsAuthor Commented:

thats a bit warmer - what is packet marking? (a google serach throws up results about packet tagging in relation to QOS, but nothing that seems to cover this scenario), where can i find info on this relevent to my problem?

the only problem i see, is that i don't want to change the ip addresses on the webservers, they need to stay the same (vender restrictions)

i think this is al leading to the conclusion that this can't be done without a 'proxy',  and i am asking the impossible.

on reading my original post i realised my last section was confusing and incorrect, it should have read:
(amendments in CAPS)
"I have not ruled out the idea of HAVING To throw extra kit at this to make it fly (proxy boxes/load balancers etc) BUT I AM REALLY LOOKING FOR SUGGESTIONS THAT AVOID THIS"
sorry for any confusion.
in iptables there is Firewall marks, and Connection marks.   FW mark is a mark added on a packet and lost with the packet, Connection mark is more longterm and can stay for the the duration of a tcp session.

       This  module matches the netfilter mark field associated with a connec
       tion (which can be set using the CONNMARK target below).

       This module matches the netfilter mark field associated with  a  packet
       (which can be set using the MARK target below).

I am planning to setup a very similar setup with dual providers.
Back to your problem.

Why do you need both's ip's configured on the Servers ?
Windows is dumb if it gets to this kind of routing and 2ip's will be a problem.
Why won't one internal(on the box) and 2external(Visible external) ip's work ?

nickhillsAuthor Commented:
the reason i am trying to keep the IPs on the servers stems from the fact that some of our webservers pull data from systems both inside and outside our lan. Originally we had this configured for our servers via our old ISP, and some IPs have been hardcoded (i know, i know!) on the backend servers. What we are trying to achive is present the webservers via our new second ISP without making any (many) changes to the current setup (yet). Unfortunately, the backend connections are a combination of RMI, SOAP and SQL so http proxying is not going to work for these systems.

the plan is (was) to get the webservers responding on the new IPs, then change the A records to point to the new addresses, and then setup load balacing via rrdns, and pound.
Ok, if the server only has one IP, the OLD_ip it could also work.  I think.

Here is how i understand your setup.

Internal_backend <-> Server_oldIP  <-> Linux_Nat_Loadbalance
                                              <-> FW1+FW1 to Provider1 + Provider2

For systems inside_backend they will only see the one IP,  outside systems_backend will need to handle both ip's from the server if you want to use both ISP's, if not it will also work but only use one ISP for those connections.

Thus the plan.   For outgoing connections (Originated from the server) only use the old IP, to keep everyone happy.   But for incoming connections to the old and new server external ip, send them to the server old_ip, using NAT + Policy rouing in the case where someone connects to the new IP to send it to the server old_ip, and then reply's backout the new_ispGW using the new_server IP.

Summary:  1 Ip on server - Valid old ip, routed in and out old_isp
2nd IP handled by Linux, nats to old_ip in and out and uses new_isp

nickhillsAuthor Commented:
thanks for your advice diepes.

fortunately, however, i have got my way! I am now assigning private IP's in the dmz and performing NAT for the public external IP addresses. The load balancing is being provided by the native ISP load balancing built into the IP390.

This seems to cover all of my original problems without the use of extra equipment. I guess the outcome of this topic is that windows routing is not very clever, and there is no 'hack' to resolve this shortcoming.

if you want to avoid this problem in the future - consider Linux web servers!

thanks all!
how did you overcome your one requirement

"the only problem i see, is that i don't want to change the ip addresses on the webservers, they need to stay the same (vender restrictions)"
nickhillsAuthor Commented:
i managed to convince all parties that we can change the webservers to a 10.10.10.x address and use NAT to present the original IP address with no ill effects. Now i have a green light for the concept, i am just running through the implementation. I now have to prove that we can do this without any vender intervention before we apply the config.
It's mainly red tape, but i have dragged people round to my way of thinking.

This solution means we can have differnt ISP assigned IP addresses allocated on each firewall from both ISPs. Then i can nat the 10.10.10.x IPs onto the ISP range on both firewalls. This means that i can present the IPs from the new ISP as the advertised A records, and then use the Old IPs for backend comms. The IP390 with NGX R65 allows me to set the ISPs up in either load balanced or fault tolerant config for outgoing packets, whilst allowing backend servers to access the old addresses from both internal and external sides!

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial

I must have a look again at the Checkpoint software.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.