2 Inbound ISPs how to setup some sort of routing

To answer part two of the question first.

Network traffic always leaves (answer on TCP) through the interface the inbound request come in on.
(This also applies to [NATTED]/FORWARDED packets...)
So far so good. UDP might be more difficult you might need to bind to specific interfaces.

If the link to provider A goes down you need to tell the world they need a different address.
If DNS is involved, please use a RR (DNS resource record) with a short timeout, and update your zone file when things change
(remove a reference to an interface that goes down).During the short period that the updates are not spread yet (i.e.  remote systems ask the name again, due to chaed record that were removed when the timeout occurred). connections may fail.

Outgoing traffic might be easier if you have some routing protocol installed, OSPF is a good candidate as it can assign different cost to interfaces that are up or down, convergence of the routing is fast. you might loose one or maybe two  ping packets.

Your system should act with a routing daemon on the OSPF announcements of the router(s). (quagga http://quagga.net/) is a good candidate.
You also might need some extra routes for the nets of the ISP's themselves (in your example: 1.0.0.0/8 to ISP A's router and
2.0.0.0/8 ti ISP B' modem.) As your ISP might not allow certain traffic f.e. only POP from withing the providers network.

Hi,

I have a few questions.

1. What tipe of router do you have?
2. Is links to ISP A and B on static IP's?
3. See if you can setup a route on the Router with a metric.

EG on cisco

ip route 0.0.0.0 0.0.0.0 1.1.1.1    (Default gateway )
ip route 0.0.0.0 0.0.0.0 2.2.2.2 metric 100 (Default gateway if interface of 1.1.1.1 is down )

Please let me know.

l;)

Link A is static, Link B is Dynamic (But will hold the IP long enough for VOIP registration and transfer)
NETGEAR ProSafe VPN Firewall FVX538 Dual WAN  but I want to use the Load Balancing (for the rest of my network) and not failover WAN1->WAN2
Since the traffic is SIP it will always be UDP.
No need to update DNS as the SIP registration packet will update our VOIP providers servers as to the new IP every 30 seconds.

So on the router I bind eth0 to WAN 1 and eth1 to WAN 2.

Now I just need a way to say ok bind all traffic to eth0 until WAN1 goes down, once WAN1 is down leave eth0 for local traffic (ie: voip phones) then open eth1 for outbound internet traffic and corresponding inbound traffic.

There is a way in webmin to run a command when a ping fails, so I was thinking disable eth0 and enable eth1 if ping fails and vice versa for other way, but I will not be able to keep the phones registered as they are registering to eth0, so maybe could I change the default gw or something like that?

Add 2 routes to your server.
1. Default gw

2 other route eg "route add -net 10.0.0.0 netmask 255.0.0.0 gw 2.2.2.2 metric 100"

When default gateway is not available the metric 100 route will be used.

If you can see the line up/down on your gateway system (with both eth0&eth1)
If it is either WAN1 or WAN2 then you need a route with cost (=metric)  like dextermain suggests.

If your system cannot see if the link goes down or not (like when you have a router between your gateway system with both links attached)
then you need to be told that the link is up or down. (i.e a router update protocol, preferred is OSPF in this case because it converges fast.
RIP is widely used but it can take up to two minutes to converge  => notice a link goin down and comunicate it to your central system).

Load balancing is more tricky, you need to be able to select a route based on remaining capacity.
This article describes howto do loadbalancing using iptables. http://www.sysresccd.org/Sysresccd-networking_en_Iptables-and-netfilter-load-balancing-using-connmark

Unfortunately both IPs are on internal network and both ISP links are on the same router ie: same gateway IP address.  Load balancing is being handled by the router and if WAN 1 goes down the router will stop using it, but will the linux box still send out eth0 even though it cannot communicate?  Can I add metrics to the actual interfaces?  Lets say WAN 1 goes down will the linux box still communicate to local LAN through eth0 and switch to eth1 since it cannot reach second hop for internet traffic?  If so would there be times that if udp traffic comes in eth0 would it go out eth1 and not eth0?

Now let's draw a picture.....
Does your network look like this??? (see attached file)

If so all handling needs to be done on the LEFT router...
You can set the routes on the left linux system but the
routing dissision for ISP1/ISP2 will be made by the left one.

Now the attached file did upload...
tmp-ee.png

Hi

What type of router do you have?

NETGEAR ProSafe VPN Firewall FVX538 Dual WAN

Yes that is a good picture.  The routing is handling it, but whats to stop stuff coming in 1.1.1.1 to eth0 (defined by router) going out from eth1 to 2.2.2.2 (bound by router)?  and if 1.1.1.1 goes down will the local UDP traffic going to 10.10.9.1 (eth0) go out eth1?  Those are the problems I am trying to stop

Here I am going to make this simpler.  Let's forget the router and the IP addresses.

Things I want to accomplish:
1. UDP traffic coming in eth0 goes out eth0
2. UDP traffic coming in eth1 goes out eth1
3. a ping to the first hop going out eth0 fails route all outbound traffic through eth1 and leave eth0 open for local traffic in and out.
4. a ping to the first hop going out eth1 fails disable eth1 and check every 5 minutes or so if it is back up.

The NAT& CONNECTION tracking should take care of pt. 1&2 (If answer are getting returned quick enough < 1 minute)
Ping normaly follows routing rules so that would complicate things.

How about the following:
Your concerns are failure of an interface...

Why not bundle them into a bridge
br0 uses (eth0 & eth1)
Then use br0:0 for one interface and br0:1 fo the other interface, this should give you your required failover, without the hassle of routing transformations.

if either interface fails then the other takes over. (transparent for upper layers).

nocci if I use your way, will traffic slip out of eth1 when eth0 is still up?

and I am concerned with failing ISP link so eth0 will still show as up when ISP A is down.  and vice versa with eth1 and ISP B.

The interface will only go down if the cable gets unplugged or the equipment ont the other end of the cable goes powerdown.

With using the bridge on two ethernet adapters you ensure that a single interface failure will never bring your system down (the router is still a SPOF though).

All routing desissions need to be taken in the NETGEAR router.
You can still use different source addresses to help routing (if source routing works) by using alias addresses on the linux box (br0:0 & br0:1).

There will never be a physical link between ISPA direct to eth0 unless you plug the cable into your linux box, then link down will mean interface down.
(BTW is that an option?, linux has a more than capable firewall on board, has a pretty good routing engine .....?)

And why should eth0 do down if ISP1 goes down?

To answer yoru last question I figured it would be best to not send traffic out it.

To answer your question about using linux as a router/firewall...it may be an idea for the future, but we are switching over to VOIP this thursday, would just like to get this part working first.  Plus the computer that is hosting the VOIP is a couple years old and no RAID or any kind of redudancy (boss is cheap).

Yes router is SPOF, but we limited budget, but then again how often do routers go down, and would probably take me 10 minutes to switch it out with our onsite cold spare.

Ok so if I use the "bridge" erthernet will UDP traffic coming in through br0:0  be guaranteed to go out br0:0 and same with br0:1 in and out.

To answer this one too:

To answer yoru last question I figured it would be best to not send traffic out it.

The problem might be that if you do ifconfig eth0 down on a system where you bind you application to an address
that the application will get an exception as its socket get closed. (if down -> ip address removal => specific bound sockets get closed).
This might buy you more trouble than it save you.
(But this depends on application configuration.)