Configuring 2 ADSL Routers on same subnet (for contingency). Problem with port forwarding from both routers to the same server.

We have 2 ADSL connections via different ISP's for resilliance.
Both are serviced by 3COM ADSL routers.
Currently one acts as the default gateway for our network - the other is connected to a Microsoft ISA server which then forwards traffic to the same subnet as the first router.
So for browsing we have a choice - default gateway or proxy server.
for public access - both public IP addresses are used.  The first router is configured to forward certain ports to respective servers.  The second router forwards traffiic for a certain port to the ISA server which then forwards it to the respective server.

I now want to do away with the ISA server and have both routers acting as gateways.  
I've modified the addressing on the second router and connected it to the LAN.  Ive then changed the 'virtual server' rules that define the port forwarding so that now the router passes traffic straight to the servers in question .

So router A, Router B pass SMTP to Server A

Server A has default gateway of Router A and now I specify secondary gateway of Router B

I can only communicate with the server from the outside world through Router A.  
If I change default gateway to Router B then I can communicate only through router B.

I think the difference when I was using ISA server is that each rule is set to make the traffic appear to have come from the ISA server (not the original client)

Is this behaviour down to the Routers?  Can I get a model of Router that will allow multiple routers to forward traffic to the same server?

Or is there a way to configure the server to be able to communicate with both Routers?