Redundant server routing

My company operates a supermarket chain.  We have recently brought all of our gift card and check processing in house.  We have one server at our corporate office that performs both of these tasks.

We really want to get some redundancy in place in case there is a problem with the connection at our office or if there is a problem with the server.  The best solution was to have a redundant server at one of our stores.  We got the server, got it installed, set up the database replication, but now we've come to the hard part of the setup.

The networking part of this project is a little tricky.  The IP range of the corporate office and the IP range of the store with the redundant server are different.  Unfortunately our point of sale system only allows us to set up on IP address per tender type.  If we could enter a fail over IP in the POS then this would be easy, but we can't.  Since we can only have one IP address per tender we need to have the main and redundant servers set up as the same IP address.  

We originally thought that we would add a second router to the network at the redundant site that would have the same internal range as the range at the office and then add an additional route to all of the routers at the other stores to the redundant server with a lower weight than the route to the office.  I would then set the 2nd NIC in the redundant server to the same IP as the server at the office and route traffic accordingly if the connection at the office dropped or if the server at the office was down.  There was a problem with this design.  Everything is fine and dandy until I hook up the 2nd NIC to the 2nd router at the redundant site.  After doing that the two servers are unable to communicate.  Logically this makes sense to me.  A packet is sent from the main server ( to the redundant server ( and when the redundant server tries to send back the ACK to the main server at it just gets routed to the 2nd NIC of the redundant server.  So in the case of a failure of the server or the connection at the office, this design works, but I'm unable to replicate my data to the redundant server.

I've tried setting up a static route on the redunant server telling it to send all traffic destined for the office out of the interface with the IP and bypass the 2nd router entirely.  It didn't work.  

I've sat and thought about how to get this to work as an automatic fail over and I'm stumped.  The only way that I have figured out to get this to work would be to call someone at the store in the case of a failure and have them plug in the cable to the 2nd NIC.  This is obviously not an automated process and that's what we're trying to achieve.

Anyone have any good ideas about how we can get this to work?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

It might be possible to use some sort of load balancer to achieve this. You set your POS machines to that of the load balancer, and have the load balancer forward requests to whichever server is more appropriate.

The above load balancer can be configured as a high availability cluster, so there is no single point of failure.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
shtaffaAuthor Commented:
A load balancer is a good idea.  I'm not sure that it would work in my situation.  Our inter-store connections are maintained by our ISP.  For a load balancer to work for my intentions it couldn't reside at any of our stores or our office.  if it did and we lost the connection to that site we would lose our connectivity to our payment server entirely.

I am however going to bring this up to our ISP and see if they can set something like this up on their network.
A load balancer would also mean having two ISP's and having to maintain (pay for,...a.k.a.,...monthly bill) two Internet accounts with such ISP's at every location.

In your situation the redundancy can only be performed by the ISP that you have,...they supply your connection,..therefore the redundancy and quality of that connection is completely their jurisdiction.  The truth is that an ISP is already going to have redundancy out the wazzo inside their facility. That is what ISP's do, is their job,...and the quality of their service is how they stay in business.  The only thing that is not redundant is the "last mile" link to each facility and if you want that to be redundant,... and not have to pay two ISPs for an account,... then the existing ISP can bring in a second Line to each facility and tie it into their equipment that they put in place and the redundancy is handled by Dymanic Routing Protocols (like BGP for example) between the differnt ISP Routers.

This of course means your ISP must be providing you with an Industrial connection based on industrial technology,...rather than Home-User Technology (like CableTV or DSL)
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

shtaffaAuthor Commented:
I was hoping there would be some silver bullet to this problem and it doesn't appear that there is.  We're going to have to rethink what it is that we are trying to achieve.

Thanks to those that provided input.  You both provided viable solutions, just ones that won't work for our particular situation.
shtaffaAuthor Commented:
Both experts provided viable solutions, unfortunately neither are applicable in our company's situation.
You could also flip it around and say that the company is not applicable to the correct solution, the company has to change it's methods so that it can correctly use the correct solution.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Architecture

From novice to tech pro — start learning today.