Solved

server failure recovery

Posted on 2014-12-16
15
106 Views
Last Modified: 2015-01-02
I am trying to setup a server failure recovery strategy.

How does it sound?

1. I will use multiple A records pointing the same address to 2 different servers (see the discussion here: http://webmasters.stackexchange.com/questions/10927/using-multiple-a-records-for-my-domain-do-web-browsers-ever-try-more-than-one)
2. At normal conditions the server B forwards the traffic to server A programmatically to avoid any problem of multi A records
3. Server B checks if Server A runs good. If B sees that A is not running then stops forwarding the traffic.
4. Server B also starts triggering  a page at server A to stop it to avoid temporary recoveries (eg. every 5 seconds server B requests the page: IP_of_server_A/stoprunning.php)
5. Thanks to multi A records the browser will forward the requests to B when A is not running.
0
Comment
Question by:myyis
  • 6
  • 5
  • 2
  • +1
15 Comments
 
LVL 69

Expert Comment

by:Qlemo
ID: 40504415
Why do you want to prevent fallback to A? You are not concerned about sync, I suppose, as B isn't in sync anyway.
Other than that, it seems to be a working concept.
0
 

Author Comment

by:myyis
ID: 40504485
There is a continuous DB insert transactions. I will use one way sync. That means the A will write to B continuously. Therefore when A is down I want that no body inserts data. When A is online and stable again  I will write the new DB records to A.

How does that sound?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40521658
Where is the DB insert uniqueness handled? In PHP or in database?
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 

Author Comment

by:myyis
ID: 40521946
Sorry I don't understand what you mean DB insert uniqueness? Can you give an example?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40521961
How do you know that only one server can be up and running?
Normally you can have 2 or 200 inserting new records in database (how do you think facebook works)
0
 

Author Comment

by:myyis
ID: 40522039
1. In normal conditions 2 servers are up (multi A records),  all requests to backup server(B) are forwarded  to main server (A). So only A is receives requests.
2. Every insert query at A is also triggered at B
3. Always checks the inserted ID  after the transaction at B, it should be equal to the inserted ID of related transaction at A
 Therefore both servers always have the same records.

When server A  is down,
1.  it is made permanently down to avoid temporary recoveries (explained above)
2. Forwarding from B to A is stopped and the requests are accepted at B
0
 
LVL 14

Expert Comment

by:kenfcamp
ID: 40522202
What happens when server B goes down?
0
 

Author Comment

by:myyis
ID: 40522384
For DB consistency:
All the insert (also update and delete) queries generated at A  is written to a table. If they are successfully executed at B the related query is deleted from the table. If can not be successful that means that B is down, the row will not be deleted.  When B is recovered it will execute the queries that are at the table (at the queue).

At the user site: When B is down the browser will check for the working server (A) thanks to multi A records. (http://webmasters.stackexchange.com/questions/10927/using-multiple-a-records-for-my-domain-do-web-browsers-ever-try-more-than-one))
0
 
LVL 69

Assisted Solution

by:Qlemo
Qlemo earned 250 total points
ID: 40522412
Sounds good, but recovery from B to A needs some effort. But that could be eased if you use the same "logging table" approach on B.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40522616
Could it be mirrored database that fails over and two or 10 decoupled webservers approach highly available replication master of the moment ? Then you just need failover script for DB and wise config for webserver
0
 

Author Comment

by:myyis
ID: 40522671
Hi gheist,
If you're suggesting replication features (master-slave) of DB I can't do that because I am on shared environment and I don't have root access to the server.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40522702
I am just telling that you can use primitive DNS load balancing to balance between two apache app servers. There is no need to complicate the picture with kill scripts ensuring one application is running.

They both approach database you have at the same time

Problem solved, no radical customization. One server goes down, all user browsers handle failover until Monday you get back to work. Maybe add 3rd apacce server for christmas holiday, but "works for me" like this.
0
 

Author Comment

by:myyis
ID: 40522728
Hi Gheist
I have checked this link (https://www.digitalocean.com/community/tutorials/how-to-configure-dns-round-robin-load-balancing-for-high-availability)

At the end of the article it mentions that I have to have  DB replication. But I cannot have replication feature, that's why I am thinking a more complicated solution.
0
 
LVL 62

Accepted Solution

by:
gheist earned 250 total points
ID: 40522742
You just stated: There is no way you can make database highly available (in your position)

2 identical web servers will serve highly available web interface (if DB goes down both applications are down)
You can add some caching etc to survive DB restart, but that is all you can have.
For testing - set up both servers, just gracefully stop one apache and check that you can still browse your site.
rsync is your friend to synchronize content.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Never store passwords in plain text or just their hash: it seems a no-brainier, but there are still plenty of people doing that. I present the why and how on this subject, offering my own real life solution that you can implement right away, bringin…
Developer portfolios can be a bit of an enigma—how do you present yourself to employers without burying them in lines of code?  A modern portfolio is more than just work samples, it’s also a statement of how you work.
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question