Web page Failover and Failback with AWS Route53 and S3: Browser DNS cache causing issues?

We have our corporate public website hosted on an internal server. We want to use Amazon Route 53, and the Health Checks feature to monitor this website, and automatically redirect visitors to an "under maintenance" page (hosted in an S3 bucket) in the event that the primary site goes down.

I have everything working like this:
- Route53 PRIMARY record: mydomain.com --> [IP Address of internal Web Server]
- Route53 SECONDARY record: mydomain.com --> [ALIAS for S3 bucket mydomain.com] --> S3 Web Redirect for all requests --> maintenance.mydomain.com (S3 bucket)

This all works great. If the primary goes down, visitors are redirected to maintenance.mydomain.com.

However, the problem is that browsers seem to be caching the redirect and maintenance page, so even once Route53 has switched back to the PRIMARY record, if the visitor tries to go back mydomain.com, they go straight back to the maintenance.mydomain.com page. Even closing and opening the browser doesn't help. Only manually clearing the browser cache, or switching to a different browser seems to help.

Any idea on how I could change anything to get the 'failback' working better?
SWCBTechServicesAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Anthony GarciaDevops StaffCommented:
Unfortunately that is a downside of doing  a down page that way. You may try adding a no-cache header to your error pages, but I think a better solution is to use Cloudfront.

If you are using Cloudfront you can set custom error pages which will show instantly without waiting for any ttl to expire or any of the associated wait times with DNS changes.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages.html
0
SWCBTechServicesAuthor Commented:
Thanks for the input. I'd have to start paying for all my traffic to go through CloudFront then wouldn't I?

I've tried the no-cache headers, but I don't think the problem is with cached version of the actual content, but with a cache of the DNS resolution of my domain name.
0
Shalom CarmelCTOCommented:
What is the TTL on the DNS record?
Whenever you want failover to work, you must have really low TTL, preferably 30 seconds or less.
And yes, you are going to pay for it. Failover costs, one way or another.
0
Top Threats of Q1 & How to Defend Against Them

WEBINAR: Join WatchGuard CTO and our Threat Research Team on Aug. 2nd to hear the findings from our Q1 Internet Security Report! Learn more about the top threats detected in the first quarter and how you can defend your business against them!

SWCBTechServicesAuthor Commented:
TTL is 60 seconds. nslookup confirms the DNS record is failing over and failing back very quickly. It appears that IE is caching the DNS resolution.
0
Anthony GarciaDevops StaffCommented:
Yes unfortunately you would need to have your traffic go through cloudfront. The browser caching the dns entries is not something I have found a way around. Another option is that if you use a load balancer in front you your application you could have a script monitoring the number of healthy hosts on the load balancer. If it goes to 0 you can use the aws api to add a small instance with you maintenance page set up on it. This also eliminates any changes to DNS on your part.
0
SWCBTechServicesAuthor Commented:
Thanks everybody. After some further testing it looks like the failover/failback problem is really only bad with IE and Edge. They each seem to cache the DNS resolution for about 30 minutes.

Chrome, Safari, and Firefox seem to cache for only a minute or less, and failover/failback quickly.

Since IE/Edge makes up only about 7% of the market, and Chrome/Safari/FF make up 83%, I think we will just keep using the Route53 DNS failover technique.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SWCBTechServicesAuthor Commented:
In the end, each browser handles DNS caching differently.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
DNS

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.