Clone/cache a whole website?

Good Afternoon,

Is anyone aware of a piece of software which may allow you to clone/index an entire website?  A potentially useful application for this could be used for business continuity.  We're looking for a way to spider our existing site, and then serve it up elsewhere.  There are some complexities behind the scenes with our site in regards to database access reads, but truth be told, the core content, while it exists in a DB, really could be served up as "static" content.  The desirability of a solution as described could be ultra low BC/DR costs for such a product.

Is anyone aware of such a program/service?  Again, we are fully aware that you can simply copy a site to another location, but the site in question is not a simple flat site.

Thanks!
lighthousekeeperAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

akahanCommented:
I've used HTTTrack in the past; it works pretty well.
It spiders the site in the same way Google would (but it ignores files that say "don't spider me!"), and copies all the contents to the local drive.

https://www.httrack.com/

I would DEFINITELY recommend that after you make your local copy you test it before you actually need it, to makes sure you got everything.
1
Dave BaldwinFixer of ProblemsCommented:
HTTrack is great for static sites.  If the site is generated through programming a databases, you really need FTP and database access to make a working copy.  In addition, you may need to install or enable software on the server.  You can't count on simply being able to copy the site and have it work.
0
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
You may want to serve all your files from a CDN (Content Delivery Network).  You upload your site files to a cdn once, then the CDN distributes the files around their network.  If somebody in NY is viewing your site, then the files are served from an east coast server and same for somebody in CA, they would get served by files from a west coast server.  

The side benefit is if one server fails, there will be several others around the country (or world) to take over without noticing.  This site by the way I believe uses https://www.cloudflare.com/.  There are plenty of CDN's and you will want to do your own research.  As youmention, you can script your site to generate static html from the db making this a good solution.

The other issue is high availability of a database. Mirroring is the option that allows you to write to one database and simultaneously update others. https://msdn.microsoft.com/en-us/library/ms189852.aspx.  I use sql server and for what I do, this can be more costly than what I need.  My hosting service offers a continual back up and this is the route I have taken.  It has been used once and I was thankful it was there.  After a crash, it took about 30 minutes to restore 20 active DB's.   Being down for 30 or 60 minutes one time in about 8 years far exceeds the pay off for spending $1,500 per month on what I would need to mirror.

I have been through a good handful of hosting services and have had my dedicated service at https://www.liquidweb.com/ for a long time now.  They have a private cloud allowing you to quickly spin up servers.  Worth looking into.

In short, if you can produce static content, a CDN will be your best bet.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Servers

From novice to tech pro — start learning today.