Link to home
Start Free TrialLog in
Avatar of Christopher Baxter
Christopher Baxter

asked on

Using wget to download sites daily

I was tasked to download websites that we host and its entirety and have them backed up daily. But I think I'm in way over my head with WGET

1) How do I download/get all website information from several sites
2) Save them to a directory on our network in .zip using today's date as the file name.
3) Run this daily in a script

Thanks,
Avatar of Apurva Bhandari
Apurva Bhandari
Flag of India image

Write a script to automate full task after preparing try 2-3 times it is working or not.
for multiple website you can use for loop in script with wget
make folder give permission after that tar it and save tar with today's date with date command format and then copy over the network through scp to the location
now after successful running script put it in crontab with particular time when you required . it will generate data on daily basis with date in tar format
Avatar of noci
noci

This is only doable is the websites are static HTML and don't contain any PHP, or PERL or other scripting parts or are based on some database being used.
Also there may be hidden files (like .htaccess) that you won't get this way.... and you will miss the webserver (and possibly other applications on the server) configuration.
Wget is a very bad way to perform backup on a web site.  I would suggest rsync, scp, or some other method based on file copying, not GET based HTTP calls. Wget was not designed for this purpose. It's a simple download tool for known target URLs, and not meant to be used in the manner you are describing. It can only grab things that are linked via hyperlink. Any scripting, as Noci above stated, such as php, perl, etc., it would only grab what was rendered into html, not the actual scripting code that generated it. Therefore, if you were to restore it, the scripts, at the very least, would not function.
If you want an example how your site will look then go to Archive.org...  and lookup some familiar sites... It's more or less a static picture show. (mostly)
This question needs an answer!
Become an EE member today
7 DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.