Link to home
Start Free TrialLog in
Avatar of Pete
PeteFlag for United Kingdom of Great Britain and Northern Ireland

asked on

Offsite storage of large, deep nested folders

Hello we backup critical, financial files to the cloud and a domain controller VM. But we can’t afford to backup all the user and curriculum files, videos, etc to the cloud? It is massive, around 2TB. So what can we do for offsite backup of these folders?

Looked at google drive, as a school we have unlimited storage, and a sync tool but it never really got going, errors, 4K/sec...apparently down to the many small files and deep nested folders. Testing a single massive file and it worked ok.

in a mega disaster (someone’s cleared out all the on-site IT, which is kind of plausible) I need some offsite backup of everything....

Ideas?

Thanks for any ideas,
Pete
Avatar of noci
noci

2TB,   that can fit easily on a few NAS's having preferably a RAID1 or RAID 10 config.
If you have a few ( say 3 ) you can make a backup copy one one of them and keep it off-site, next week the next one and keep it off-site.
next week a the third and keep it off-site and then recycle the first one.  etc.
The last backup of a year could maybe be kept a bit longer (special NAS for that one?).

Preferably don't use RAID 5, RAID 6 can be done if needed, RAID 1 (or 10) is better wrt. recovery.
QNAP or SYNOLOGY are usable.  FreeNAS or OpenFiler can be an option.
If systems need to move a lot or quite some distance then using SDD might be advisable.

Key is that the storage is:
Off-line to keep the data safe (against hacking, break-ins, operational errors, ransom-ware).
Off-site to prevent damage in case you building collapses/burns.
Hi, why not migrating all data to gsuite/drive instead of working offline?  No more need to backup if you're low on the budget...
You can easily create a drive label on your network to link it to the software you're using.
kind regards,
Jan
cheapest on cloud is wasabi.com followed closely by Amazon Glacier.
You don't need a NAS for that.  Just copy it to a single external 3TB, 4TB, 5TB, 6TB, 8TB, or 10TB disk, and put it in the bank safe deposit box.  Get a 2nd disk and rotate it each week.  No NAS needed, and a simple copy stored off site with backups to another single disk.  Cheaper than any plan out there.  If you need continuous backup, and the files are stored on a single system, Backblaze might be a good cheap way to do it.
Avatar of Pete

ASKER

I don't feel manual copy to external disks and then taking them home is a modern solution.

Wasabi indeed is very cheap but how do we sync the files to them? Google drive is unlimited size and free but the sync tools we've tried cant cope with the many 1000s of small files and deep nested folders.

Thanks
How about Backblaze?  https://www.backblaze.com/  They're $50/year per system with unlimited backup.  You can put all your files on a single file server and just run the backup to Backblaze.

I don't feel manual copy to external disks and then taking them home is a modern solution.
I didn't say to take it home.  I said put it in a safe deposit box.  You shouldn't put it in your home, as that's the school district's data, not yours.  If you aren't up to tasking someone in the district to take it offsite, there are legitimate backup solution provided by Iron Mountain, Offsite Data Sync along with others that take your cloned data disks offsite for you.  It is far quicker to swap disks in the days of TB to PB sized storage.  Even Amazon gives you disks as a quick way to "upload" data to S3.

Also, you wouldn't do manual copies.  You would have a scheduled backup or run a backup program to clone to the swapped disks, with a set schedule to take one set offsite.
If you need regular backups of 2TB... that is a lot of data to send down a link.
(also when recovery is needed).  
How much time is involved../ Allowed?    
Define old fashioned..., even Amazon parks a containerized datacenter near your building if you need to move a LOT.
off-site is off-site, doesn't need to be home.  Other building on the premises might be sufficient.

Raw speeds:
SAS: 2000GB @ 4Gbps =  16000Gb / ~4 Gbps = 4000 seconds  (raw transfer time, will take more than an hour to copy).
Internet @ 1Gbps : (non-shared full speed access 100% of the time full length and continuous streaming = 16000 seconds ... (will take more than 5 hours in practice).
100Mbps continuous site to site connection:    160000 seconds..   44 Hours (theoretical), i doubt it will be done within 2 days. More likely it will take > 5 days to copy.  (assuming you can expend 50% of bandwidth. Then there is latency .. which might have a bigger impact.
So Cloud storage might be more "modern"/"sexy" but is it usable? You decide on times that can be acceptable.
And "old fashioned" might be best..., i still use a hammer to drive a nail into wood. Surely "ancient" technology.
We use BackBlaze for a number of off-site cloud based storage. They are about the best price wise going. Make sure to tune the client thread count to bring the upload speeds up (ignore the warnings).

2TB is not a lot of data to back up today. We are now using Seagate EX8 SATA disks for backup destinations. In a USB 3 dock they are pulling in 220MiB/Second. That's pretty decent for a SATA spinner.

An option for backup would be either Veeam or ShadowProtect. Veeam v9 Update 4 now has a Community Edition that's free to use. So, back up the data using that! :)
how do we sync the files to them?
High tech solutions like the ones mentioned above
-or-
low tech like this
https://www.experts-exchange.com/articles/33206/Storing-Virtual-Drives-on-Cloud-Storage.html
Have you tried syncbackpro? My calibre library has thousand of files and they get nested pretty deep
ASKER CERTIFIED SOLUTION
Avatar of Pete
Pete
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial