Link to home
Start Free TrialLog in
Avatar of Machete44601
Machete44601

asked on

cost effective, automated solution for AWS S3 or Glacier offsite backups

Hello,

I'm looking for a cost effective, automated solution for AWS/Glacier offsite backups.  I understand the fundamental differences between Glacier and regular storage in AWS (atleast I *think* I do).

Here are the items that need backed up regularly:
(2) Local machines (Win7/Win10)
(1) Plex Media server
(1) WD NAS (stores backup of Plex Media, plus non-automated backups of local machines)

Local machines are currently backed up in irregular intervals manually with Beyond Compare to the NAS.
Media files from Plex are automated backups to NAS.

Manually once a year using Beyond Compare we run a full comparison on the media files on AWS and the NAS.
Is there an easier automated way to zip these media files (so there's less Put/Get/List Requests) and only have updated/changed zipped files uploaded to S3?  To automate individual file scanning/comparison on a routine basis gets "expensive".  I can't control costs related to amount of storage needed - that's a given, but when I run Beyond Compare against the S3 storage - the amount of requests to compare each file for changes gets extreme - upwards of $50/month, which maybe isn't expensive to some, but for a home set-up that's a little much IMHO.

The local machine Beyond Compare backups to the NAS are just file replication - copy/overwrite so it's not even a great backup solution - but atleast if a drive on a local machine dies, mission critical files exist somewhere else... not the best by any means, but better than nothing.

Since we're talking about over 2TB of data (and growing) in the past any off the shelf solution such as Carbonite, etc. has been unfeasible.

I've been out of the IT game for a while and have lost touch with some of the more robust solutions out there. I don't mind investing in an application that will complete the task - I just have to think that there's a better way to regularly update storage objects in S3 with local backups that doesn't include comparing 100,000+ objects everytime.

Thanks!
Avatar of David Favor
David Favor
Flag of United States of America image

https://www.ovh.com/world/dedicated-servers/all_servers.xml?range=storage cheapest online solution by far.

You can even go cheaper by setting up your on local storage of whatever size you like, then exporting your local disk to your public servers.

I use to do this, till OVH came up with their cheap data storage systems.
Avatar of Machete44601
Machete44601

ASKER

Thanks David - I guess I wasn't looking for (or had thought about) a dedicated server - unless you're suggesting what I want to accomplish isn't possible/feasible with AWS/S3?
I should have been more clear...

You said, "I'm looking for a cost effective, automated solution for AWS/Glacier offsite backups..."

Likely the only way to access this type of storage is via AWS API calls, so this type of storage isn't general purpose storage like a disk with a filesystem.

To determine how best to access this storage, refer to AWS Glacier docs... Try these starting points...

https://www.youtube.com/watch?time_continue=11&v=EKaJENJqD8E suggests you can use tar by treating AWS Glacier as a tape device.

And... If you look at OVH pricing, they're actually slightly cheaper than AWS Glacier + you end up with normal filesystem based data with is accessible instantly, rather than waiting for various lengths of time (1 minute to 12 hours) for data retrieval.

This can be tricky, as you can't simply access data.

You must build in a sequence of steps to initiate data retrieval, wait for retrieval, access data, destroy data.

Seems overly complex, compared to using a 100% always available data storage approach.
you could use cloudberry backup and go direct to glacier
Have you looked at Wasabi.com's pricing? no api usage/egress costs
Thanks to both David's your ideas have helped me further define the parameters of the project and think through some logistics.

First David F - you're correct, I initially did state AWS/Glacier.  I omitted S3 as an option - I have 90K+ media files currently in S3 and set to convert to S3-Glacier storage after X time.  I'm not concerned about recovery time - as these are off-site backups and I'm not typically in need of fast recovery.

David J - I'll look at both solutions you posted.

After thinking about this more, what I'm looking for may be able to be done within Beyond Compare (as it talks to S3 / can list S3 buckets)... What I want is to be able to zip folders, upload to S3 and then have a program compare the contents of the local directory to the .zip contents on S3. This should reduce the amount of objects in my buckets and thus reduce the amount of API calls Beyond Compare makes to S3.

I can then script everything else to be automated... I think.  I've reached out to the authors of Beyond Compare to see if this is possible.
The problem with S3 + Glacier is the real cost.

Let's say you have 90K files, just to scan these files will incur a per object scan cost.

So you must be very careful using S3 or Glacier. Simple directory listing or file searches, can put you in the poor house.

I prefer OVH or local storage (exported via sshfs) for massive disk space, because these incur fixed overhead, so you don't have to worry about some developer having an good idea which incurs per object access fees... which no one knows about... till a heart stopping bill arrives...
Hi David -

Yes, I've gotten one of those heart stopping bills - about $50/month for scanning updating those 90K files.  When I'm not polling those, it's only $15/month.  Hoping to get somewhere in the $25-$30/month level...

Just looking for an automated and less costly way to perform this.  Thinking that compressed files might help reduce those overhead costs and allow me to backup offsite on a more frequent basis (rather than every 12 months).
This question needs an answer!
Become an EE member today
7 DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.