S3 file version managment

Bill Herde
Bill Herde used Ask the Experts™
on
For compliance we need to maintain native SQL backups. On-prem systems use Veeam backup which easily gathers daily SQL backups and manages a retention policy of daily for 2 weeks, end of month for 6 months, end of year for 7 years.  The company is now building the next technology stack on Amazon Web Services and the S3 bucket version management is woefully simplistic.  I have created a lambda to trigger a native SQL backup once a day, and hoping to be able to manage the version retention, it simply overwrites the file in S3. This DB is expected to grow to around 4TB by the end of the year, so paying to store every version every day for 7 years is out of the question.  
Has anyone in this group come across, or written a lambda (or other widget) that can be triggered to look through S3 previous versions and prune the excess according to a selected or defined retention policy as described?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
David FavorFractional CTO
Distinguished Expert 2018

Commented:
https://www.ovh.com/world/dedicated-servers/storage/prices might be a better choice than S3.

You can lease around 12TB for <$90 USD/month. These are Linux machines, so no upload/download bandwidth charges. No per object access charges. No other charges. Just one flat rate.

You can simply create versioned tarball files each midnight, pruning by number of days or maintain some percentage of free disk space.

Tip: You can also merge the two approaches, pulling a copy of your S3 bucket(s) each midnight to your OVH machine, then generating versioned tarball files.
Bill HerdeOwner

Author

Commented:
Thank you David.
I also see that wasabi.com has hot storage for cheeap as well.  This will likely be part of the plan, but the truth is I am being lazy.  If someone has already written a tool to accomplish the goal, I don't mind spending a little for it.  Free95 is always better, but a few sheckels to defer me from reinventing a wheel is worth it.  Somebody has been down this same road and probably found or made a nice elegant solution.
Fractional CTO
Distinguished Expert 2018
Commented:
Using S3 always involve more time + cost than any other solution + any estimates... so for use professionally lazy folks, we avoid S3 like the plague. :-)

If you must use S3, then far better to pull copies of S3 data somewhere else + create a tarball of your data.

To my knowledge I've never seen any type of off the shelf S3 versioning system, as this would bloat/increase monthly S3 charges into the stratosphere for large amounts of data, which is a likely reason for versioning. You can always create your own offsite tarball accessing S3 data, then copy the tarball back to S3. This seems like an easy option.
Bill HerdeOwner

Author

Commented:
Thanks for the input David.  I will be moving forward with a plan to move the data somewhere else to maintain compliance archives.
David FavorFractional CTO
Distinguished Expert 2018

Commented:
You're welcome!

Disk space is so cheap these days, I always cringe just a little when people start talking about use S3.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial