Avatar of Bill Herde
Bill Herde
Flag for United States of America asked on

S3 file version managment

For compliance we need to maintain native SQL backups. On-prem systems use Veeam backup which easily gathers daily SQL backups and manages a retention policy of daily for 2 weeks, end of month for 6 months, end of year for 7 years.  The company is now building the next technology stack on Amazon Web Services and the S3 bucket version management is woefully simplistic.  I have created a lambda to trigger a native SQL backup once a day, and hoping to be able to manage the version retention, it simply overwrites the file in S3. This DB is expected to grow to around 4TB by the end of the year, so paying to store every version every day for 7 years is out of the question.  
Has anyone in this group come across, or written a lambda (or other widget) that can be triggered to look through S3 previous versions and prune the excess according to a selected or defined retention policy as described?
Storage SoftwareAWS* aws_lambda

Avatar of undefined
Last Comment
David Favor

8/22/2022 - Mon
David Favor

https://www.ovh.com/world/dedicated-servers/storage/prices might be a better choice than S3.

You can lease around 12TB for <$90 USD/month. These are Linux machines, so no upload/download bandwidth charges. No per object access charges. No other charges. Just one flat rate.

You can simply create versioned tarball files each midnight, pruning by number of days or maintain some percentage of free disk space.

Tip: You can also merge the two approaches, pulling a copy of your S3 bucket(s) each midnight to your OVH machine, then generating versioned tarball files.
Bill Herde

Thank you David.
I also see that wasabi.com has hot storage for cheeap as well.  This will likely be part of the plan, but the truth is I am being lazy.  If someone has already written a tool to accomplish the goal, I don't mind spending a little for it.  Free95 is always better, but a few sheckels to defer me from reinventing a wheel is worth it.  Somebody has been down this same road and probably found or made a nice elegant solution.
David Favor

View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Bill Herde

Thanks for the input David.  I will be moving forward with a plan to move the data somewhere else to maintain compliance archives.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
David Favor

You're welcome!

Disk space is so cheap these days, I always cringe just a little when people start talking about use S3.