asked on S3 file version managment
For compliance we need to maintain native SQL backups. On-prem systems use Veeam backup which easily gathers daily SQL backups and manages a retention policy of daily for 2 weeks, end of month for 6 months, end of year for 7 years. The company is now building the next technology stack on Amazon Web Services and the S3 bucket version management is woefully simplistic. I have created a lambda to trigger a native SQL backup once a day, and hoping to be able to manage the version retention, it simply overwrites the file in S3. This DB is expected to grow to around 4TB by the end of the year, so paying to store every version every day for 7 years is out of the question.
Has anyone in this group come across, or written a lambda (or other widget) that can be triggered to look through S3 previous versions and prune the excess according to a selected or defined retention policy as described?
Has anyone in this group come across, or written a lambda (or other widget) that can be triggered to look through S3 previous versions and prune the excess according to a selected or defined retention policy as described?
Storage SoftwareAWS* aws_lambda
Last Comment
David Favor
ASKER
Thank you David.
I also see that wasabi.com has hot storage for cheeap as well. This will likely be part of the plan, but the truth is I am being lazy. If someone has already written a tool to accomplish the goal, I don't mind spending a little for it. Free95 is always better, but a few sheckels to defer me from reinventing a wheel is worth it. Somebody has been down this same road and probably found or made a nice elegant solution.
I also see that wasabi.com has hot storage for cheeap as well. This will likely be part of the plan, but the truth is I am being lazy. If someone has already written a tool to accomplish the goal, I don't mind spending a little for it. Free95 is always better, but a few sheckels to defer me from reinventing a wheel is worth it. Somebody has been down this same road and probably found or made a nice elegant solution.
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Thanks for the input David. I will be moving forward with a plan to move the data somewhere else to maintain compliance archives.
You're welcome!
Disk space is so cheap these days, I always cringe just a little when people start talking about use S3.
Disk space is so cheap these days, I always cringe just a little when people start talking about use S3.
You can lease around 12TB for <$90 USD/month. These are Linux machines, so no upload/download bandwidth charges. No per object access charges. No other charges. Just one flat rate.
You can simply create versioned tarball files each midnight, pruning by number of days or maintain some percentage of free disk space.
Tip: You can also merge the two approaches, pulling a copy of your S3 bucket(s) each midnight to your OVH machine, then generating versioned tarball files.