We help IT Professionals succeed at work.
Get Started

S3 file version managment

Last Modified: 2019-12-21
For compliance we need to maintain native SQL backups. On-prem systems use Veeam backup which easily gathers daily SQL backups and manages a retention policy of daily for 2 weeks, end of month for 6 months, end of year for 7 years.  The company is now building the next technology stack on Amazon Web Services and the S3 bucket version management is woefully simplistic.  I have created a lambda to trigger a native SQL backup once a day, and hoping to be able to manage the version retention, it simply overwrites the file in S3. This DB is expected to grow to around 4TB by the end of the year, so paying to store every version every day for 7 years is out of the question.  
Has anyone in this group come across, or written a lambda (or other widget) that can be triggered to look through S3 previous versions and prune the excess according to a selected or defined retention policy as described?
Watch Question
Fractional CTO
Distinguished Expert 2020
This problem has been solved!
Unlock 1 Answer and 5 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE