A Series of Unfortunate Backup Events

Published:
Updated:
Learn how the use of a bunch of disparate tools requiring a lot of manual attention led to a series of unfortunate backup events for one company.
Gitlab.com is a very popular Software as Service (SaaS) platform for software developers. Recently, they suffered a series of unfortunate events due to incomplete backup practices that resulted in customer data loss and service down time. They are back up and running now (minus some essential client data) and there are a few important lessons we can learn from their backup approach and resulting experience.
A unified backup solution is the best approach
Using a series of disparate tools that require a lot of manual attention isn’t the optimum way to provide data protection and improved uptime. Running a patchwork of one-off “point solutions” can be challenging and error prone, as the Gitlabs.com loss showed. “Is that script running? What is the backup status? Are those files actually being backed up?” should not be questions your team should ever have to ask.
Contrast that with a unified solution that backs up and protects all platforms -- from Windows to Linux to Azure to database applications and more -- one that has embedded monitoring and alerting to provide ongoing at-a-glance assurance.
A unified backup platform should:
  • Simplify operations. From installation to maintenance to daily operation, a unified backup solution that is easy-to-use is essential.  Why?  Because it is more likely to be set up and used correctly and consistently. A well-designed solution requires very little special knowledge, while scripted solutions often only make sense to the person who create the script in the first place.
  • Provide a consistent experience. Having a different backup solution for Linux servers, Azure workloads, database applications, and whatever other data sources a business depends on, means having different solutions to understand and different consoles to check for statuses. A unified solution protects all platforms equally and from one console.
  • Be easy to control and monitor.  An integrated solution quickly and effectively lets multiple staff know when everything is okay or when there is something that requires attention.  A single management console is ideal, because even the best IT people can get overwhelmed in the heat of battle.  Having an automatic and easy-to-understand management console is as important as reliable backup and recovery.
Automated workstation backup should be part of your plan
Be sure to understand the ways in which your SaaS provider backs up and the service level agreement (and technologies) they use to guarantee this.
Even with the best services, having additional backups of each team members’ workstation is the way to go. It makes sense to deploy a solution that keeps an additional backup of individual workstations. For example, imagine you are a SaaS client who lost work using a service that went down. Having another local or alternate cloud backup of each workstation is an additional layer of protection to keep up work continuity.
Testing backups is essential
The Gitlabs.com issue illustrates that testing your recovery process before you really need to recover is essential.  Does your recovery process work? Why back up at all if your recovery processes does not work? Testing your backups and backup processes should be a regular part of your data protection plan.
When possible, even cloud services themselves should be backed up
A great example of a cloud service that should be backed up is Office 365.  Office 365 provides excellent in-cloud data replication for availability, but does not provide backup for data retention and recovery. Having a solution that can back up and recover application level data (like Office 365) is essential to prepare for data loss due to service provider failure or employee error.
In addition, some industries and regions require a backup copy for compliance reasons and having a local backup of a cloud service or a cloud-to-cloud backup can help meet these requirements.
 
0
812 Views

Comments (0)

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.