3 Keys for Successful Deployments

Deploying our service is a grudge match between customer benefits and customer pain. In one corner, rolling out fixes (yay!) and delivering new features (double yay!). In the other corner, training on new features (boo – sounds like work), and change management processes (more work).

We put a great deal of effort into optimizing these processes, so it’s important to share how we think through these optimizations. For now, I’ll narrow focus on the issue-fix aspect of the deployment process, and in later blogs I’ll cover new feature delivery.

The fine printBefore I start, a quick disclaimer: what I’m about to describe is subject to change. We’ve been a fanatical agile shop for nearly seven years and we believe change is an important principle of the agile approach. So, if you’re reading this in 2017 or later, please understand that we might have tweaked some of these details. OK, now for the good stuff…

Bug zapping

You might not be surprised to hear that bugs are occasionally found in our service (I know, the horror!). Nobody likes bugs, so we need to address them as soon as possible. And because our service uses a number of telecommunication providers to make text messages flong (on my iPhone, anyway), phones ring, and other third-party communication channels do their thing, managing the infrastructure related to these providers requires constant effort. In short, we need to be able to release and deploy fixes FAST.

Keep those cards and letters coming

It also might not surprise you to hear that our customers ask for new features (the audacity!). From July 1st to Sept 30th, we received 133 service enhancement requests – that’s 2+ per business day. The product managers who handle these requests interpret them into new features and then work with our engineers to build them.

So far, so good – and in theory we should be getting these features into your hands ASAP. But these features are product changes: while the customers who asked for a change are willing to take on any additional testing and training work to get their new features, customers who don’t intend to use the new features don’t want that burden imposed on them. That means we need a process that doesn’t force new features on customers too quickly.

Balancing the need for speed

Okay, so we can probably agree we want fixes fast and features not too fast – how can we possibly strike this balance?

First, by using deployment speed: we deploy at least weekly, if not more frequently. Our core sprint work uses a continuous delivery process which we deploy on a weekly schedule so that we can seamlessly deliver non-critical fixes at that cadence. We can also generate and deploy critical fixes in between these weekly deployments as necessary to address issues like zero-day vulnerabilities.

Second, by using feature flags: deployments include the latest feature developments, but some features are not quite ready for your use after just one sprint. And as previously mentioned, your user base might be confused if new buttons and features show up every week. So we use feature flags/toggles on our new features so that we can conditionally turn them on when we’re ready to release them. That allows the new fixes to get out the door without having new features show up. (We’ll talk more about feature flags in another blog, but those of you who are new to xMatters might want to check out our Early Access Program if you want to play with new features faster).

Third, through thoughtful communication: because fixes can cause behavior changes in the system, we have implemented a new communication process to advertise the parts of the service we worked on so customers can keep an eye out for any unexpected behavior. Our support notes are designed to provide this information (here’s a sample from the deployments leading up to our Rogue release). Those notes work in conjunction with the scheduled maintenance posts on our status page to ensure this information is delivered by the latest technologies.

Using these three mechanisms, we can offer the best mix of quick fixes – without forcing training and change management on our customers.