During critical outages, DevOps teams automatically contact developers and Ops. Effective DevOps stakeholder management informs us who else to apprise.
When a critical outage occurs, DevOps teams automatically know to contact the relevant programmers or ITOps or SecOps teams for help in resolving the problem and managing the incident. But what about everybody else? Don’t they matter, too? What about the marketing and sales and executive teams? Don’t they deserve to know?
The goal of this blog is to:
- Investigate who should be apprised when a critical incident occurs
- Look at how to alert DevOps and stakeholders and keep them apprised during an incident
- What are stakeholder management best practices?
Who you should apprise during an incident?
DevOps philosophy holds that:
[D]evelopers, operations staff, and support people must work closely together on a regular basis. An implication is that they must see one other as important stakeholders and actively seek to work together.
To this end, there should be a process in place to inform the various stakeholders when an important incident occurs which impacts business continuity. Ideally, teams will have a predefined message that will be sent to the necessary teams so that they all learn about the message at the same time. Messages should also be customized according to the level of knowledge each group needs to have to manage the incident.
Teams to alert during a critical incident as part of improved stakeholder management:
- Customer Support: When customers call in to ask when the service will be up and running again, customer support can help with temporary work arounds or advise when the service will be up and running again
- Dev/Ops: During critical downtime, these are the first people to be alerted. Individual IT teams can decide what their workflow should be here. If it is a server issue, the Ops team will need notification. If there is an API call that is stalling, it might be the Dev team that needs to get to work.
- Management Team/ Executives: Usually, these individuals only want and need a high level update at this point. They might want further details later though. Management wants to know that there is an issue and that relevant teams have been notified.
- Marketing: Marketing needs to know about the incident so they can post relevant information to the website and indicate when a resolution is expected.
- Sales: Similar to marketing, sales needs to be aware of the issue so if major external customers call, they can be aware of the issue and keep customers up to date.
How to alert individuals
How you alert individuals to important technical outages is as important as who you alert. In the previous section we discussed who to alert. In this section, we will discuss how to alert the necessary individuals in your group. Ideally, you need to have a way of alerting individuals that is persistent, gets the important information across and escalates if a given individual is not available.
Teams need an IT alerting tool that provides persistent and consistent alerting. Ideally, the alerting platform will have the following characteristics:
- Ability to integrate into ticketing tools
- An on-call scheduler
- An alert escalation policy
- Alerting redundancy so important teams are alerted on smartphones, email and SMS
DevOps stakeholder management best practices
Best practices for stakeholder management require teams to think about incident management long before the critical incident occurs. This thinking begins with a plan and answers the questions of who needs to be alerted and how should they be alerted.
Furthermore, best practices take into account:
You need to make sure you include these components in your plan.
Stakeholder management enables teams to not only think about problems before they happen but also to ensure that during an incident, necessary individuals are apprised of critical incidents. By enabling this critical workflow to occur, teams are able to better understand not only various groups in the company work but also experience a sense of camaraderie and cohesion.
Interested to learn more DevOps stakeholder best practices? Download our whitepaper Stakeholder Management for IT Teams