Incident postmortem and RCA templates for IT teams

So here's how Zylker performed its incident postmortem. The IT team at Zylker started by designating John Clarke as the incident postmortem owner. John was part of the incident response team and had first-hand knowledge about the outage. He prepares the following postmortem document for Zylker.

One of the most crucial steps involved in the incident postmortem is the root cause analysis (RCA).

An RCA is a systematic method to find the underlying reasons for an incident. The team gathers and analyzes all the available information—such as event logs, server configurations, etc—and nails down the root cause. This helps the organization learn from the incident and ensure similar incidents do not occur in the future.

Here are a few methods Zylker used in their RCA:

Five whys method

The five whys method is a technique for thinking through an incident by repeatedly asking the question "why?" until the root cause is identified, usually in about five whys.

In Zylker's case, let's see how their IT team identified the root cause:

• Why were customers unable to place orders? Because the order management system was unavailable.
• Why was the order management system unavailable? Because the database server was disconnected from the application servers.
• Why did the disconnection happen? Because the upgraded version of Microsoft SQL server was incompatible with the application codebase.
• Why was the database upgraded without adequate analysis? Because the CMDB was not updated to reflect the latest dependency relationships, so the analysis that was performed did not reflect the live environment.
• Why was the CMDB not updated with the latest dependency relationships? Because communication lapses within the team lead to the standard operating procedure for dependency mapping to be no longer followed.

Fishbone diagram

Sometimes known as a Ishikawa diagram or a cause-and-effect diagram, this visual tool identifies potential contributing factors categorized by root causes like people, processes, technology, and environment.

The incident response team would create a fishbone diagram with the central problem being the "Order management system outage." Each of the branches, which would then represent a potential root cause category:

Kepner-Tregoe (KT) method

This structured problem-solving approach uses a series of questions to define the problem, analyze potential causes, and develop corrective actions.

Here's how the KT model was applied to find the root cause:

• Define the problem: The order management system suffered an outage, leading to a significant loss of revenue.
• Analyze potential causes: Based on the information gathered in the postmortem and brainstorming from the fishbone diagram, the team identifies potential causes like communication lapses and incomplete configuration analysis.
• Develop and test solutions: The team then evaluates and prioritizes solutions for each potential cause. This could involve implementing a real-time CMDB with granular dependency mapping, updated CAB policies, revised change management workflows, and instituting change freeze windows.

By employing these three RCA methods, Zylker identified the root cause of the outage. Remember that an RCA is a crucial step in the incident postmortem and directs the problem management efforts that will follow through in cases of repeated incidents or major outages.

At organizations that do not actively engage in the exercise of comprehensive incident postmortem, their IT teams lose out on the valuable opportunity to retrace their steps during incident response and identify insights that could make or break their incident and problem management strategy. However, among those teams that do perform incident postmortems, the exercise can quickly descend into a blame-game of pointing fingers. For an incident postmortem and RCA to be truly valuable, IT teams must run both exercises objectively—without directing blame toward anyone—and leverage the incident experience as knowledge that can help infuse resilience into their IT services.