Pushing the boundaries of major incident response with agentic AI

Everything fails, all the time.

The famous words from Amazon's CTO Werner Vogels still hold true!

Time and again we've seen that a single point of failure can bring the digital world to it’s knees. Think about last year’s CrowdStrike debacle or the recent AWS outage. These involved two very different players, but for each, one small failure quickly cascaded into widespread disruption. While end users were feeling the full brunt of downtime, IT teams were quite literally racing against the clock to restore services.

Now, if there's one thing major outages like these have made clear, it's that sometimes they're simply inevitable and they catch you off guard.

That's exactly why, instead of relying solely on traditional incident management workflows, IT teams need to diversify and modernize their approach by infusing intelligence into their incident response flows. This can help incident response teams (IRTs) spot subtle anomalies that could otherwise slip through human analysts. And as we now step into the agentic era, it’s clear that AI will only play an even greater role in reshaping how IT incidents are detected, analyzed, and resolved.

The evolution of AI application for major incident response

Incident response has involved AI for awhile, and over the years, incident management practices have benefited from ML through the application of smart categories, subcategory predictions and intelligent technician assignments. The advent of GenAI and its subsequent adoption in IT service management platforms has further enabled technicians to accelerate resolution times, and has helped end users self-resolve their issues by making relevant knowledge much more accessible.

Given the high stakes of major incidents, both the process and its stakeholders stand to gain from other AI capabilities, like AI-powered impact and root cause analysis, streamlined contextual communications, and more. Today, with the rise of AI agents and agentic AI capabilities, we can now craft more powerful major incident management workflows that minimize the business impact of major incidents and proactively help avoid them. Let's look at a quick use case to understand how the application of AI for major incident management has evolved over the years.

A quick use case

a) Challenge:

A global retail chain decides to perform an organization-wide digital transformation project to enhance its operations. As part of this effort, the IT team begins upgrading its database infrastructure to a newer version of Microsoft SQL server. Soon after the rollout, the point-of-sale (POS) systems across multiple stores start going offline. Store employees are unable to process transactions, customers are queuing up, and operations come to a standstill.

b) Root cause:

The cause is later traced back to an incompatibility between the upgraded SQL server and the existing POS systems. This problem went unnoticed because compatibility tests weren’t conducted beforehand.

c) Remediation through:

• Traditional best practice workflow
• Best practice workflow with simple AI features
• Agentic AI augmented workflow

1. Traditional best practice workflow with simple rule-based automations

• Retail store employees across multiple stores start flooding the service desk with incident tickets.
• Rule-based automations are triggered to triage the tickets that meet the set conditions.
• Technicians manually review the tickets to identify similarities and determine whether they are a part of a larger issue.
• Once patterns are recognized, the technicians manually link the related incidents to a major incident record.
• The IRT scours through all of the technician notes and ticket conversations to understand the issue.
• The IRT sorts through an overload of data from other disjointed sources like the UEBA logs, recent change records, privileged access logs, database activity, and third-party update histories.
• This eats up a lot of IRT time as members are debating the probable root cause.
• Stakeholder communication is automated but it follows standard, canned notification templates that provide limited details.
• The IRT, after a long root cause analysis, determines that a recent DB upgrade is causing the POS software to fail and reverts to the previous version to fix the issue.
• The workflow adheres to best practices, but it is reactive, labor-intensive, and slow to restore service.

2. Best practice workflow enhanced through assistive intelligence from simple AI capabilities

• As the POS failures begin, monitoring tools flag alerts and log them as tickets within the ITSM platform.
• AI-powered triaging help automatically categorize, prioritize, and route incoming tickets. This reduces the need for each incident ticket to meet rigid rule sets for the triaging automation to kick in.
• AI case clustering helps consolidate related tickets under one major incident record, eliminating duplication of effort and manual correlation of similar tickets.
• Meanwhile, a GenAI-powered virtual support agent helps generate tailored updates for different stakeholder groups including organizational announcements, end user responses, and technician notes. Instead of relying on static templates, these communications are generated on demand.
• The virtual agent generates instant ticket summaries to bring IRT members up to speed by giving an overall gist of the ticket conversations, ticket parameters, and technician notes.
• The IRT then performs a root cause analysis and confirms the database compatibility issue to be the root cause and deploys a fix.
• After remediation, the virtual agent assists with generating the post-incident review report, reducing the documentation effort required from the team.

3. Agentic AI augmented workflow to advance incident response maturity

• An AI agent with access to the observability dashboards detects a surge in logs showing failed POS API calls to the SQL server.
• It checks the network traffic, authentication attempts, and system logs and observes that these failures are occurring across multiple retail locations, indicating a widespread issue.
• Meanwhile, service desk queues begin to flood with tickets from store employees.
• The AI agent, which also has access to the ticketing system, notifies the IRT with a summary of its findings and asks whether it should create a major incident ticket and initiate the response workflow.
• Upon approval, it clusters all similar tickets together and links them to a single major incident ticket.
• It then automatically responds to the end users who raised those tickets, informing them that IT is aware of the issue and is actively working on a fix. (The responses are sent autonomously instead of being generated on demand.)
• Another AI agent, with access to the organization’s change management records, correlates the timing of the incident with a recent database upgrade. It finds that shortly after the SQL server was upgraded, the POS systems began failing to connect.
• The agent compiles these insights and shares them with the IRT, enabling the team to quickly identify the DB upgrade as the root cause instead of spending valuable time troubleshooting from scratch.
• The AI agent, being trained on domain-specific knowledge and having access to the service desk's historical change documentation, recommends rolling back to the previous SQL server version to resolve the incompatibility issue.
• To stop the current version and restore the backup of the older one, the AI agent also suggests a remediation script outlining steps.
• Once the IRT approves, the AI agent assists in executing the rollback, helping restore normal operations across all stores.

Looking ahead...

It’s safe to say that AI has seen massive advancements over the years. From deterministic chatbots to GenAI-powered virtual agents and now to autonomous AI agents, we have come a long way. Thankfully, ITSM as a discipline has managed to keep pace and has done a decent job of evolving alongside these changes. As we step into what looks like the agentic era, the use cases for AI-driven ITSM, and major incident management in particular, will only continue to grow. The focus will shift from simply speeding up workflows to building systems that can think, decide, and act with minimal human input, changing the way organizations handle disruptions and deliver value.