Our enterprise IT change management framework

2) Responsibilities

Throughout a change, we provide clarity and ensure those involved in the change are aware of their responsibilities. Each person or team involved plays a role, and we lay it out for them through policies and guidelines.

Amongst the many roles, a significant one is that of the CAB. It is a group of people who the change manager trusts as guides in the decision-making process. The CAB evaluates the change once the change manager approves it and an implementation plan is in sight. It provides crucial recommendations and oversees the implementation. Let’s consider a major change that affects the entire organization, like shifting a division of our IT operations from one office to another.

Here's what our CAB looked like when we implemented that change in 2015:

As you can see, this is a mixture of experienced individuals and experts from various levels of the organization. For a change like migrating a service to a new data center, the CAB would be different and include more members from the IT team.

• The CAB must have a good mix of various job roles and teams.
• A large CAB could slow down the process. We choose CAB members on a minimalist basis.
• We appoint experienced individuals from various facets of the business. This may include C-level executives, managers, technical teams, finance teams, and more, depending on the nature, severity, and scale of the change. Their experience in helping us spot hidden challenges cannot be overstated.
• We document and communicate the role of CAB members clearly to them. They are responsible for assessing the change from the initial stage to the review stage, providing their perspectives, recommendations, and course corrections.
• Their contributions will be crucial while creating risk mitigation plans, monitoring the change for red flags, and suggesting improvements by reviewing the changes.

We also ensure change managers and implementers clearly understand the roles of the CAB. Not all changes need to go through a CAB, and neither is the CAB a rigid establishment. That's why the change manager takes charge of the change and decides how the CAB members must get involved.

A CAB is just one crucial role in our change management model. Apart from the CAB, we have four critical roles with their own sets of responsibilities:

3) Review

Review has helped us shape our framework over the years. While being an integral part of our change management process itself, it is also a pillar of learning and improvement. Review goes beyond just reviewing a change on a case-by-case basis. IT leaders review how we perform changes overall and where we're headed, using a set of metrics and parameters.

• Percent of successful changes: During change closure, the change manager records how well the change went and whether it was successful or not. The review team determines why the change was a success or failure based on how well the objective was achieved and if resources were wasted. We aim to improve this percentage every quarter.
• Patterns of unsuccessful changes that were rolled back: Be it a simple network change or a complex data center migration, a rollback indicates we didn't do something right in the planning stage. It is normal to roll back changes, but we ensure we understand the reason and notice patterns in these changes.
• Proportion of change requests in a particular division: Too many change requests in firewalls, switches, or routers could indicate a need to revisit this infrastructure. Too many changes in the data center could indicate more risk around the corner, or it could just be due to increased customers, new employees, or a host of other reasons. Either way, it is worth looking out for this parameter to be on the safe side.
• Frequency of incidents caused by change: Obviously, if a change causes incidents, it needs a revised process or more checklists. Further, we see if a particular area of change (like in networks or devices) frequently causes minor or major incidents.
• Percent increase in emergency changes: A high percent increase is a huge red flag, especially if the emergency changes are unforced. A DDoS attack or a disaster may not be in our control, but a normal change gone wrong is certainly something we want to avoid. We investigate such emergency changes and record the learnings.
• Cost incurred on changes: This can be a bit difficult to calculate at times due to the sheer breadth of our operations. However, the objective here is to see what proportion of our resources (person hours, tools, and direct costs) we spend on changes. It need not be an exact amount, but we must know if it is increasing every month or even every quarter.
• Time taken to complete various levels of changes:We have our SLAs set for change requests, and we ensure we maintain SLAs at all levels.
• Approval rate: This is the percent of changes approved by the CAB. A high percentage means our planning is good.

The review team evaluates each change, updates the knowledge management database, and documents the learnings. The IT leaders review the overall progress of our change management journey and make decisions to improve our processes.

Scenario 1: A change escalation

One of the most interesting perspectives of our leaders is a counterintuitive one. Here's how some of our wise people put it:

Here's a good example of what not to do and the type of escalation one should avoid:

Let's start with a standard change of updating security patches in a server. Since it's a no-brainer that it's critical, you might have categorized it as a standard change, checked for the pre-approved status and the documentation, and went on to update the server. However, some of your critical applications might be running on that server, and they'd still only be compatible with older versions. Now those applications are not functioning properly, and you must move them to a backup server. This will be a major change. You'll now have to involve a CAB and follow a different process. Let's say this backup server does not have a disaster-recovery mechanism, and your critical application data is lost. This is an emergency change as it affects many users, and your C-level executives are now responsible for cleaning up this mess.

What should have been a standard change has now moved on to become an emergency change. This is exactly what our IT leaders try to avoid. Here is the road you should avoid too:

A change management roadmap to be avoided

The methods we discuss in the following chapters will help you avoid this road by using frameworks for each type of change.

Scenario 4: Establishing a system for major changes

Major changes are mostly normal changes that could turn dangerous without more vigorous filters. For example, firewall policy changes are mostly normal, but whether one is major or not depends on which firewall it is. A firewall for a back-end internal application like the one we use for tracking attendance would be a normal change; it’ll only affect a small proportion of users for a limited time. But if the same firewall is for a data center that hosts critical services, it could be a major change. Which applications run in that data center, who uses those applications, and what is the impact if the change goes south—all these questions would come into the picture. The impact ultimately decides whether it’s a major or minor change.

Most major changes affect the entire organization. One of our IT leaders identifies the need for it first and calculates the benefits the change would bring to us. Then, they assign a suitable change manager to collect information about how this major change would affect our operations. It is then a matter of executing our general methodology we discussed earlier.

Also, since it's a major change with considerable impact, we need to tackle these challenges with even more care. Let's see how these hidden challenges cropped up when we implemented a major change many years back.

One of our primary services, our in-house email service, was hosted in a primary data center, which we’ll call DC A. We have primary and secondary data centers along with a backup for each. Over a period of time, we started seeing minor incidents about the email service being slow for some users in our company. When our incident management team analyzed this with the help of our IT operations team and technical staff from the email team, we initially figured the issue might be due to our internet service providers (ISPs).

However, the ISPs were firm that the problem was on our end. This back-and-forth communication didn't take us anywhere, so we decided to make a major change: transfer the email service to another secondary data center (let's say DC B). Then we'd move on to analyzing DC A and sort the problem out once and for all.

Moving a service to another data center is a major change, and we had a framework with the following:

• Well-documented procedures to carry out the implementation
• Trained technicians
• An asset management database coupled with our change management module
• In-house tools for monitoring, implementing, and reviewing the change

On paper, these are enough to carry out our elemental approach and execute the change. But that wasn't enough, and that's where we needed a filter to spot and tackle hidden challenges. When we went on to implement the change, here's what we found:

• DC B's backup was not configured well.
• There were other issues with DC B, like firmware and hardware problems.
• These issues were not documented according to procedure.
• There were other applications that had to be moved to DC B, but they weren't moved because they were not compatible with the hardware and OS.

Luckily, our experienced IT leaders had the eye to notice these issues early, and we rolled back the change soon. But if we hadn't, it could have paved the way for a major incident that affected multiple users in our company and might have even turned into an emergency change.

We had a proven framework that helps us all day with normal and standard changes, but something was lacking. What was it?

In other words, our CMDB was not equipped to cover all aspects of a change. The CMDB that helped us execute normal changes well needed an even more vigorous upgrade. A CMDB is a machine that facilitates decision-making. Technically, a CMDB has a list of items that must be managed to ensure services function properly, and it also has the relationship between these items. Practically, though, it is a machine that needs training; the more versatile the information you feed it, the better it'll help you make decisions.

In this example, a 360-degree CMDB would have included the technician, the secondary data center, the applications running on it, its history, the other technicians involved in its operation, and a lot more. A review of the architecture of the secondary data center would have saved us from this risk.

When you make a major change, you go from path A to path B. A 360-degree CMDB ensures path B isn't a dead end first.

Before you make a major change, ask yourself a set of questions related to the CMDB along the lines of compatibility, security, availability, and integrity. Let's see how such a filter would help for major changes.

Creating a filter for major changes

The table below gives you a checklist that would have helped us when we implemented the change. There is also a generalized version of the same list that you can use to create a filter for major changes.

This checklist would have reduced the risk we were about to face. In fact, we'd have not planned the change at all before fixing the problems with DC B. A CMDB trained with all the above information should be the starting point of a major change.

The above is not an exhaustive checklist at all. Ultimately, it's up to you and the other IT leaders in your company to come up with such checklists for different scenarios of major changes that you go through. To create such a filter for yourself, you can focus on these aspects: resources, responsibilities, and review.

Applying the filter

We once had to upgrade our email servers. Since this is a critical service, our change owner categorized it as a major change and decided to implement it during the weekend so that the business impact would be minimal.

Our CAB consisted of senior engineers, IT department technicians, and also the head of operations since Zoho Corp. (the parent company of ManageEngine) was itself a user of the email service. We followed the general approach of initiating the change, planning it, and analyzing the impact. We estimated there would be considerable downtime and contemplated our backup resources and other security measures.

Since it is a major change, we decided to run it through the filter we discussed earlier.

Let’s consider the following questions from the filter we created earlier:

• Which services are dependent on these entities?
• Which stakeholders depend on these services during the change implementation window?

With our CMDB, we identified the stakeholders involved in the services affected by the change, and we then came to know that a series of HR interviews were scheduled during the weekend (the scheduled downtime). There was a CAB member from operations who we consulted with. But we didn't necessarily pull back the change since the IT leaders in the CAB recommended that this change was crucial, and not doing it immediately could cause the company to lose money. The change manager took that recommendation, and we rescheduled the HR interviews. If we did not have a well-trained CMDB and a filter, the HR team would not have known about the change, and the interviews would have happened as scheduled. They'd have been disrupted by the email server upgrade, and the change would have gone down as an unsuccessful one.

This is the importance of handling the planning stage using a filter, and also how crucial it is to communicate with stakeholders. Let's take a look at another example, the one we discussed already in Chapter 2 when we saw how an escalation could create huge problems. Let's see how this filter can prevent the scenario from turning into an escalation:

“Let's start with a standard change of updating security patches in a server. Since it's a no-brainer that it's critical, you might have categorized it as a standard change, checked for the pre-approved status and the documentation, and went on to update the server. However, some of your critical applications might be running on that server, and they'd still only be compatible with older versions. Now those applications are not functioning properly, and you must move them to a backup server. This will be a major change. You'll now have to involve a CAB and follow a different process. Let's say this backup server does not have a disaster-recovery mechanism, and your critical application data is lost. This is an emergency change as it affects many users, and your C-level executives are now responsible for cleaning up this mess.”

The second stage here is a major change. You're supposed to move some applications in your main server to a backup server. Without the filter for the major change, we know it's going to result in an emergency change. But with the filter, the following questions come up:

Note: The entities here would be the main server and the backup server, according to the filter.

• Which services are dependent on these entities?
• What is the BCDR setup for these entities?
• Who is the owner of the BCDR setup?

These three questions would have helped you realize that there's no backup, and you'd have held off on the change until you fixed that problem first.

Scenario 5: Preparing for emergency changes

An emergency change is when you realize something is broken; it's predominantly a reaction to an incident. In our incident management handbook, we discussed how ManageEngine deals with such incidents and carries out root cause analysis (RCA). Corrective and preventive action (CAPA) is a natural result of RCA. The corrective actions mostly coincide with how we must execute the emergency change.

Emergency changes could belong to one of these two categories:

• A reaction to external events
• An attempt at fixing an unsuccessful normal or major change

The frameworks we discussed so far will help you reduce the chance of falling under the second category. We will also see how else you can prevent these emergencies. For the first one, however, you need to have a system of measures to help you handle those events with enough conviction.

A system for the outside

ManageEngine's leaders focus on one prime aspect when they are faced with an emergency from the outside: the ability to balance urgency and caution.

When the emergency change is forced from outside, we are looking at a potentially huge impact and the need to hurry to reduce damage.

4 stages of the emergency change life cycle