2.2Change management

Changing a live service is itself a leading cause of incidents. Yet routing every change through a heavy approval process would stall even routine daily work and be too late in an emergency. This section covers the thinking behind sorting changes into three paths—standard changes, normal changes, and emergency changes—the role played by the CAB (change advisory board) and the ECAB (emergency change advisory board) in each, and the risk assessment and back-out plan that must always accompany a change, all through the lens of a service manager's judgment.

2.2.1The three paths: standard, normal, emergency

• A standard change is a pre-approved, low-risk, routine change with an established procedure. It needs no per-instance CAB deliberation and can be carried out following a defined procedure (e.g., adding a user account or applying a routine patch by a pre-approved procedure). It is the lightweight path for high-frequency, low-impact changes.
• A normal change is a change that is not a standard change and requires assessment of risk and impact. A request for change (RFC) is raised, and the CAB (change advisory board) assesses the risk, impact, implementation plan, back-out plan, and so on before approving and scheduling it. Most planned changes go through this path.
• An emergency change is an urgent change that cannot wait for normal CAB deliberation, such as recovering from a major incident. The ECAB (emergency change advisory board), a smaller group that decides quickly, approves it; the change is made first and then documented afterward (recorded and reviewed). Because speed comes at the cost of higher risk, its scope is kept to the minimum.

2.2.2Risk assessment and the back-out plan

• Risk assessment estimates in advance the scope of impact and likelihood if a change fails, and decides whether to proceed, the implementation window (a time when impact on the service is smallest), and the necessary standby staffing. It is the CAB's central role, and the higher the risk, the more careful the approval and preparation required.
• A back-out plan (rollback plan) is the procedure for reliably returning to the pre-change state when a change does not behave as expected. It defines in advance whether rollback is possible, how long it takes, and the deadline for the rollback decision (e.g., roll back if normal operation cannot be confirmed by a certain time). A change with no back-out plan should, in principle, not be carried out—even an emergency change.

A service manager running a live member-facing online service discovers a serious published vulnerability in the authentication platform's middleware and must decide which change path to use to apply the fix. First, the manager checks whether this change qualifies as a standard change already covered by a "pre-approved routine procedure," but a patch to the core middleware of the authentication platform has a wide impact scope and may cause side effects on existing features or require a restart, so it is not a routine low-risk change and cannot be treated as a standard change. Next, the manager assesses the severity of the vulnerability and the urgency of exploitation. If no actual exploitation is observed and there is a few days' leeway, it is appropriate to raise a request for change (RFC), take it to the CAB, and proceed as a normal change—approved and scheduled after the CAB deliberates the risk assessment, the implementation window (a low-usage late-night slot), standby staffing, and the back-out plan (the procedure to revert to the old version if authentication fails after patching, and the deadline for that decision). On the other hand, if the vulnerability is already being exploited in real attacks and leaving it would directly lead to a major incident of spreading account compromise, there is no time to wait for normal CAB deliberation. In that case the manager takes the emergency change path, convening the ECAB (emergency change advisory board) to decide quickly, and applies the patch first to protect the service. Even for an emergency change, however, the back-out plan must not be skipped: a procedure to revert immediately if the patch breaks authentication is prepared, and the scope is kept to the minimum needed to contain the vulnerability. After execution, the change content, rationale, and result are documented afterward and reviewed, feeding into permanent prevention (reflecting it in the formal patch-operation procedure). Choosing a change path is thus not a binary of "fast or careful," but the judgment to sort among standard/normal/emergency according to the risk of the vulnerability's urgency and impact scope, while never dropping risk assessment and the back-out plan on any path.

2.2.3Section summary

• Standard change = pre-approved, low-risk routine (no CAB deliberation); normal change = RFC raised and approved after CAB assessment; emergency change = ECAB approves quickly and it is documented afterward
• Choose the change path according to risk (urgency and impact scope), judging by the trade-off between speed and caution
• Risk assessment and a back-out plan must never be omitted on any path, including emergency changes