What's changed: Initial version
6.2Backup operation, automation, and operating procedures
Covers operational automation that hands routine work to machines, the runbook / operating procedures that document steps, human-error prevention through double-checks and call-and-response, and safely applying configuration changes (change-management linkage and rollback) to production. The key is separating routine work to automate from exceptions requiring human judgment.
The quality of operations grows more stable the more you reduce "the number of times a human operates by hand." A service manager must judge how to automate high-frequency routine work to cut human error and effort, standardize work needing judgment or exception handling with a runbook, and design error-prevention measures such as double-checking for the dangerous operations humans still perform. In addition, work that changes the production configuration must be linked with change management and paired with a rollback procedure so it can be reverted on failure.
6.2.1Operational automation and the runbook
- Operational automation hands routine, repetitive work—backups, log collection, job launching, health checks—to scripts or tools. It eliminates human error, cuts effort, and makes execution times precise, but forcing automation onto work that involves judgment or exception handling can misbehave on unexpected input and become dangerous. Automation suits "routine work with fixed inputs and steps"; exceptions needing judgment are, in principle, handled by people.
- Runbook (operating procedure) is a document that spells out the steps, preconditions, checks, and abnormal-case handling so that any operator produces the same result. It prevents dependence on individuals and underpins quality even during handovers or overnight first response. A good runbook includes not only the "normal procedure" but the rollback and escalation contacts for abnormal cases. Because a stale procedure causes accidents, it is premised on being updated (maintained) every time the configuration changes.
6.2.2Human-error prevention and applying configuration changes
- Human-error prevention guards against dangerous operations where a person touches production. Effective measures include double-checking by declaring the target and having a second person confirm before work, pointing-and-calling / checklists before issuing a command, restricting the target and adding a confirmation prompt for irreversible operations such as deletion, and, where possible, rehearsing in a verification environment equivalent to production. "Skip checks because the operator is experienced" is the most dangerous operational culture.
- Applying configuration changes—altering the settings, hardware, or software of the production system—must not be improvised but linked with the change-management process. Run it through CAB approval (normal change) or the standard-change pre-approval path, prepare a rollback procedure and failure criteria before execution, and afterward update configuration management (CMDB/CI) to prevent drift between the physical estate and the ledger. Avoid changes that cannot be rolled back, and localize impact via staged application or a maintenance window.
Most-tested points: "automate routine, repetitive work; humans handle judgment/exceptions," "a runbook includes normal steps plus rollback/escalation and is maintained at every configuration change," "prevent errors on dangerous operations with double-checks and confirmation prompts," and "link configuration changes with change management, pairing rollback with a CMDB update." Questions test judgment on whether to automate based on routine-versus-exception.
6.2.3Designing automation, procedures, and rollback
Suppose a service manager reviews a routine task that operators had performed by hand every night: "take a backup -> delete the oldest generation -> check the success log." This task follows the same steps daily, its inputs (target volume, generations retained) are fixed, and there is almost no room for judgment. The manager therefore first makes the backup and success-log check the target of operational automation, launching on schedule and alerting operators only on failure—this reduces the nightly manual work and the breeding ground for the operator error of deleting the wrong target. However, "deleting the oldest generation" is irreversible, and a mistaken parameter could erase generations that are still needed, so the manager carves out the exceptions needing judgment: deletion runs automatically only when the retention policy (e.g., generation count and a successful latest backup) is satisfied, and cases in doubt are not auto-deleted but routed to operator confirmation. Next, the rare, non-routine work involving judgment and exception handling, such as replacing a production storage device, is not forced into automation but documented as a runbook spelling out steps, preconditions, checks, and the rollback and escalation contacts for failure. Because this device swap changes the production configuration, it is linked with the change-management process and executed after CAB approval, with a rollback procedure (the conditions and steps to revert to the old device) prepared beforehand and a double-check (declare the target device -> confirm with a second person) imposed on the dangerous removal step. Afterward, configuration management (CMDB) is updated with the new device's information to prevent drift between ledger and reality. Separating "routine/repetitive to automation, judgment/exceptions to runbook plus people, configuration changes paired with change management and rollback" is the judgment of operational design.
| Nature of work | Primary owner | Key point |
|---|---|---|
| Routine, repetitive, fixed inputs | Automation (script/tool) | Alert humans only on failure; cut effort and error |
| Non-routine with judgment/exceptions | Human, following the runbook | Document rollback and escalation contacts |
| Changes to production configuration | Human plus approval, via change management | Pair rollback with a CMDB update |
Trap: "Operations should automate everything to cut manpower" is wrong—automating work with judgment or exception handling misbehaves on unexpected input and is dangerous, so automation is limited to routine, repetitive work as a rule. "An experienced operator needs no double-check or procedure" is a dangerous misconception—errors happen even to experts, and the more irreversible the operation (such as deletion), the more a double-check and confirmation prompt are needed. "Just do production-config changes quickly and defer the rollback procedure" is also wrong—without pairing a rollback procedure with a CMDB update, you cannot recover on failure and the ledger goes stale.
6.2.4Section summary
- Limit operational automation to routine, repetitive, fixed-input work; humans handle judgment/exceptions (forced automation misbehaves on unexpected input)
- A runbook documents normal steps plus rollback/escalation and is maintained at each configuration change; prevent errors on dangerous operations with double-checks
- Apply configuration changes via change management, pairing a rollback procedure with a CMDB update to enable recovery on failure and keep the ledger consistent
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. There is a nightly backup and log check performed by the same steps every night, and a once-a-month production storage-device replacement (involving judgment and rollback). Which separation of automation scope is most appropriate?
Q2. When an experienced operator manually deletes old-generation data from production storage, you want to prevent the loss of important data due to a wrong target. Which human-error-prevention measure is most appropriate?
Q3. You are planning to change middleware settings on the production system. There is a lesson from a past change that could not be rolled back and caused a prolonged outage. Which approach is most appropriate?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.