What's changed: Initial version
6.1System monitoring and job management
Covers monitoring items (CPU, memory, disk, response time) and the design of thresholds and alerts that catch anomalies early, job scheduling that protects the SLA, the dependencies and abnormal-end handling (abort, rerun, skip) of a job net that ties multiple jobs together, and two-stage warning/critical thresholds that suppress false alarms.
System operation begins with "catching anomalies before they become an SLA breach." A service manager must design which items to monitor at which thresholds so that problems are noticed while still just a sign, build a schedule that runs many jobs—such as an overnight batch—within the time window while preserving their dependencies, and decide in advance the handling that minimizes service impact when a job ends abnormally (abort, rerun, skip). Monitor too finely and an alert flood buries the real anomaly; too coarsely and it is too late—designing this balance is the core of the practice.
6.1.1Monitoring items, thresholds, and alerts
- Monitoring items are indicators of service health. Distinguish resource-level items (CPU utilization, memory, free disk, network bandwidth) from user-facing service-level items (response time, transaction success rate, error rate). If the SLA is stated in terms of "response time," you must monitor the response time the user actually experiences, not just CPU utilization, or you cannot directly detect the SLA breach.
- Thresholds are the boundary values at which an alert fires. Rather than a single threshold, the standard practice is a two-stage split into warning and critical. Warning lets you begin remediation with margin and prevent the SLA breach before reaching critical. Setting thresholds too tight causes false positives and frequent alerts, so operators become desensitized and miss the real ones (alert fatigue).
6.1.2Job scheduling and the job net
- Job scheduling automatically launches jobs (such as batch processing) according to their execution order, start time, and trigger (the normal completion of a preceding job, the arrival of a specific file, and so on). Reliably completing the overnight batch within the time window before business hours (the batch window) is a prerequisite for meeting the SLA.
- Job net is an execution unit that links multiple jobs by dependencies (predecessor/successor order). It defines chains such as launching job B only on the normal completion of job A. Unless you define how successors are treated when a predecessor job ends abnormally (stop or skip), successors may run on incomplete data and the failure cascades downstream.
- Abnormal-end handling means choosing, per situation, among (1) abort (stop successors, notify operators, and move to investigation or manual handling), (2) rerun (restart from the failed job after recovery), and (3) skip (pass over a low-impact job to run an important successor first). For rerun, the design of the restart point (checkpoints, re-runnability) is essential so that partially updated data is not double-counted.
Most-tested points: "if the SLA is response time, monitor response time itself," "use two-stage warning/critical thresholds to remediate with margin," "thresholds set too tight cause false positives and alert fatigue," "a job net is an execution unit linked by dependencies," and "on a predecessor failure, choose abort/rerun/skip by impact." Questions test judgment based on whether the batch can complete within its window.
6.1.3Designing thresholds and abnormal handling
Suppose a service manager operates the job net for a nightly sales-aggregation batch behind an online service. The SLA states "the previous day's sales report is available by 8 a.m.," and the batch window is 1 a.m. to 7 a.m. The job net depends in the order "A: extract -> B: aggregate -> C: generate report -> D: distribute," and C often slows down under capacity pressure. For monitoring design, the manager places a two-stage threshold of warning at 80% and critical at 90% on disk utilization, so that instead of first raising alarm only when a single threshold hits 90%, remediation—cleaning up temp files or adding capacity—can begin with margin at the 80% warning stage. Because the SLA is not response time but "completion by 8 a.m.," the manager also adds the delay of each job against its planned completion time (planned-vs-actual gap) itself as a monitoring item. Next, abnormal handling: if B (aggregate) ends abnormally, running successors C and D would distribute an incomplete aggregate, so abort is correct and skipping ahead would be wrong. Meanwhile, B itself is built with checkpoints so it is re-runnable, restarting after recovery from the point of interruption rather than from the very beginning, aiming to recover within the limited batch window. Conversely, if job E—which only builds a supplementary reference metric in the report, is independent of D, and is not essential to customer delivery—fails, the manager skips E to get the main D distribution out by 8 a.m. and completes E manually during the next business hours. Working backward from the SLA target (completion by 8 a.m.), setting two-stage thresholds that give early warning, and choosing abort/rerun/skip by each job's downstream impact is the judgment of operational design.
| Handling | What it does | When it fits |
|---|---|---|
| Abort | Stop successors, notify, investigate | When successors on incomplete data cause harm |
| Rerun | Restart from the checkpoint after recovery | Essential jobs to recover within the window |
| Skip | Pass over a low-impact job | Independent/non-essential; prioritize the main flow |
Trap: "One critical threshold is enough" is wrong—by the time critical is reached there is no margin to remediate and an SLA breach is likely, so the standard is to begin early with two-stage warning/critical. "Always skip and move on when a job fails so the service does not stop" is a dangerous misconception—an essential job with downstream impact, such as aggregation, must be aborted; skipping distributes incomplete data. "Even if the SLA is response time, monitoring CPU utilization alone suffices" is also wrong—you must monitor the response time the user actually experiences to detect the breach directly.
6.1.4Section summary
- Match monitoring items to the SLA measure (for a response-time SLA, response time itself); use two-stage warning/critical thresholds to remediate with margin
- Thresholds set too tight cause false positives and alert fatigue; a job net links jobs by dependency and aims to finish within the batch window
- On a predecessor failure, choose abort/rerun/skip by downstream impact (abort essential jobs with downstream impact; skip independent, non-essential ones)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. A service's SLA is defined as "response time within 3 seconds." The operations team monitored only CPU utilization, but an SLA breach occurred as response time worsened despite ample CPU headroom. Which correction to the monitoring design is most appropriate?
Q2. While operating the overnight-batch job net "A: extract -> B: aggregate -> C: generate report -> D: distribute," the aggregation job B ended abnormally partway through. The SLA requires the report by 8 a.m., and the batch window still has margin. Which abnormal-end handling is most appropriate?
Q3. For disk-utilization monitoring, only a single critical threshold at 90% was set; by the time 90% was detected there was no margin to remediate, the batch overflowed, and an SLA breach occurred. Which improvement to the threshold design is most appropriate?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.