What's changed: In-scope coverage: monitoring/ops/cost/security services
4.1Model Monitoring and Drift
Understand drift detection with SageMaker Model Monitor, endpoint monitoring with CloudWatch, and retraining triggered by drift. The starting point for "Monitoring, Maintenance, and Security" in MLA-C01.
Production models degrade over time (drift). Continuous monitoring and retraining when needed are key operations.
4.1.1Monitoring and drift detection
- Model Monitor: compares production data to the training baseline to detect data drift / model-quality drift / bias drift.
- CloudWatch: monitors endpoint latency, error rate, invocations, and raises alarms.
- Retraining: trigger retraining/redeployment via Pipelines on drift detection or schedule.
Common on MLA: production accuracy degrades = data/model drift (detect with Model Monitor), detect → retrain (Pipelines), endpoint latency/error monitoring = CloudWatch. Drift comes from shifting data distributions.
Production models degrade over time, so design "what to monitor and what triggers retraining." SageMaker Model Monitor builds a baseline from training data and periodically evaluates deviation to detect four drift types: data-quality drift (input distribution shift = covariate shift), model-quality drift (accuracy drop vs. ground-truth labels), bias drift (changing attribute skew), and feature-attribution drift (changing importances). Endpoint health (latency, error rate, invocations, instance utilization) is monitored with CloudWatch, and request/response capture is saved to S3 via Data Capture. On degradation, automate retraining/redeployment with EventBridge→SageMaker Pipelines to keep the model fresh (linking to the prior chapter’s MLOps). The axes: "cause of accuracy drop = drift (Model Monitor)," "detect → retrain (Pipelines)," "endpoint ops metrics = CloudWatch." Since drift’s root cause is shifting data, use proxy metrics when labels arrive late.
| What to monitor | Use |
|---|---|
| Input/accuracy/bias change | Model Monitor (drift) |
| Latency/error/utilization | CloudWatch |
| Capture I/O | Data Capture → S3 |
| Retrain on degradation | EventBridge → Pipelines |
Scenario: a recommender that started accurate degrades over months. The cause is data/model-quality drift from shifting user preferences. Detect it via Model Monitor baseline comparison, and watch endpoint latency/errors with CloudWatch. Catch threshold breaches with EventBridge to trigger retraining in SageMaker Pipelines, deploying the new model gradually after the evaluation step passes.
Q. Cause of production accuracy drop? Data/model drift. Q. Detection? Model Monitor (baseline comparison). Q. Latency/error monitoring? CloudWatch. Q. Capture I/O? Data Capture. Q. Retrain after detection? EventBridge → Pipelines.
Watch the mix-ups: (1) Model Monitor = drift/quality, CloudWatch = endpoint ops metrics—don’t swap. (2) Model-quality drift needs ground-truth labels—use proxies/data-quality drift when labels lag. (3) Model Monitor only detects—automate retraining/rollback separately. (4) Without enabling Data Capture, you can’t analyze I/O afterward.
Model Monitor builds a "baseline" from training data and periodically evaluates how far production inputs/predictions deviate from it.
4.1.2Section summary
- Model Monitor (drift) + CloudWatch (endpoint monitoring)
- Connect drift detection to retraining (Pipelines)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. A model’s accuracy gradually drops after going live. What is the most common cause and detection tool?
Q2. You want to monitor endpoint latency/error rate and raise alarms. What do you use?
Q3. You want to auto-retrain/redeploy when drift is detected. Which fits best?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.