What's changed: Created Professional Machine Learning Engineer Chapter 6 (Domain 6 "Monitoring": secure AI systems (theft/poisoning defense), Google Responsible AI (bias monitoring), fairness/readiness assessment, Vertex Explainable AI (feature attributions); Model Monitoring on Gemini Enterprise Agent Platform continuous evaluation, training/serving skew, feature attribution drift, performance vs baselines/simpler models/over time, common training/serving error monitoring).
6.2Monitoring and troubleshooting
Understand establishing continuous evaluation metrics (Model Monitoring on Gemini Enterprise Agent Platform, Explainable AI), monitoring for training-serving skew, monitoring for feature attribution drift, monitoring model performance against baselines/simpler models/over time, and monitoring for common training and serving errors.
Production models degrade quietly. Monitor continuously to detect skew and drift early and trigger retraining or correction.
6.2.1Continuous evaluation and skew/drift
Continuously evaluate production models with Model Monitoring on Gemini Enterprise Agent Platform. Two key targets: training/serving skew (input distributions differ between training and serving) and feature attribution/prediction drift (input distributions/contributions change over time). Detect skew by comparing "training vs serving data," and drift by comparing "past vs current serving data." On exceeding thresholds, alert and trigger retraining. Map "training vs prod input mismatch = training/serving skew" and "prod input change over time = drift."
6.2.2Performance monitoring and errors
Monitor model performance against a baseline (prior version or simple rules/model) and over time—continuously confirm the new model truly improves and does not degrade. When labels are delayed, Explainable AI attribution changes can serve as proxies. Also monitor common training/serving errors: input-schema mismatches, missing/anomalous values, latency spikes, dependent-service failures. Map "confirm improvement vs old/simple models = baseline comparison" and "detect prod input/output anomalies = error monitoring."
Common: symptom → means. E.g., "input distribution differs training vs serving" = training/serving skew monitoring; "prod input changes over time" = drift monitoring; "continuously evaluate prod model and alert" = Model Monitoring on Gemini Enterprise Agent Platform; "confirm the new model is truly better" = compare vs baselines/simpler models; "proxy for degradation when labels are delayed" = feature attribution change.
Watch the mix-ups: (1) Skew (training vs serving difference) vs drift (change over time) target different things. (2) Delayed labels prevent direct performance measurement—use proxies like attribution change. (3) Unmonitored production misses degradation—always set Model Monitoring with threshold alerts.
6.2.3Section summary
- Continuous prod evaluation = Model Monitoring on Gemini Enterprise Agent Platform; targets = training/serving skew and feature/prediction drift
- Compare performance vs baselines/simpler models/over time; use attribution change as proxy when labels are delayed
- Monitor common training/serving errors (schema mismatch/anomalies/latency) and alert→retrain
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To continuously evaluate a production model and alert on input-distribution change or performance degradation, which is best?
Q2. The distribution of training data differs from the input data arriving at serving time. What is this?
Q3. The distribution of incoming input data and feature contributions change over time in production. What is this?
Q4. To judge whether a new model truly improves, against what should you compare?
Q5. In production, labels arrive late so you can't measure performance immediately. Which is a useful proxy for degradation?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.