What's changed: In-scope coverage: model dev/AI services
2.3Model Evaluation Metrics
Understand evaluation metrics for classification (accuracy, precision/recall, F1, AUC) and regression (RMSE, MAE, R-squared), the confusion matrix, and metric choice under imbalance.
Whether a model is "good" is measured by metrics suited to the problem. Classification and regression use different metrics, and imbalance needs care.
2.3.1Metrics by problem type
- Classification: accuracy, precision/recall, F1, AUC/ROC; understood via the confusion matrix (TP/FP/TN/FN).
- Regression: RMSE, MAE (error magnitude), R-squared.
- Imbalanced data: accuracy is misleading—favor precision/recall/F1/AUC.
- Precision vs. recall: emphasize recall (fewer misses) or precision (fewer false alarms) by use case.
Common on MLA: accuracy is wrong for imbalanced classification = use F1/AUC or precision/recall, reduce misses = favor recall, reduce false alarms = favor precision, regression = RMSE/MAE.
Always pick "metrics suited to the problem type." Classification rests on the confusion matrix (TP/FP/TN/FN): precision (of predicted positives, how many are correct) and recall (of actual positives, how many are caught) trade off, and their harmonic mean is F1. Sweeping the threshold gives ROC (→AUC) and PR curves (→PR-AUC); for imbalanced data, accuracy misleads, so use F1/PR-AUC (predicting all majority still scores high accuracy). Regression uses RMSE (penalizes large errors), MAE (robust to outliers), and R-squared. Emphasis shifts by use case: misses are critical (fraud/disease) → recall; false positives costly → precision. Multiclass uses macro/micro averaging, probability calibration uses log loss, and forecasting uses MAPE, etc. Tune the threshold to balance precision/recall to business needs, and do final evaluation on untouched test data.
| Goal | Metric |
|---|---|
| Reduce misses (fraud/disease) | Recall |
| Reduce false alarms | Precision |
| Overall on imbalanced classification | F1 / PR-AUC / AUC |
| Regression error | RMSE / MAE / R² |
Scenario: disease screening with 1% positives. Is a 99%-accuracy model good? "All negative" also scores 99%, so accuracy is inappropriate. Since misses (FN) are critical, prioritize recall and evaluate with PR-AUC/F1. Lower the threshold to raise recall, absorbing the added false positives (lower precision) with secondary testing—tune the threshold to business needs.
Q. Why is accuracy bad for imbalance? Majority-prediction scores high—use F1/PR-AUC. Q. Reduce misses? Recall. Q. Reduce false alarms? Precision. Q. Regression metrics? RMSE/MAE/R². Q. Adjust precision/recall? The threshold.
Watch the mix-ups: (1) Using accuracy on imbalance overstates performance—use F1/PR-AUC. (2) Don’t swap precision and recall (predicted-positive vs. actual-positive basis). (3) RMSE is outlier-sensitive, MAE robust—choose by goal. (4) Don’t mix classification metrics (F1/AUC) with regression metrics (RMSE/MAE).
For fraud or disease detection (misses are critical) favor recall; where false positives are costly favor precision. F1 balances both.
2.3.2In-scope services for model development and AI
The core of model development is Amazon SageMaker AI, which integrates data prep, training, tuning, deployment, and inference and governs the ML lifecycle via feature store/pipelines/model registry. For generative AI, use Amazon Bedrock (multi-provider foundation models via one serverless API), and for coding/ops AI assistance, Amazon Q. Purpose-built AI services that need no custom training are also in scope: text analysis = Amazon Comprehend, enterprise search = Amazon Kendra, recommendations = Amazon Personalize, online fraud detection = Amazon Fraud Detector, image/video recognition = Amazon Rekognition, speech synthesis = Amazon Polly. Anomaly detection: Amazon Lookout for Equipment (equipment) and others (note: Lookout for Metrics and Lookout for Vision are closed to new use and being retired; use CloudWatch Anomaly Detection for metrics and Rekognition Custom Labels for image defects instead). Health data lake = AWS HealthLake. Human review of predictions = Amazon Augmented AI (A2I), and ML-based code-quality review = Amazon CodeGuru.
2.3.3Section summary
- Classification = accuracy/precision/recall/F1/AUC; regression = RMSE/MAE/R²
- For imbalance, avoid accuracy; use F1/AUC, precision/recall
- Core = SageMaker AI; gen-AI = Bedrock/Q; purpose-built AI = Comprehend/Kendra/Personalize/Rekognition/Lookout/A2I and more
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. In a 99%-normal / 1%-abnormal imbalanced classification, why is accuracy a poor evaluation metric?
Q2. In fraud detection you most want to reduce misses (actual fraud not caught). Which metric to emphasize?
Q3. Which is an appropriate metric for a regression model predicting continuous values?