What's changed: In-scope coverage: added core service explanations
3.3Model Evaluation Metrics
Measure quality—understand the confusion matrix, precision/recall/F1, ROC-AUC, regression metrics (RMSE/MAE/R²), and metric choice under imbalance. Evaluate with the right metric.
Measure model quality with the right metric. Metrics differ for classification vs regression, and crucially, accuracy is misleading under class imbalance.
3.3.1Classification and regression metrics
- Precision: of predicted positives, the fraction truly positive. Prioritize when false positives are costly.
- Recall: of actual positives, the fraction detected. Prioritize when misses are critical (fraud/disease).
- F1 / ROC-AUC: F1 is the harmonic mean of precision/recall; AUC is a threshold-free overall measure.
- Regression metrics: RMSE (penalizes large errors)/MAE/R². For imbalanced classification, avoid accuracy.
Common on MLS-C01: reduce misses (fraud/disease) = prioritize recall, reduce false alarms = prioritize precision, accuracy is wrong for imbalanced data → F1/AUC, and penalize large errors = RMSE. Classic example: with 99% negatives, "predict all negative" still gets 99% accuracy—meaningless.
MLS-C01 probes tying metrics to business cost, threshold tuning, and ROC vs PR. Most classifiers output probabilities, and moving the decision threshold trades off precision and recall (lowering it raises recall, lowers precision). The ROC curve (FPR on x, TPR on y) and its area AUC summarize this trade-off, but under strong imbalance ROC-AUC tends to look optimistic, so the PR curve (precision–recall) and PR-AUC better reflect reality. Choose the metric by cost: prioritize recall where misses (false negatives) are critical (medical diagnosis, fraud), precision where false positives are costly (spam, credit), F1 when you need balance, and F-beta (β reweights recall vs precision) when the false-positive/false-negative cost ratio is skewed. For multiclass, distinguish macro average (equal per class) from weighted (count-weighted). For regression, choose RMSE (squares errors—penalizes big misses, same unit as target), MAE (absolute error—robust to outliers), MAPE (percentage error—scale-free but unstable near zero), and R² (fraction of variance explained) based on outlier handling and interpretability. Always evaluate on held-out/cross-validation data not used in training, set the threshold on validation, and confirm on test. In SageMaker, emit the objective metric to CloudWatch during training so AMT searches to maximize/minimize it.
| Situation/goal | Metric | Why |
|---|---|---|
| Misses critical (fraud/disease) | Recall | Reduce missed true positives |
| False alarms costly (spam/credit) | Precision | Raise correctness of positive predictions |
| Overall under strong imbalance | PR-AUC / F1 | ROC-AUC tends to be optimistic |
| Regression, penalize big errors | RMSE | Squares errors |
| Regression, robust to outliers | MAE | Absolute error, less sensitive |
Scenario: in fraud detection (0.3% positives), ROC-AUC is a high 0.97, yet in production there are too many false alarms and operations break down. → Under strong imbalance ROC-AUC looks optimistic. Re-evaluate with the PR curve/PR-AUC and pick an operable threshold from the precision–recall trade-off. Tune F-beta or the threshold to the miss-vs-false-alarm cost ratio, set it on validation, and confirm on test.
FAQ: Q. ROC-AUC vs PR-AUC? A. ROC for roughly balanced classes; PR reflects reality better under strong imbalance. Q. Raise precision and recall together? A. Usually a trade-off—tune via threshold, optimize F1/F-beta if you need both. Q. RMSE vs MAE? A. RMSE to weight big errors heavily, MAE to be robust to outliers.
Trap: "high ROC-AUC means you’re safe even on imbalanced data" is false—abundant true negatives can inflate ROC-AUC and mask missed minority positives or excessive false alarms. Evaluate with PR-AUC and goal-appropriate precision/recall/F-beta. "High accuracy = good model" is also wrong under imbalance.
3.3.2In-scope services for modeling
The core of modeling is Amazon SageMaker AI, which supports built-in algorithms, automatic tuning (AMT), distributed training, and various inference, integrating from data prep through training, tuning, deployment, and inference. To quickly stand up a custom deep-learning environment on EC2, AWS Deep Learning AMIs (DLAMI) provide major frameworks (TensorFlow/PyTorch) and GPU drivers/libraries (CUDA, etc.) preconfigured. For development/data assistance, use Amazon Q, a generative-AI assistant for coding and AWS questions.
3.3.3Section summary
- Classification = precision/recall/F1/AUC (avoid accuracy under imbalance)
- Regression = RMSE/MAE/R²; minimize misses = recall
- Core = SageMaker AI; custom DL env = DLAMI; dev assist = Amazon Q
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. In fraud detection, you want to minimize misses (failing to catch actual fraud). Which metric?
Q2. With 99% normal and 1% anomaly, predicting "all normal" still yields 99% accuracy. Use which metric instead?
Q3. For a regression model, you want to penalize large errors more strongly. Which metric?