What's changed: In-scope coverage: model dev/AI services
2.2Hyperparameter Tuning and Overfitting
Understand hyperparameter search with SageMaker Automatic Model Tuning, overfitting vs. underfitting, and remedies like regularization and adding data.
Model performance varies greatly with hyperparameters. Balancing overfitting (too much) and underfitting (too little) is key.
2.2.1Tuning and fit balance
- Automatic Model Tuning: search hyperparameters via Bayesian/random to optimize a chosen metric.
- Overfitting: fits training too well and generalizes poorly; remedies = regularization, dropout, more data, early stopping.
- Underfitting: model too simple/undertrained; remedies = more complexity, more features, more training.
Common on MLA: auto hyperparameter search = Automatic Model Tuning, high train but low test = overfitting (regularize/add data/early stop), both low = underfitting.
Tuning means "optimize performance via hyperparameters while balancing over/underfitting." SageMaker Automatic Model Tuning (AMT) defines a search space (learning rate, tree depth, regularization, etc.) and finds the combination that maximizes/minimizes a validation metric via Bayesian optimization (efficient with few trials) or random/grid/Hyperband (which early-stops unpromising trials). Understand failure modes via bias–variance: underfitting (high bias) is low on both train and test → add complexity/features/more training; overfitting (high variance) is high on train but low on test → use regularization (L1/L2), dropout, more data/augmentation, and early stopping. Diagnose with learning curves (train vs. validation), and when validation starts degrading, early stop to both curb overfitting and save cost. Note AMT launches many training jobs, so cost depends on parallelism and total jobs.
| Symptom | Condition | Remedy |
|---|---|---|
| Low on train and test | Underfit (high bias) | Add complexity/features |
| High train, low test | Overfit (high variance) | Regularize/more data/early stop |
| Find best hyperparameters | — | AMT (Bayesian) |
Scenario: 98% train vs. 72% test, unstable in production. Classic overfitting. Add L2 regularization and dropout, increase effective data with augmentation, and early stop via validation monitoring. Search hyperparameters with Automatic Model Tuning (Bayesian), using Hyperband to cut unpromising trials and control cost.
Q. Auto hyperparameter search? AMT (Bayesian). Q. High train, low test? Overfit → regularize/more data/early stop. Q. Both low? Underfit → add complexity/features. Q. Stop training efficiently? Early stopping. Q. Smartly cut trials? Hyperband.
Watch the mix-ups: (1) "Add complexity and keep training" worsens overfitting—regularize/add data/early stop. (2) Strengthening regularization worsens underfitting (even simpler). (3) Over-tuning on validation overfits to it—do final evaluation on untouched test. (4) AMT cost scales with job count × parallelism.
Early stopping—halting when validation performance stalls—both curbs overfitting and saves cost.
2.2.2Section summary
- Search hyperparameters with Automatic Model Tuning
- Overfit = regularize/add data/early stop; underfit = add complexity
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to auto-search the best hyperparameter combination. Which SageMaker feature?
Q2. High accuracy on training but much lower on test. What is the most likely condition?
Q3. Which is NOT an appropriate remedy for overfitting?