What's changed: In-scope coverage: added core service explanations
3.2Training, Overfitting, and Hyperparameter Tuning
Train well—understand overfitting/underfitting, bias/variance, regularization/early stopping, hyperparameter tuning (SageMaker Automatic Model Tuning), and cross-validation.
Train the model so it generalizes (performs on unseen data). The key is balancing overfitting and underfitting, diagnosed from the gap between training and validation performance.
3.2.1Overfitting and underfitting
- Underfitting (high bias): both train and val are poor. Fix via more features, bigger model, longer training.
- Overfitting (high variance): train high, val low. Fix via regularization, dropout, more data, early stopping.
- Hyperparameter tuning: search learning rate, tree depth, regularization, etc. with SageMaker Automatic Model Tuning.
- Cross-validation: split data and evaluate multiple times to estimate generalization reliably.
Common on MLS-C01: train good/val bad = overfitting (regularization/more data/early stopping), both bad = underfitting (bigger model/more features), and automatic hyperparameter search = SageMaker Automatic Model Tuning (Bayesian optimization). Remember overfitting = variance, underfitting = bias.
L1 regularization can zero out coefficients (feature selection); L2 shrinks coefficients to reduce overfitting.
MLS-C01 tests the hyperparameter-search strategies and XGBoost/neural-net-specific tuning. SageMaker Automatic Model Tuning (AMT) offers random search (highly parallel, broad), grid search (exhaustive over discrete candidates), Bayesian optimization (infers promising regions from past trials for efficient convergence), and Hyperband (early-stops low-promise configs to focus compute)—prefer Bayesian when there are many continuous values and trials are costly, and Hyperband when iterative training allows early stopping. Specify ranges on a log scale (e.g., learning rate where orders of magnitude matter) and use warm start to carry over prior tuning. XGBoost’s key HPs are max_depth (deeper = more overfit), eta (learning rate), subsample/colsample_bytree (sampling curbs overfit), min_child_weight/gamma (split conservativeness), alpha/lambda (L1/L2 regularization), and num_round (rounds + early stopping); for overfitting, lower depth, apply subsampling, and increase regularization. For neural nets, the main HPs are learning rate, batch size, epochs, dropout rate, and layers/units—too-high a learning rate diverges, too-low converges slowly. Estimate generalization with k-fold cross-validation (time-ordered for time series); CV matters more the less data you have. Read learning curves (train/val score vs data size or epochs): both low = underfitting (bias), large gap = overfitting (variance), leading to bigger model/more features for bias and more data/regularization/early stopping for variance.
| Search strategy | Characteristic | Fits when |
|---|---|---|
| Grid search | Exhaustive over discrete grid | Few discrete candidates |
| Random search | Broad, highly parallel | Many dims, limited budget |
| Bayesian optimization | Infers promising regions | Continuous, costly trials |
| Hyperband | Early-stops weak configs | Iterative training, early-stoppable |
Scenario: XGBoost’s validation AUC is much lower than training AUC—overfitting. → Lower max_depth, apply subsample/colsample_bytree sampling, and strengthen regularization with alpha/lambda (L1/L2) and gamma/min_child_weight. Set num_round high and use early stopping (stop when the validation metric stops improving) to halt at the optimum. Search these jointly with AMT Bayesian optimization.
FAQ: Q. Bayesian vs Hyperband? A. Bayesian for many continuous values with costly trials; Hyperband when iterative training allows early stopping. Q. How to range the learning rate? A. On a log scale, since orders of magnitude matter. Q. Validation unstable with little data? A. Use k-fold cross-validation and average (time-ordered for time series).
Trap: "fix overfitting by adding epochs" is false—training longer can worsen overfitting. For overfitting use more data, regularization, dropout, and early stopping. More epochs or a bigger model help underfitting (high bias). Diagnose via the train/validation gap (learning curves) and match the remedy to the symptom.
3.2.2Section summary
- Overfitting = regularization/early stopping/more data; underfitting = bigger model
- Tuning = SageMaker Automatic Model Tuning; evaluation = cross-validation
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Very high accuracy on training data but much lower on validation. What is happening?
Q2. Which is NOT an appropriate way to reduce overfitting?
Q3. Efficiently auto-search hyperparameters (learning rate, tree depth). What?