What's changed: Created Professional Machine Learning Engineer Chapter 3 (Domain 3 "Scaling": framework/architecture choice, interpretability, training data org, file ingestion, training SDKs (Vertex AI custom training/Kubeflow on GKE/AutoML/Tabular Workflows), distributed training, hyperparameter tuning (Vizier), troubleshooting training failures, fine-tuning foundation models; hardware (CPU/GPU/TPU/edge), TPU/GPU distributed training (Reduction Server/Horovod/TPU Pod)).
3.1Building and training models
Understand choosing ML framework and model architecture, modeling techniques per interpretability requirements, organizing training data (tabular/text/speech/image/video), ingesting file types (CSV/JSON/images/Hadoop/DB), training with SDKs (Vertex AI custom training, Kubeflow on GKE, AutoML, Tabular Workflows), reliable distributed training, hyperparameter tuning, troubleshooting training failures, and fine-tuning foundation models.
This grows a prototype into a production-quality model. Choose framework and architecture, build training that scales, and lift accuracy with tuning and debugging.
3.1.1Model choice and training SDKs
Choose the ML framework (TensorFlow/PyTorch/JAX/sklearn/XGBoost) and model architecture by problem. If regulation/accountability requires interpretability, prefer explainable methods (linear models, boosted trees) or pair with Explainable AI. Pick the training implementation by requirement: Vertex AI custom training for flexible own code (custom containers ok), Kubeflow on GKE for portable Kubernetes pipelines, AutoML for no-code, and Tabular Workflows for advanced tabular control. Organize training data in Cloud Storage/BigQuery and ingest diverse files (CSV/JSON/images/Hadoop/DB). Map "flexible training with own code = Vertex AI custom training" and "no-code = AutoML."
3.1.2Distributed training, tuning, and debugging
Build large data/models into reliable pipelines with distributed training (data/model parallelism). Lift accuracy with hyperparameter tuning (Vertex AI Vizier/Hyperparameter Tuning, e.g., Bayesian optimization). When training fails/stalls, troubleshoot: OOM → revisit batch size/accelerators; NaN loss → learning rate/data anomalies; overfitting → regularization/early stopping. To adapt a foundation model to your task, use fine-tuning (PEFT, etc., on Vertex AI/Model Garden) to tune efficiently from small data rather than retraining. Map "auto search to raise accuracy = hyperparameter tuning" and "adapt a foundation model to a task = fine-tuning."
Common: requirement → means. E.g., "run own training code/containers flexibly" = Vertex AI custom training; "no-code training" = AutoML; "portable Kubernetes pipeline" = Kubeflow on GKE; "search params to raise accuracy" = hyperparameter tuning (Vizier); "adapt a foundation model to your task (small data)" = fine-tuning; "accountability required" = interpretable models/Explainable AI.
Watch the mix-ups: (1) Fine-tuning (retune the model on small data) vs RAG (provide context via retrieval) differ—task adaptation = fine-tuning; fresh knowledge = RAG. (2) Hyperparameter tuning is compute-costly—design the search space and trial count. (3) OOM/NaN/overfitting have different causes—triage by symptom.
3.1.3Section summary
- Training SDKs = Vertex AI custom training (flexible)/Kubeflow on GKE/AutoML (no-code)/Tabular Workflows
- Large = distributed training; raise accuracy = hyperparameter tuning; triage failures by symptom
- Adapt a foundation model = fine-tuning (fresh knowledge = RAG)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To run training jobs flexibly with your own TensorFlow/PyTorch code or custom containers, which is best?
Q2. To efficiently search hyperparameters (learning rate, layers) to raise accuracy, which is best?
Q3. To adapt an existing foundation model (LLM) to a specific task with a small amount of your data, which is best?
Q4. Training fails with GPU out-of-memory (OOM). What should you try first?
Q5. For compliance, you must be able to explain the model's decisions. Which approach is best?