What's changed: Created Professional Machine Learning Engineer Chapter 3 (Domain 3 "Scaling": framework/architecture choice, interpretability, training data org, file ingestion, training SDKs (Vertex AI custom training/Kubeflow on GKE/AutoML/Tabular Workflows), distributed training, hyperparameter tuning (Vizier), troubleshooting training failures, fine-tuning foundation models; hardware (CPU/GPU/TPU/edge), TPU/GPU distributed training (Reduction Server/Horovod/TPU Pod)).
3.2Choosing training hardware
Understand evaluating compute and accelerator options (CPU, GPU, TPU, edge devices) and distributed training with TPUs and GPUs (Reduction Server on Vertex AI, Horovod).
Training speed and cost hinge on hardware choice. Pick the accelerator and distribution that fit the workload.
3.2.1Choosing accelerators
Choose accelerators by workload: CPU for light/preprocessing or small scale; GPU for general deep learning (diverse frameworks, small–mid scale); cost-efficient TPU for large matrix-heavy TensorFlow/JAX training; and edge devices (Coral/Edge TPU) for on-device inference. Evaluate by requirements (throughput, latency, cost, framework support). Map "large matrix-heavy TF/JAX = TPU," "general deep learning, broad frameworks = GPU," and "on-device inference = edge."
3.2.2Distributed training
Scale beyond a single device with distributed training on TPU/GPU. In multi-GPU/node data parallelism, gradient aggregation (all-reduce) is the bottleneck; Vertex AI Reduction Server offloads aggregation to dedicated nodes to speed communication. For framework-agnostic distribution, use Horovod. Scale large TPUs with TPU Pods. Map "speed up gradient aggregation in GPU distribution = Reduction Server" and "framework-agnostic distribution = Horovod."
Common: workload → hardware. E.g., "large cost-efficient TensorFlow/JAX training" = TPU; "deep learning across diverse frameworks (e.g., PyTorch)" = GPU; "on-device/offline inference" = edge (Edge TPU/Coral); "speed up gradient aggregation across GPUs" = Reduction Server; "framework-agnostic distribution" = Horovod; "light preprocessing/small scale" = CPU.
Watch the mix-ups: (1) TPU suits large matrix-heavy TF/JAX; GPU supports broad frameworks—choose by framework. (2) Reduction Server speeds GPU data-parallel aggregation, not TPU-to-TPU. (3) Edge is for inference—not large-scale training.
3.2.3Section summary
- Large TF/JAX = TPU; general deep learning = GPU; on-device inference = edge; light/small = CPU
- Speed GPU data-parallel aggregation = Reduction Server; framework-agnostic distribution = Horovod
- Evaluate by throughput/latency/cost/framework support
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To train a large TensorFlow model (matrix-heavy) cost-efficiently, which accelerator is best?
Q2. To train deep learning across diverse frameworks (incl. PyTorch) at small–mid scale generally, which accelerator?
Q3. In multi-GPU data-parallel training, gradient aggregation (all-reduce) communication is the bottleneck. Which is best?
Q4. To run inference on-device (offline) in the field with unstable networks, which is best?
Q5. To implement multi-node distributed training in a framework-agnostic way, which is a common option?