What's changed: Created Professional Machine Learning Engineer Chapter 4 (Domain 4 "Serving": batch/online inference (Vertex AI/Dataflow/BigQuery ML/Dataproc), frameworks (PyTorch/XGBoost), Vertex AI Model Registry, A/B testing (traffic splitting); Vertex AI Feature Store online serving, public/private endpoints, hardware (CPU/GPU/TPU/edge), Vertex AI Prediction autoscaling/containerized serving, latency/throughput optimization via quantization/distillation/pruning).
4.1Serving models
Understand batch and online inference (Vertex AI, Dataflow, BigQuery ML, Dataproc), serving with frameworks (PyTorch, XGBoost, etc.), organizing a model registry, and A/B testing model versions.
Serving turns a trained model into value. Choose the inference mode that fits the use, manage versions, and switch safely.
4.1.1Batch and online inference
Choose inference by use. For low-latency, one-at-a-time responses, use online inference (Vertex AI endpoints); for periodic bulk processing of large data, use batch inference (Vertex AI batch prediction; Dataflow for large transforms; BigQuery ML ML.PREDICT if data is in BigQuery; Dataproc for existing Spark). Serve with the training framework (PyTorch, XGBoost, TensorFlow) or a custom container. Map "instant single-item response = online inference," "periodic bulk = batch inference," and "predict directly on BigQuery data = BigQuery ML."
Continue reading — free sign-up
You're reading the free preview. Sign up free to read this section in full, plus every chapter (including 4+) and all questions.