What's changed: Created Professional Machine Learning Engineer Chapter 4 (Domain 4 "Serving": batch/online inference (Vertex AI/Dataflow/BigQuery ML/Dataproc), frameworks (PyTorch/XGBoost), Vertex AI Model Registry, A/B testing (traffic splitting); Vertex AI Feature Store online serving, public/private endpoints, hardware (CPU/GPU/TPU/edge), Vertex AI Prediction autoscaling/containerized serving, latency/throughput optimization via quantization/distillation/pruning).
4.2Scaling online model serving
Understand Vertex AI Feature Store, Vertex AI public and private endpoints, choosing appropriate hardware (CPU/GPU/TPU/edge), scaling the serving backend by throughput (Vertex AI Prediction, containerized serving), and tuning models for training/serving in production (simplification, optimizing performance/latency/memory/throughput).
The crux of online serving is scaling "fast, cheap, and reliably." Combine endpoints, features, hardware, and optimization.
4.2.1Endpoints and features
Deploy online inference to Vertex AI endpoints. Choose a public endpoint for internet access or a private endpoint (Private Service Connect/private) to keep it inside the VPC with low latency. For real-time inference fetching fresh features at low latency, use Vertex AI Feature Store online serving, using the same features as training to prevent skew. Map "low-latency inference inside the VPC = private endpoint" and "fetch fresh features at low latency for inference = Feature Store online serving."
4.2.2Scaling and optimization
Scale serving by throughput: use Vertex AI Prediction autoscaling (min/max replicas, concurrency) or containerized serving (custom containers) to follow demand. Choose CPU/GPU/TPU/edge hardware by inference load. To cut production latency/cost, optimize the model with simplification techniques (quantization, distillation, pruning), batching, and the right accelerator, optimizing performance/latency/memory/throughput. Map "autoscale to follow demand = Vertex AI Prediction" and "cut latency/cost = quantization/distillation/pruning."
Continue reading — free sign-up
You're reading the free preview. Sign up free to read this section in full, plus every chapter (including 4+) and all questions.