What's changed: Created Professional Machine Learning Engineer Chapter 4 (Domain 4 "Serving": batch/online inference (Agent Platform/Dataflow/BigQuery ML/Dataproc), frameworks (PyTorch/XGBoost), Model Registry on Gemini Enterprise Agent Platform, A/B testing (traffic splitting); Feature Store on Gemini Enterprise Agent Platform online serving, public/private endpoints, hardware (CPU/GPU/TPU/edge), Vertex AI Prediction autoscaling/containerized serving, latency/throughput optimization via quantization/distillation/pruning).
4.2Scaling online model serving
Understand Feature Store on Gemini Enterprise Agent Platform, Agent Platform public and private endpoints, choosing appropriate hardware (CPU/GPU/TPU/edge), scaling the serving backend by throughput (Vertex AI Prediction, containerized serving), and tuning models for training/serving in production (simplification, optimizing performance/latency/memory/throughput).
The crux of online serving is scaling "fast, cheap, and reliably." Combine endpoints, features, hardware, and optimization.
4.2.1Endpoints and features
Deploy online inference to Agent Platform endpoints. Choose a public endpoint for internet access or a private endpoint (Private Service Connect/private) to keep it inside the VPC with low latency. For real-time inference fetching fresh features at low latency, use Feature Store on Gemini Enterprise Agent Platform online serving, using the same features as training to prevent skew. Map "low-latency inference inside the VPC = private endpoint" and "fetch fresh features at low latency for inference = Feature Store online serving."
4.2.2Scaling and optimization
Scale serving by throughput: use Vertex AI Prediction autoscaling (min/max replicas, concurrency) or containerized serving (custom containers) to follow demand. Choose CPU/GPU/TPU/edge hardware by inference load. To cut production latency/cost, optimize the model with simplification techniques (quantization, distillation, pruning), batching, and the right accelerator, optimizing performance/latency/memory/throughput. Map "autoscale to follow demand = Vertex AI Prediction" and "cut latency/cost = quantization/distillation/pruning."
Common: requirement → means. E.g., "keep inference inside the VPC, low latency/private" = private endpoint; "fetch fresh features at low latency for inference" = Feature Store online serving; "autoscale to demand" = Vertex AI Prediction autoscaling; "reduce latency/model size" = quantization/distillation/pruning; "serve with custom dependencies" = containerized serving.
Watch the mix-ups: (1) Choose public (internet) vs private (in-VPC) endpoints by security/latency. (2) Do not confuse Feature Store online (low-latency, serving) vs offline (bulk, training). (3) Accuracy vs latency/cost is a trade-off—balance with simplification techniques.
4.2.3Section summary
- Endpoints = public/private (in-VPC, low-latency/private); features at inference = Feature Store online serving
- Follow demand = Vertex AI Prediction autoscaling/containerized serving; pick hardware by load
- Optimize latency/cost = simplification techniques (quantization/distillation/pruning)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To keep online inference inside the VPC, not exposed to the internet, with low latency, which is best?
Q2. For real-time inference, to fetch fresh features at low latency using the same values as training to prevent skew, which is best?
Q3. To autoscale the serving backend as inference request volume changes, which is best?
Q4. To reduce inference latency and model size in production, which is a representative simplification technique?
Q5. To serve a model with custom dependency libraries and custom pre/post-processing, which is best?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.