What's changed: In-scope coverage: deployment/runtime/IaC services
3.1Choosing Inference Options
Understand SageMaker inference options (real-time, serverless, asynchronous, batch transform) and choosing by traffic/latency. The starting point for "Deployment and Orchestration" in MLA-C01.
A trained model becomes valuable only when serving inference (predictions). SageMaker offers multiple inference options by traffic/latency needs.
3.1.1Four inference options
- Real-time: an always-on endpoint with low latency; for steady traffic.
- Serverless: scales to zero when idle; cost-efficient for intermittent traffic.
- Asynchronous: handles large payloads / long processing; queues requests and returns results.
- Batch transform: no endpoint; scores an entire dataset offline in bulk.
Common on MLA: steady low latency = real-time, intermittent and cost-sensitive = serverless, large input/long-running = async, bulk offline = batch transform. Choose by latency/traffic/payload.
Choose inference by four axes: "latency, traffic, payload, cost." Real-time uses an always-on endpoint for consistent low latency, follows load via Auto Scaling, hosts many small models with a multi-model endpoint (MME), and co-locates different models via multi-container/inference pipelines to cut cost. Serverless inference scales to zero when idle—ideal for intermittent traffic (with cold starts). Asynchronous inference queues large payloads (up to 1GB) or long-running jobs and can scale to zero when idle. Batch transform needs no endpoint and scores an entire dataset offline in bulk. Real-time options support A-B via weighted production variants, and Inference Recommender helps find the best instance/scaling settings. The axes: "steady/low-latency = real-time," "intermittent/cost = serverless," "large input/long-running = async," "bulk/offline = batch transform."
| Requirement | Inference option |
|---|---|
| Steady traffic, low latency | Real-time |
| Intermittent, no idle cost | Serverless |
| Large input, long processing | Asynchronous |
| Bulk, offline | Batch transform |
Scenario: (1) instant recommendations in a web app, (2) overnight scoring of all customers, (3) analyzing large video files. (1) real-time (Auto Scaling; serverless if intermittent), (2) batch transform (whole dataset, no endpoint), (3) asynchronous (large payloads/long-running via a queue). Pick a different option per use case.
Q. Steady/low-latency? Real-time. Q. Intermittent, avoid idle cost? Serverless. Q. Large input/long-running? Async. Q. Bulk offline? Batch transform. Q. Host many models cheaply? Multi-model endpoint.
Watch the mix-ups: (1) Batch transform ≠ asynchronous inference—batch scores whole datasets offline; async handles individual requests with large payloads/long runtimes. (2) Serverless has cold starts—unfit for strict low-latency SLAs. (3) Real-time bills continuously—use serverless for intermittent. (4) Async/serverless can scale to zero; real-time keeps at least one instance.
Real-time endpoints can auto-scale with load; multi-model endpoints host many models on one endpoint to cut cost.
3.1.2Section summary
- Choose real-time/serverless/async/batch transform by requirement
- Follow load with Auto Scaling; many models via multi-model endpoints
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You need low-latency predictions under steady traffic. Which inference option fits best?
Q2. You want to score an entire dataset in bulk overnight (no real-time need). Which fits best?
Q3. Traffic is intermittent and you do not want cost while idle. Which inference option fits best?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.