What's changed: In-scope coverage: deployment/runtime/IaC services
3.1Choosing Inference Options
Understand SageMaker inference options (real-time, serverless, asynchronous, batch transform) and choosing by traffic/latency. The starting point for "Deployment and Orchestration" in MLA-C01.
A trained model becomes valuable only when serving inference (predictions). SageMaker offers multiple inference options by traffic/latency needs.
3.1.1Four inference options
- Real-time: an always-on endpoint with low latency; for steady traffic.
- Serverless: scales to zero when idle; cost-efficient for intermittent traffic.
- Asynchronous: handles large payloads / long processing; queues requests and returns results.
- Batch transform: no endpoint; scores an entire dataset offline in bulk.
Common on MLA: steady low latency = real-time, intermittent and cost-sensitive = serverless, large input/long-running = async, bulk offline = batch transform. Choose by latency/traffic/payload.
Choose inference by four axes: "latency, traffic, payload, cost." Real-time uses an always-on endpoint for consistent low latency, follows load via Auto Scaling, hosts many small models with a multi-model endpoint (MME), and co-locates different models via multi-container/inference pipelines to cut cost. Serverless inference scales to zero when idle—ideal for intermittent traffic (with cold starts). Asynchronous inference queues large payloads (up to 1GB) or long-running jobs and can scale to zero when idle. Batch transform needs no endpoint and scores an entire dataset offline in bulk. Real-time options support A-B via weighted production variants, and Inference Recommender helps find the best instance/scaling settings. The axes: "steady/low-latency = real-time," "intermittent/cost = serverless," "large input/long-running = async," "bulk/offline = batch transform."
| Requirement | Inference option |
|---|---|
| Steady traffic, low latency | Real-time |
| Intermittent, no idle cost | Serverless |
| Large input, long processing | Asynchronous |
| Bulk, offline | Batch transform |
Scenario: (1) instant recommendations in a web app, (2) overnight scoring of all customers, (3) analyzing large video files. (1) real-time (Auto Scaling; serverless if intermittent), (2) batch transform (whole dataset, no endpoint), (3) asynchronous (large payloads/long-running via a queue). Pick a different option per use case.
Q. Steady/low-latency? Real-time. Q. Intermittent, avoid idle cost? Serverless. Q. Large input/long-running? Async. Q. Bulk offline? Batch transform. Q. Host many models cheaply? Multi-model endpoint.
Watch the mix-ups: (1) Batch transform ≠ asynchronous inference—batch scores whole datasets offline; async handles individual requests with large payloads/long runtimes. (2) Serverless has cold starts—unfit for strict low-latency SLAs. (3) Real-time bills continuously—use serverless for intermittent. (4) Async/serverless can scale to zero; real-time keeps at least one instance.
Real-time endpoints can auto-scale with load; multi-model endpoints host many models on one endpoint to cut cost.
3.1.2Section summary
- Choose real-time/serverless/async/batch transform by requirement
- Follow load with Auto Scaling; many models via multi-model endpoints
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You need low-latency predictions under steady traffic. Which inference option fits best?
Q2. You want to score an entire dataset in bulk overnight (no real-time need). Which fits best?
Q3. Traffic is intermittent and you do not want cost while idle. Which inference option fits best?