What's changed: Initial: 5 sections for Domain 2 (implementation and integration)
2.2Model Deployment Strategies
Learn FM deployment matched to app needs: on-demand invocation with Lambda, Bedrock provisioned throughput, SageMaker endpoints, containers (GPU/memory/token throughput), and model cascading.
Deploying FMs has challenges beyond traditional ML inference (huge models, GPUs, token processing, cold starts). Choose the method by traffic profile and latency needs.
2.2.1Choosing a deployment method
- Lambda on-demand invocation: for sporadic/bursty traffic; call Bedrock on-demand, elastic and serverless.
- Bedrock provisioned throughput: for steady high throughput or strict latency SLAs; reserve capacity.
- SageMaker endpoints: host self-managed/custom models or hybrid setups; real-time/async/batch.
- Model cascading: route easy queries to small/cheap models and only hard ones to large models—balancing cost and quality.
Common: sporadic traffic = Lambda on-demand, steady high throughput / latency SLA = provisioned throughput, host custom models = SageMaker endpoints, cheap small model for easy queries = model cascading.
Decide deployment by the kind of FM and the traffic profile. Call Bedrock managed FMs on-demand (elastic/pay-per-use) or via provisioned throughput (reserved capacity, stable latency), exposing the app from Lambda/ECS/Fargate through API Gateway. Host self-managed/open-weight models or special runtimes on SageMaker endpoints (real-time/async/batch transform), designing containers for GPU utilization, memory, and token-processing capacity and using model-loading strategies to reduce cold starts. For cost, use model cascading (route easy queries to small models) and task-specific small pre-trained models. Rather than running one giant model at full capacity 24/7, the professional pattern is to be elastic with traffic and tier models by difficulty.
| Traffic/need | Choose | Why |
|---|---|---|
| Sporadic/bursty | Lambda on-demand | Elastic, pay-per-use |
| Steady high throughput | Provisioned throughput | Reserved capacity, stable latency |
| Custom/self-managed model | SageMaker endpoints | Real-time/async/batch |
| Balance cost and quality | Model cascading | Tier models by difficulty |
Trap: “always serve all traffic with provisioned throughput” is wrong—on-demand is cheaper for sporadic traffic; reserve capacity only for steady high load. Match the method to the requirement.
2.2.2Section summary
- Sporadic = Lambda on-demand / steady = provisioned throughput / custom = SageMaker
- Cost-optimal = model cascading (tier models by difficulty)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to handle sporadic Bedrock inference requests from an internal tool elastically without managing servers. Best option?
Q2. For a high-traffic production app, you need stable throughput on Bedrock that meets a strict latency SLA. Best option?
Q3. Most queries are simple but a few need advanced reasoning. Which deployment strategy keeps cost low while preserving quality?