What's changed: Initial: 5 sections for Domain 2 (implementation and integration)
2.4FM API Integrations
Learn to integrate FM APIs robustly: sync/async (SQS), Bedrock streaming (WebSocket/SSE), exponential backoff/rate limiting/fallback, observability with X-Ray, and model routing.
FM calls are slow, can fail, and are token-billed. Integrate robustly with the right sync/async choice, streaming, retries, and routing.
2.4.1Robust API integration
- Sync/async: synchronous for immediate responses; offload long tasks to SQS async so callers are not blocked.
- Streaming: stream tokens incrementally with the Bedrock streaming API + WebSocket/SSE to reduce perceived latency.
- Resilience: handle 429/throttling with exponential backoff, rate-limit at API Gateway, and degrade gracefully via fallback.
- Observability: trace FM API calls across service boundaries with AWS X-Ray.
- Model routing: route to specialized FMs statically/dynamically by content (Step Functions/API Gateway transformations).
Common: stream for perceived speed = streaming (WebSocket/SSE), 429/throttling = exponential backoff + API Gateway rate limiting, long tasks = async via SQS, trace FM calls = X-Ray, best model by content = model routing.
Robust integration is measured by “do not block, do not drop, can trace.” Stream chat with the Bedrock streaming API + API Gateway (WebSocket/SSE, chunked transfer); offload long batches to SQS async processed by Lambda workers. Retry throttling (429) with AWS SDK exponential backoff + jitter, rate-limit clients at API Gateway, and degrade gracefully with a fallback model/cache under overload. Make failures visible by tracing across service boundaries with AWS X-Ray, pinpointing latency bottlenecks (embedding/retrieval/generation). Route statically via API Gateway request transformations for simple cases, or dynamically with Step Functions for context-dependent selection. Together these build an API layer resilient to latency, failure, and cost.
| Problem | Fix | Note |
|---|---|---|
| Feels slow | Streaming | Incremental via WebSocket/SSE |
| 429/throttling | Exponential backoff + rate limit | SDK + API Gateway |
| Long-running | Async via SQS | Free the caller |
| Find bottleneck | X-Ray | Trace across services |
Trap: “fix throttling by immediately maxing out retries” is wrong—retry with exponential backoff + jitter; the root fix is capacity (provisioned/Cross-Region). Also “just block synchronously for long generations” is wrong—offload async via SQS and stream.
2.4.2Section summary
- Do not block = streaming/SQS async / do not drop = backoff + rate limit + fallback
- Traceable = X-Ray / optimize = model routing
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. In a chat UI, you want to reduce perceived latency for long answers that take seconds to complete. Best approach?
Q2. During peaks, Bedrock returns 429 (throttling). What is the appropriate first client-side measure?
Q3. For a request spanning multiple microservices and Bedrock calls, you want to find which hop is the latency bottleneck. Best option?