What's changed: Initial: 3 sections for Domain 4 (operational efficiency and optimization)
4.3Monitoring Systems for GenAI Applications
Learn observability for FM apps: token usage/hallucination rate/response quality in CloudWatch, Bedrock Model Invocation Logs, anomaly detection, tool-calling observability, and hallucination detection with golden datasets.
GenAI has failure modes absent in traditional ML (hallucinations, response drift, token bursts). Monitor GenAI-specific quality/cost metrics in addition to operational ones.
4.3.1Observability building blocks
- Operational + GenAI metrics: track latency/errors plus token usage, prompt effectiveness, hallucination rate, and response quality in CloudWatch.
- Model Invocation Logs: analyze each request/response in detail with Bedrock Model Invocation Logs.
- Anomaly detection: detect token bursts/response drift, Cost Anomaly Detection, and performance benchmarks.
- GenAI-specific troubleshooting: detect hallucinations with a golden dataset, response consistency via output diffing, and logical errors via reasoning-path tracing.
Common: detailed request/response logs = Bedrock Model Invocation Logs, continuously detect hallucinations = compare against a golden dataset, monitor token usage/hallucination rate = CloudWatch custom metrics, tool-calling observability. X-Ray provides distributed tracing.
Monitoring layers “operational” and “GenAI-specific.” Watch latency, errors, and throughput in CloudWatch, and visualize GenAI-specific metrics (token usage, prompt effectiveness, hallucination rate, response quality, cost) via custom metrics/dashboards alongside business-impact metrics. For deep analysis, use Bedrock Model Invocation Logs (per-call I/O to S3/CloudWatch Logs) and trace across services with X-Ray. Catch anomalies with detection on token burst patterns and response drift, plus Cost Anomaly Detection. Detect GenAI-specific failures by comparing to a golden dataset (expected answers) for hallucinations, checking response consistency via output diffing, and isolating logical errors via reasoning-path tracing. For agents, monitor tool-calling observability (call patterns/multi-agent coordination); for vector stores, monitor retrieval performance and data quality. Retain audit logs for forensic traceability.
| Goal | Use | Point |
|---|---|---|
| Detailed call analysis | Bedrock Model Invocation Logs | Record I/O |
| GenAI metrics | CloudWatch custom metrics | Tokens/hallucination/quality |
| Detect hallucinations | Golden dataset comparison | Diff vs expected |
| Distributed tracing | X-Ray | Across services |
Trap: “watching latency and error rate is enough for GenAI ops” is wrong—you also need GenAI-specific metrics (hallucination rate, response quality, token usage) and golden-dataset comparison. Also “Model Invocation Logs are only for billing” is wrong—they enable detailed request/response analysis and quality monitoring.
4.3.2Section summary
- Deep analysis = Model Invocation Logs / GenAI monitoring = CloudWatch custom metrics (tokens/hallucination/quality)
- Hallucination detection = golden dataset comparison / tracing = X-Ray
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to record the content of each Bedrock request and response in detail for quality analysis and auditing. Best option?
Q2. You want a mechanism to continuously detect whether hallucinations are increasing in a RAG bot answers. Best approach?
Q3. You want to operationally monitor a GenAI app cost and quality. Which GenAI-specific metrics should you track beyond traditional ones?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.