What's changed: Initial: 2 sections for Domain 5 (testing, validation, and troubleshooting)
5.1Evaluation Systems for GenAI
Learn to systematically evaluate FM output quality: Bedrock Model Evaluations, RAG evaluation, LLM-as-a-Judge, Bedrock Agent evaluations, A/B and canary testing, relevance/factual/consistency/fluency metrics, and regression testing.
GenAI cannot be measured by traditional ML accuracy alone. Evaluate multi-dimensionally—relevance, factual accuracy, consistency, fluency—combining automated and human evaluation to ensure quality.
5.1.1Evaluation methods
- Bedrock Model Evaluations: systematically evaluate models/configurations to identify optimal settings (automated metrics + human).
- LLM-as-a-Judge: use another FM as an evaluator to auto-score output quality at scale.
- RAG/Agent evaluation: assess retrieval+generation with RAG evaluation, and task completion/tool use/reasoning quality with Bedrock Agent evaluations.
- A/B/canary/regression: A/B and canary for production comparison, pre/post-deploy regression tests and quality gates, and user feedback for continuous improvement.
Common: systematic model/config evaluation = Bedrock Model Evaluations, auto-score at scale = LLM-as-a-Judge, retrieval+generation quality = RAG evaluation, agent task completion = Agent evaluations, safe production comparison = A/B/canary. Axes: relevance/factual/consistency/fluency.
Evaluation combines metrics, automation, and production validation. Define output quality with multi-dimensional metrics (relevance, factual accuracy, consistency, fluency), and systematize model/config comparison with Bedrock Model Evaluations (automated metrics + human). Scale with LLM-as-a-Judge (an evaluator FM scores), complemented by human annotation (e.g., SageMaker Ground Truth). Assess RAG with RAG evaluation (retrieval relevance + generation faithfulness) and agents with Bedrock Agent evaluations (task completion, tool-use effectiveness, multi-step reasoning quality). Promote to production safely via A/B / canary, stop regressions with pre/post-deploy regression tests and automated quality gates, and validate AI-specific output (hallucination rate, semantic drift) with synthetic user flows (e.g., CloudWatch Synthetics). Communicate results to stakeholders via visualization reports/model comparisons, and close the loop with user feedback for continuous improvement.
| Target | Use | Measures |
|---|---|---|
| Model/config comparison | Bedrock Model Evaluations | Automated metrics + human |
| Quality scoring at scale | LLM-as-a-Judge | Relevance/factual/consistency |
| RAG quality | RAG evaluation | Retrieval relevance + faithfulness |
| Agent quality | Agent evaluations | Task completion/tool use |
Trap: “accuracy alone measures GenAI quality” is wrong—evaluate generation across relevance, factual accuracy, consistency, and fluency. Also “LLM-as-a-Judge fully replaces human evaluation” is wrong—it scales automation but complement critical judgments with human evaluation/Ground Truth.
5.1.2Section summary
- Systematic = Model Evaluations / at scale = LLM-as-a-Judge / RAG = RAG evaluation / agents = Agent evaluations
- Production = A/B·canary + regression + quality gates / axes = relevance/factual/consistency/fluency
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to systematically compare which of several FMs/configurations best fits your use case, using both automated metrics and human evaluation. Best option?
Q2. You want to auto-score quality (relevance, factual accuracy, etc.) of tens of thousands of generated answers at scale and consistently, without relying on humans alone. Best method?
Q3. When promoting a new prompt version to production, you want to compare and validate quality on real traffic while limiting impact on existing users. Best option?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.