Chapter 5 · Testing, Validation, and Troubleshooting·v1.0.0·Updated 6/22/2026·~12 min
What's changed: Initial: 2 sections for Domain 5 (testing, validation, and troubleshooting)
5.1Evaluation Systems for GenAI
Key points
Learn to systematically evaluate FM output quality: Bedrock Model Evaluations, RAG evaluation, LLM-as-a-Judge, Bedrock Agent evaluations, A/B and canary testing, relevance/factual/consistency/fluency metrics, and regression testing.
GenAI cannot be measured by traditional ML accuracy alone. Evaluate multi-dimensionally—relevance, factual accuracy, consistency, fluency—combining automated and human evaluation to ensure quality.
5.1.1Evaluation methods
- Bedrock Model Evaluations: systematically evaluate models/configurations to identify optimal settings (automated metrics + human).
- LLM-as-a-Judge: use another FM as an evaluator to auto-score output quality at scale.
- RAG/Agent evaluation: assess retrieval+generation with RAG evaluation, and task completion/tool use/reasoning quality with Bedrock Agent evaluations.
- A/B/canary/regression: A/B and canary for production comparison, pre/post-deploy regression tests and quality gates, and user feedback for continuous improvement.
Continue reading — free sign-up
You're reading the free preview. Sign up free to read this section in full, plus every chapter (including 4+) and all questions.