What's changed: In-scope coverage: added an AI/ML support in-scope services block to Ch3 §3 (Amazon SageMaker AI/Amazon Personalize/Amazon Augmented AI = A2I)
3.3Selecting and Evaluating Foundation Models
Understand criteria for choosing a foundation model (accuracy, cost, latency), how to evaluate, and risks like hallucination.
There is no single foundation model. Choose by balancing accuracy for the task, cost, and latency.
3.3.1Selection criteria
- Accuracy/quality: evaluate on your task/data to confirm fit; the largest model is not always best.
- Cost: usually priced per token; larger models can be more accurate but pricier.
- Latency: response speed—especially important for real-time uses like chat.
- Others: supported languages, multimodality, context-window size, and license/customizability also matter.
Judging a model is fundamentally about "actually testing and measuring on your task." Evaluation includes automatic metrics against references (e.g., summarization/translation quality scores) and human evaluation of outputs. High benchmark numbers don’t guarantee fit for your use, so compare on representative inputs. AIF-C01 tests the mindset of choosing by the accuracy/cost/latency balance for the task, not just "pick the biggest/newest model."
| Criterion | What to check | Note |
|---|---|---|
| Accuracy/quality | Evaluate on your data | Biggest ≠ best |
| Cost | Per-token price × usage | Larger = pricier |
| Latency | Response speed | Key for real-time |
Scenario: choosing a chatbot model. Since real-time response is needed, prioritize latency and first evaluate small/medium models on your representative inquiries. Route only the hard inquiries that lack accuracy to a larger model to optimize cost. Ground outputs with RAG against hallucination, and have humans review important answers. The point: don’t default to the largest model.
Generative output can contain hallucinations (plausible but wrong). For important uses, add human review (human in the loop) and grounding via RAG. Fluency does not equal correctness.
Q. Is the largest model always safe? No—it tends to be costly and high-latency, and may be overkill. Evaluate on your task. Q. How to evaluate? Automatic metrics plus human evaluation on representative inputs. Q. Does model choice eliminate hallucination? No—mitigate operationally with RAG and human review.
Common: choose by trading off accuracy/cost/latency, evaluate on your own data (automatic metrics + human evaluation), pricing is roughly per token, the largest model is not always best, mitigate hallucination with human review/RAG.
3.3.2In-scope AI/ML services that support applications
AWS AI/ML services that support applying and evaluating foundation models are also in scope for AIF-C01. Amazon SageMaker AI (formerly Amazon SageMaker) is the integrated platform to build, train, and deploy ML models on your data, and also to use and tune foundation models. Amazon Personalize is a managed service that delivers per-user recommendations without requiring your own ML expertise. Most relevant to this section’s evaluation/robustness theme is Amazon Augmented AI (A2I), which makes it easy to add human review of low-confidence or high-stakes ML/generative predictions (human-in-the-loop)—for example, routing only the low-confidence fields from document extraction to a human for confirmation.
3.3.3Section summary
- Selection = balance accuracy/cost/latency; evaluate on your own data (automatic + human)
- The largest/newest model isn’t always best (watch excess cost/latency)
- Watch for hallucination; reinforce with human review/RAG
- Supporting services = SageMaker AI (build/train/deploy) / Personalize (recommendations) / Augmented AI = A2I (human review)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which set of criteria is appropriate when choosing a foundation model?
Q2. What is a common basis for foundation model pricing?
Q3. Which is a risk to watch for in generative AI output?