What's changed: Created Generative AI Leader Chapter 1 (Domain 1 "Fundamentals of generative AI": AI/ML/gen AI relationship = nesting/learning types/data types/modalities-multimodal; LLMs and foundation models = how LLMs work/tokens/prompts/pretraining-fine-tuning/embeddings; limitations and risks = hallucination/bias/knowledge cutoff/privacy).
1.2Large language models and foundation models
Understand the idea and generality of foundation models, how large language models (LLMs) predict the next token using tokens and prompts, the difference between pretraining and fine-tuning, terms like parameters and context window, and the basics of embeddings—from a business perspective.
A foundation model is a general-purpose model pretrained on large, diverse data and adaptable to many uses. One foundation model can be reused across many tasks—summarization, translation, classification, chat—reducing the need to build a model from scratch per use case. A leading example for text is the large language model (LLM).
1.2.1How an LLM works
An LLM splits input into tokens (units of words or substrings) and generates text by repeatedly predicting the "most likely next token" from the context so far. The instruction or question the user provides is the prompt, and output quality depends heavily on it. The maximum tokens handled at once is the context window—larger means more context per call. Model size is roughly expressed by the number of parameters; larger generally means more capable but also more compute cost.
1.2.2Pretraining and fine-tuning
| Stage | What it is | Purpose |
|---|---|---|
| Pretraining | Acquire broad knowledge from vast data | Build a general base |
| Fine-tuning | Further train on domain data | Adapt to a task/industry |
A foundation model first gains broad knowledge via pretraining, then is adapted to specialized tasks with fine-tuning (further training on domain data) as needed. However, as covered later, adapting to your own data does not always require fine-tuning—often grounding or RAG (Chapter 3) to reference external data is simpler. Relatedly, embeddings that represent the meaning of text as numeric vectors underpin semantic search and similarity.
Common: term → meaning. E.g., "pretrained model reusable for many tasks" = foundation model; "foundation model for text" = LLM; "input instruction" = prompt; "processing unit" = token; "context handled at once" = context window; "further training on a domain" = fine-tuning; "represent meaning as vectors" = embeddings.
Watch the mix-ups: (1) pretraining (general knowledge) vs fine-tuning (specific adaptation). (2) Using your own data does not always mean fine-tuning—grounding/RAG is often simpler. (3) An LLM generates by "next-token prediction," which does not guarantee facts (a cause of hallucination).
1.2.3Section summary
- Foundation model = reusable pretrained model; an LLM is the text version
- LLMs predict the next token; prompt/context window/parameters are key terms
- Pretraining (general) → fine-tuning (specific); for your data, RAG/grounding is often simpler
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. What is a general-purpose model pretrained on large, diverse data and adaptable to many uses called?
Q2. Which correctly describes the basic way an LLM generates text?
Q3. What is the instruction or question a user gives the model called?
Q4. What is further training a foundation model on industry/task data to adapt it to a specialized use called?
Q5. Which term denotes the maximum number of tokens a model can handle at once?
Q6. Which represents text meaning as numeric vectors, underpinning semantic search and similarity?