What's changed: Added per-section figures (cert-figure-retrofit). Rewrote AI-103 Chapter 4 to align with official skills measured (Apr 16, 2026). Shifted from AI-102-style (prebuilt Vision + Custom Vision) to a generation-centric design: s1 = image/video generation and editing (text→image: GPT-image-1/DALL·E, inpainting = mask editing, reference media, generation/editing controls, text→video: Sora, async); s2 = multimodal understanding (captions/visual QA/alt-text/object & region ID) + Content Understanding visual characteristic extraction (single-task/pro-mode, video analysis) + multimodal Responsible AI (Content Safety image moderation, indirect prompt injection via in-image text, visual policy: watermarks/prohibited symbols/brand)
4.2Multimodal Understanding and Responsible AI
Understand using multimodal models to comprehend visual context (captions, visual question answering, accessibility alt-text), extracting visual characteristics with Azure Content Understanding, and Responsible AI specific to multimodal content (filters for unsafe visual content, indirect prompt injection via text embedded in images, and visual policy such as watermarks).
A second pillar alongside generation is understanding images and videos. In AI-103, before building a dedicated recognition model, the default is to pass an image to a multimodal model (a generative model that accepts images) and understand it flexibly via a prompt. You can offload recognition and reasoning together—"describe this image," "where is the anomaly in this diagram?"
4.2.1Visual understanding with multimodal models
- Captions: generate image descriptions—concise or detailed, for a single image or multiple images.
- Visual question answering (visual QA): answer questions grounded in the image (based on what the image shows).
- Alt-text / extended descriptions: generate alt-text and detailed descriptions aligned to accessibility guidelines.
- Identify objects, components, and regions: locate targets (parts, regions) within images or video.
4.2.2Extracting visual characteristics with Content Understanding
Beyond multimodal models, Azure Content Understanding in Foundry Tools structurally extracts visual characteristics from images and video (an analyzer returns fields per a schema you define). For video it can run video analysis that processes and interprets segments. Pipelines come in single-task (one extraction task) and pro-mode (multi-step reasoning for more complex cases); choose by requirements. Rule of thumb: "understand an image conversationally" → multimodal model; "stably, structurally extract fixed visual fields" → Content Understanding.
| Goal | Best fit |
|---|---|
| Generate a caption (concise/detailed) | Multimodal model |
| Answer questions grounded in an image | Multimodal model (visual QA) |
| Accessibility alt-text | Multimodal model |
| Structured extraction of fixed visual fields | Content Understanding (single-task / pro-mode) |
| Analyze video segments | Content Understanding (video analysis) |
4.2.3Responsible AI for multimodal content
Working with vision adds risks absent in text. Three key ones: (1) unsafe/disallowed visual content—classify and block generated or input images/video with Azure AI Content Safety image moderation. (2) indirect prompt injection—an attack where instructions embedded as text inside an image are executed by the multimodal model as if they were real instructions; mitigate by not treating in-image text as commands and detecting with prompt shields. (3) visual policy compliance—watermarks / content credentials (provenance) on generated images, detecting prohibited symbols, and honoring brand usage rules. This is consistent with Chapter 1’s governance.
| Risk | What it is | Main mitigation |
|---|---|---|
| Unsafe visual content | Inappropriate images/video (violence, sexual, etc.) | Azure AI Content Safety image moderation |
| Indirect prompt injection | Executing instructions embedded as text in an image | Don’t treat in-image text as commands / prompt shields |
| Visual policy violation | Unknown provenance, prohibited symbols, brand misuse | Watermarks/content credentials, symbol detection, brand rules |
Common: (1) "understand an image conversationally (captions, visual QA, alt-text)" = multimodal model. (2) "structured extraction of fixed visual fields / analyze video segments" = Content Understanding (single-task / pro-mode). (3) "detect/block inappropriate images" = Content Safety image moderation. (4) "instructions hijacked via text embedded in an image" = indirect prompt injection. (5) provenance of generated images = watermarks / content credentials.
Watch out: (1) don’t confuse multimodal model (flexible conversational understanding) with Content Understanding (structured extraction of fixed fields). (2) indirect prompt injection (in-image text) differs in vector from ordinary (text-input) prompt injection. (3) unsafe content (safety = Content Safety) and visual policy (provenance/brand = watermarks, etc.) are different mitigations.
4.2.4Section summary
- Conversational understanding of images (captions, visual QA, alt-text, object/region ID) = multimodal model
- Structured extraction of fixed visual fields / video analysis = Content Understanding (single-task / pro-mode)
- Multimodal Responsible AI: unsafe visual content (Content Safety) / indirect prompt injection (in-image text) / visual policy (watermarks, prohibited symbols, brand)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To pass an image directly and flexibly generate captions or answer questions grounded in it, which fits?
Q2. What is the attack where a multimodal model executes instructions embedded as text inside an image?
Q3. Which Azure capability detects/blocks inappropriate (violent/sexual, etc.) generated or input images?
Q4. To "stably, structurally extract" fixed-schema visual fields from images/video, which fits best?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.