What's changed: Added per-section figures (cert-figure-retrofit). Added a speech section (s3) to AI-103 Chapter 5 (the official text-analysis domain’s "speech" sub-area = STT/TTS, speech as agent modality, custom speech / custom neural voice, multimodal reasoning from audio, speech translation). Existing s1 (Azure AI Language) and s2 (translation/custom language models) preserved non-destructively
5.3Implementing Speech Solutions
Covers "speech," part of the official text-analysis domain. Understand—from a developer’s view—speech-to-text/text-to-speech (STT/TTS) with Azure AI Speech, speech as an agent modality with custom speech, multimodal reasoning from audio, and speech translation.
The official "text analysis" domain also includes speech. Azure AI Speech (in Foundry Tools) meets needs like operating a conversational agent by voice or transcribing meetings from audio. Developers call STT/TTS APIs and, when needed, integrate speech as an agent modality.
5.3.1Speech ↔ text (STT / TTS)
There are two directions. Speech-to-text (STT) converts spoken audio into text (transcription, voice input). Text-to-speech (TTS) converts text into audio (read-aloud, voice responses). For a voice-driven agent, transcribe the user with STT → process with the model/agent → speak the response with TTS, making speech the input/output modality.
5.3.2Speech as an agent modality and custom speech
When generic models are not accurate enough, use custom speech (a custom speech model) to adapt recognition to domain terms, product names, accents, or noisy environments. To read aloud in a brand-specific voice, use custom neural voice (the synthesis-side custom). Note that custom on the recognition side (adapt STT) and custom on the synthesis side (create a voice) are different.
5.3.3Multimodal reasoning from audio and speech translation
You can also pass audio directly to a model for multimodal reasoning from audio without a separate transcription step (audio input to an audio-capable multimodal model). Speech translation converts speech in one language into another, implemented with Azure AI Speech or language models / Foundry Tools. Distinguish "speech → another language" (speech translation) from "text → another language" (the Translator in section 2).
| Goal | Approach |
|---|---|
| Transcribe spoken audio | Speech-to-text (STT) |
| Read text aloud | Text-to-speech (TTS) |
| Improve recognition of jargon/accents | Custom speech (recognition side) |
| Read aloud in a brand voice | Custom neural voice (synthesis side) |
| Understand/reason over audio as-is | Audio-capable multimodal model |
| Speech into another language | Speech translation |
Common: (1) "speech→text" = STT, "text→speech" = TTS. (2) Voice-driven agent = STT→process→TTS speech modality. (3) Improve recognition of jargon/accents = custom speech (recognition side); brand voice read-aloud = custom neural voice (synthesis side). (4) "speech→another language" = speech translation (vs text translation = Translator). (5) Reason over audio directly = audio-capable multimodal model.
Watch out: (1) STT (recognition) and TTS (synthesis) are opposite directions. (2) custom speech (adapt recognition) vs custom neural voice (create a voice) differ. (3) Don’t confuse speech translation (speech→language) with text translation (Translator). (4) Transcribe-then-process differs from passing audio directly to a multimodal model.
5.3.4Section summary
- STT = speech→text / TTS = text→speech; agent speech modality = STT→process→TTS
- Custom speech (adapt recognition) vs custom neural voice (synthesis voice) are different
- Speech translation = speech→another language (vs Translator); multimodal reasoning over audio is also an option
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which feature converts spoken audio into text?
Q2. How do you improve speech recognition where jargon and accents are common?
Q3. Which applies when converting Japanese speech directly into English?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.