What's changed: In-scope coverage: added ingestion/migration services (AppFlow/Data Exchange/API Gateway/DataSync/Transfer Family/MGN/Discovery/Snow Family)
1.1Data Ingestion (Batch and Streaming)
Understand batch ingestion (scheduled, bulk) vs. streaming ingestion (continuous, near real-time), and choosing among Kinesis Data Streams / Data Firehose / MSK. The starting point for "Data Ingestion and Transformation" in DEA-C01.
The first step of data engineering is how to ingest data. It splits broadly into batch (scheduled, bulk) and streaming (continuous, near real-time).
1.1.1Batch and streaming
- Batch: process large data in bulk at intervals; use Glue, EMR, DMS (DB migration/replication), etc.
- Streaming: process events continuously as they arrive; use Kinesis or MSK (Kafka).
- Kinesis Data Streams: durable and replayable; parallelize via shards, process with custom consumers.
- Data Firehose: no-code delivery to S3/Redshift (with buffering). MSK is managed Apache Kafka.
Common on DEA: no-code stream delivery to S3/Redshift = Data Firehose, need replay/custom processing = Kinesis Data Streams, migrate existing Kafka = MSK, migrate/replicate a whole DB = DMS.
For ingestion, first decide "batch vs. streaming," then for streaming choose by whether you need retention/replay. Batch uses AWS Glue (serverless Spark), EMR (your own Hadoop/Spark cluster), and AWS DMS (DB migration and continuous replication, with CDC (change data capture) to ingest only deltas). Streaming uses Kinesis Data Streams (durable, replayable, parallel via shards, consumed by KCL/KPL or Lambda, scaled by on-demand/provisioned capacity modes), Data Firehose (no-code delivery to S3/Redshift/OpenSearch, buffering by size/time, Lambda transforms, Parquet conversion, no replay retention), and MSK (managed Apache Kafka, good for migrating Kafka assets). The core axes: "deliver as-is to a store = Firehose," "custom processing/multiple consumers/replay = Data Streams," "Kafka-compatibility required = MSK." For small, event-driven volumes, Lambda can ingest directly.
| Requirement | Use |
|---|---|
| No-code delivery to S3/Redshift | Data Firehose |
| Replay/custom/multi-consumer | Kinesis Data Streams |
| Migrate existing Kafka | MSK |
| Migrate/sync a whole DB | DMS (CDC) |
Scenario: ingest IoT events near real-time and re-analyze later. Ingest with Kinesis Data Streams (replayable within retention, parallel via shards). Run a parallel path that just lands data in an S3 data lake via Data Firehose (buffer + Lambda for Parquet). Continuously sync an on-prem RDB with DMS CDC, ingesting only deltas.
Q. No-code delivery? Data Firehose. Q. Replay/custom processing? Kinesis Data Streams. Q. Existing Kafka? MSK. Q. Delta DB sync? DMS (CDC). Q. Can Firehose replay? No—retention is Data Streams’ job.
Watch the mix-ups: (1) Firehose doesn’t retain for replay—use Data Streams if you need reprocessing. (2) Data Streams throughput is set by shard count (or on-demand); shortfalls cause ProvisionedThroughputExceeded/hot shards. (3) DMS is migration + ongoing sync (CDC), not a streaming backbone. (4) Firehose has a minimum buffer delay—not truly instantaneous.
Kinesis Data Streams retains data for a period (enabling replay), while Firehose focuses on delivery and does not retain for replay. Choose by requirement.
1.1.2Section summary
- Batch (Glue/EMR/DMS) / streaming (Kinesis/MSK)
- Delivery-focused = Firehose, replayable = Data Streams
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to deliver streaming data to S3 or Redshift without writing code. Which fits best?
Q2. You want to replay streaming data later and process it in parallel with custom consumers. What do you use?
Q3. You want to migrate an existing on-prem Apache Kafka to a managed AWS service. What do you use?