What's changed: In-scope coverage: added core service explanations
1.1Data Collection and Ingestion
The starting point of ML—understand data sources, streaming ingestion (Kinesis), batch ingestion, and the S3 data lake. Land training data in S3 first.
Machine learning starts with collecting data. On AWS, ingest from diverse sources and aggregate into an S3 data lake for training.
1.1.1Ingestion patterns
- Streaming ingestion: ingest real-time data with Kinesis (Data Streams/Firehose).
- Batch ingestion: bulk DB migration via DMS, periodic ETL ingestion via Glue.
- S3 data lake: aggregate ingested data into S3 as the center for training data.
- Firehose delivery: Amazon Data Firehose can auto-deliver to S3 for near-real-time accumulation.
Common on MLS-C01: real-time ingestion = Kinesis, near-real-time delivery to S3 = Amazon Data Firehose, and aggregate training data in an S3 data lake. The basic pattern: stream with Kinesis, land everything in S3.
Kinesis Data Streams is low-latency with ordering/custom processing; Firehose is fully managed for delivery to S3/Redshift. Choose by use case.
MLS-C01 tests the rationale for choosing ingestion services. Kinesis Data Streams is made of shards, each capped at 1 MB/s and 1,000 records/s write and 2 MB/s read, so you size shard count to throughput and use partition keys to avoid hot shards. On the consumer side, choose between standard (pull, shared per shard) and Enhanced Fan-Out (dedicated 2 MB/s push per consumer). Firehose is fully managed with no shard management, delivers to S3/Redshift/OpenSearch, and can transform with Lambda, convert to Parquet/ORC, and do dynamic partitioning in-flight—building a columnar data lake for ML at ingest time. The contrast is "latency and control" vs "no ops": use Data Streams for real-time inference feeds or when ordering matters, and Firehose to land in S3 near-real-time for later batch training. For loading large existing data at once, use DMS (full load + CDC for ongoing sync) from databases, direct-to-S3 for file sets, Snowball for petabyte-scale on-prem, and DataSync for ongoing file-share sync. If you have Kafka assets, MSK (managed Kafka) is also an ingestion entry point.
| Ingestion | Characteristic | Typical use |
|---|---|---|
| Kinesis Data Streams | Low latency, ordering, shard mgmt | Real-time inference feeds |
| Amazon Data Firehose | No ops, format conversion, delivery | Columnar landing in S3 |
| DMS | Full load + CDC | DB migration/ongoing sync |
| Snowball / DataSync | Bulk physical / ongoing sync | Large on-prem data |
Scenario: ingest clickstream, transform it, land it as Parquet in an S3 data lake, and train in a nightly batch (no real-time inference). → Use Amazon Data Firehose with in-flight Lambda transform + Parquet conversion + dynamic partitioning to land columnar data in S3. No shard management, and the training-optimal layout is built at ingest time.
FAQ: Q. Data Streams or Firehose? A. Data Streams if you need ordering, low latency, or custom processing; Firehose if you want no ops and just land in S3, etc. Q. Convert CSV to Parquet at ingest? A. Firehose record format conversion converts to Parquet/ORC in-flight. Q. Capture ongoing DB changes? A. Use DMS CDC.
Trap: "real-time ingestion always needs Kinesis Data Streams" is false—if you only need to land in S3 near-real-time, the no-shard Firehose is often enough. Conversely, "Firehose gives ordering or sub-second low latency" is wrong—that is Data Streams (with dedicated consumers). Firehose buffers, so it is unsuited to strict low latency.
1.1.2Section summary
- Real-time = Kinesis; bulk = batch (DMS/Glue)
- Aggregate into = S3 data lake (hub of the ML pipeline)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Ingest a real-time data stream from IoT sensors for training. Best choice?
Q2. Deliver streaming data to S3 near-real-time with no management. What?
Q3. Where do you typically aggregate ML training data?