What's changed: In-scope coverage: added purpose-built stores/storage (DocumentDB/Keyspaces/MemoryDB/S3 Tables/EFS/CloudFront/Backup)
2.1Designing an S3 Data Lake
Understand an S3-centric data lake (raw/cleaned/curated layers), partitioning and columnar formats, and cost optimization via storage classes and lifecycle. The starting point for "Data Store Management" in DEA-C01.
An S3 data lake stores diverse data cheaply at scale. Good layering, partitioning, and formats make analysis faster and cheaper.
2.1.1Layers and formats
- Layers: improve quality in stages—raw → cleaned → curated.
- Partitioning: split by date/region folders to reduce scanned data per query.
- Columnar formats: Parquet/ORC improve compression and analytics performance.
- Storage classes/lifecycle: move data to cheaper classes by access frequency to cut cost.
Common on DEA: reduce scanned data and cost = partition + Parquet, raw/cleaned/curated layering, move aging data to cheaper classes = lifecycle. S3 is the heart of the data lake.
Design an S3 data lake by "layers × layout × lifecycle." Layers raise quality through raw (bronze) → cleaned (silver) → curated (gold), separated by prefix. The core of layout is columnar (Parquet/ORC) conversion and partitioning (by commonly filtered columns like date/region), slashing downstream Athena scanned data (= cost). Many tiny files hurt performance, so compact them, and use partition projection in Athena to skip metadata management. If you need updates/deletes/time-travel, use table formats like Apache Iceberg/Hudi/Delta; centralize permissions with Lake Formation (column/row level). Lifecycle auto-transitions S3 Standard → Standard-IA → Glacier by access frequency (use Intelligent-Tiering when unsure). Encryption at rest (SSE-KMS), versioning, and Object Lock against accidental deletion are basics. S3’s eleven-nines durability makes it the heart of the data lake.
| Design axis | Lever |
|---|---|
| Quality stages | Raw → cleaned → curated |
| Reduce scan/cost | Parquet + partition + compaction |
| Update/delete/time-travel | Iceberg/Hudi/Delta |
| Storage cost | Lifecycle / Intelligent-Tiering |
Scenario: raw JSON keeps growing; analysis is slow and costly. Keep it in raw (bronze), then convert to Parquet with Glue ETL and partition by date into the cleaned (silver) layer. Athena queries silver/gold, slashing scanned data. Move old raw data to Glacier via lifecycle and protect against deletion with versioning + Object Lock.
Q. Reduce scanned data? Parquet + partitioning. Q. Layering? Raw → cleaned → curated. Q. Cost of old data? Lifecycle (Glacier, etc.). Q. Need updates/deletes? Iceberg/Hudi/Delta. Q. Centralize permissions? Lake Formation.
Watch the mix-ups: (1) Leaving row-based CSV inflates Athena scanned data (cost)—Parquet + partitioning is the norm. (2) Many tiny files hurt performance/cost (compact). (3) Plain S3 (Parquet) is poor at row-level updates—use a table format like Iceberg if you need them. (4) Glacier retrieval has latency/cost—don’t put frequently accessed data there.
Lake Formation centrally manages data-lake permissions at column/row level, easing governance (covered in Chapter 4).
2.1.2Section summary
- Layers (raw/cleaned/curated) + partitioning + Parquet
- Optimize storage cost with lifecycle
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. What most effectively reduces scanned data and cost for queries in an S3 data lake?
Q2. Which is a common layering for quality stages in a data lake?
Q3. You want to auto-reduce storage cost for aging, less-accessed data. What do you use?