What's changed: In-scope coverage: added ingestion/migration services (AppFlow/Data Exchange/API Gateway/DataSync/Transfer Family/MGN/Discovery/Snow Family)
1.2ETL and the Data Catalog with AWS Glue
Understand AWS Glue crawlers and the Data Catalog (schema inference, central metadata), serverless Spark ETL jobs, and ETL vs. ELT.
Raw ingested data is transformed into an analysis-friendly form. AWS Glue is the core of serverless ETL on AWS.
1.2.1Crawlers, catalog, and ETL jobs
- Crawler: scans sources to infer schema and registers it in the Data Catalog.
- Data Catalog: central metadata (table definitions) shared by Athena/Redshift Spectrum/EMR.
- Glue ETL jobs: run transforms/cleansing on serverless Spark (pay only when running).
- ETL vs. ELT: ETL transforms before loading; ELT loads first and transforms in the warehouse.
Common on DEA: schema inference + central metadata = Glue Data Catalog (crawlers), serverless ETL = Glue jobs, Athena reads the Glue catalog. For large custom distributed processing, EMR is also an option.
AWS Glue is the core of serverless ETL—know its parts. A crawler scans sources to infer schema and registers table definitions in the Data Catalog (central metadata shared by Athena/Redshift Spectrum/EMR). Glue ETL jobs run transforms/cleansing on serverless Spark (or a Python shell), with the no-code Glue Studio, validation via Glue Data Quality, and interactive sessions for exploration. The keys to cost/performance are output format and layout: convert to columnar (Parquet/ORC), partition by commonly filtered columns, and compress/compact small files—greatly cutting downstream Athena (charged by scanned data) and Redshift cost. Job bookmarks (tracking processed data) prevent reprocessing duplicates. Choose by requirement—"transform then load = ETL," "load first and transform in the warehouse = ELT"—and consider EMR for very large custom distributed processing.
| Role | Glue feature |
|---|---|
| Schema inference + central metadata | Crawler → Data Catalog |
| Serverless transform/cleansing | ETL jobs (Spark) |
| Avoid reprocessing | Job bookmarks |
| Cut downstream scan/cost | Parquet + partition + compress |
Scenario: cheaply analyze raw JSON logs in S3 with Athena. Register schema in the Data Catalog with a Glue crawler → convert to Parquet and partition by date in a Glue ETL job → query via Athena reading the Glue catalog. Scanned data drops sharply, cutting cost. Use job bookmarks to avoid double-processing increments.
Q. Schema inference + central metadata? Crawler + Data Catalog. Q. Serverless transform? Glue ETL jobs (Spark). Q. Lower Athena cost? Parquet + partitioning. Q. Avoid double-processing? Job bookmarks. Q. ETL vs. ELT? Transform-first vs. load-first.
Watch the mix-ups: (1) Crawlers register metadata; ETL jobs transform actual data—don’t swap. (2) Leaving row-based CSV inflates Athena scanned data (= cost)—Parquet + partitioning is the norm. (3) Many tiny files hurt performance/cost (compact them). (4) The Glue catalog is shared by Athena/Redshift Spectrum/EMR—don’t duplicate definitions.
Converting to columnar formats (Parquet/ORC) and partitioning greatly reduces scanned data and cost in downstream Athena/Redshift.
1.2.2Section summary
- Crawler → Data Catalog (metadata) → Glue ETL (Spark)
- Parquet + partitioning cuts downstream cost
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which AWS Glue feature scans sources to infer schema and register central metadata?
Q2. You want to transform/cleanse data serverlessly. Which AWS Glue feature fits best?
Q3. Which data shaping most effectively reduces scanned data and cost in downstream Athena/Redshift?