What's changed: Created Associate Data Practitioner Chapter 1 (Domain 1 "Preparation and ingestion": methodologies and quality = ETL/ELT/ETLT/data quality/cleaning (Cloud Data Fusion/BigQuery/Dataflow)/transfer (Storage Transfer Service/Transfer Appliance); formats and extract/load = CSV/JSON/Parquet/Avro/BigQuery Data Transfer Service/Database Migration Service/CLI; choosing storage and location = destinations/structured classification/regional-dual-multi-zonal).
1.1Data processing methodologies and quality
Understand the difference and selection among ETL, ELT, and ETLT data-manipulation methodologies, assessing data quality, data cleaning with Cloud Data Fusion, BigQuery, and Dataflow, and choosing the right data transfer tool (Storage Transfer Service, Transfer Appliance).
Associate Data Practitioner is for practitioners who ingest, transform, analyze, and manage data with Google Cloud data services. The foundation is "preparation and ingestion"—getting data into a usable state. First, the methodologies for how to transform data.
1.1.1ETL, ELT, ETLT
| Method | Order | Characteristic |
|---|---|---|
| ETL | Extract → Transform → Load | Transform before load (pre-shaped) |
| ELT | Extract → Load → Transform | Load first, transform in the DWH (e.g., BigQuery) |
| ETLT | Some transform → Load → more transform | A hybrid of the two |
ETL is "Extract → Transform → Load," transforming before load. ELT is "Extract → Load → Transform," loading raw data first and transforming inside a powerful data warehouse like BigQuery (a modern approach for large data). ETLT is a hybrid. As cloud warehouses have grown powerful, ELT is increasingly common. Map "transform inside the DWH = ELT" and "shape before load = ETL."
1.1.2Data quality and cleaning
Assess data quality before/after ingestion (missing values, duplicates, inconsistencies, format variance). For data cleaning, choose by scale and skills among Cloud Data Fusion (visual pipelines), BigQuery (cleaning with SQL), and Dataflow (scalable code-based processing). For data transfer into Google Cloud, choose Storage Transfer Service (over the network) or Transfer Appliance (physically ship data when the network is thin or volume is huge).
Common: situation → method/tool. E.g., "load into BigQuery then transform with SQL" = ELT; "shape before load" = ETL; "visual ETL pipelines" = Cloud Data Fusion; "scalable code-based transform" = Dataflow; "large transfer over network" = Storage Transfer Service; "physically ship when network is thin/huge volume" = Transfer Appliance.
Watch the mix-ups: (1) the order difference of ETL (transform before load) vs ELT (transform in the DWH). (2) Transfer: choose Storage Transfer Service (network) vs Transfer Appliance (physical shipping) by scale/link. (3) Choose cleaning tool by scale: Cloud Data Fusion/BigQuery/Dataflow.
1.1.3Section summary
- ETL (transform before load) / ELT (transform in the DWH, BigQuery) / ETLT (hybrid); ELT is increasingly standard
- Cleaning = Cloud Data Fusion (visual) / BigQuery (SQL) / Dataflow (code)
- Transfer = Storage Transfer Service (network) / Transfer Appliance (physical shipping)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which methodology loads raw data into BigQuery first, then transforms it with SQL inside the warehouse?
Q2. Which methodology transforms (shapes) data before loading it?
Q3. To do scalable code-based batch/streaming data transformation, which tool is best?
Q4. With a thin network and petabyte-scale data to physically move to Google Cloud, which is best?
Q5. To build ETL/ELT pipelines visually, which tool is best?
Q6. Which modern data-integration methodology has grown as cloud data warehouses became powerful?