What's changed: Created Associate Data Practitioner Chapter 3 (Domain 3 "Pipeline orchestration": designing pipelines = Dataproc/Dataflow/Cloud Data Fusion/Cloud Composer/Dataform/ELT-ETL; scheduling, automation, monitoring = scheduled queries/Cloud Scheduler/Workflows/Dataproc Workflow Templates/Pub/Sub→BigQuery/Eventarc/Dataflow job UI/Cloud Logging-Monitoring).
3.1Designing data pipelines
Understand choosing a data transformation tool (Dataproc, Dataflow, Cloud Data Fusion, Cloud Composer, Dataform) by business requirements, evaluating ELT vs ETL use cases, and combining the products needed to implement basic transformation pipelines.
The mechanism that regularly and automatically moves and transforms data is a data pipeline. Choose the right tool by requirements (existing assets, skills, scale, real-time needs).
3.1.1Choosing a transformation tool
| Tool | Best for |
|---|---|
| Dataflow | Scalable batch/streaming (Apache Beam) |
| Dataproc | Existing Spark/Hadoop jobs |
| Cloud Data Fusion | Visual ETL/ELT (low-code) |
| Dataform | SQL-based transforms in BigQuery (ELT) |
| Cloud Composer | Multi-step orchestration (Airflow) |
Dataflow processes batch/streaming at scale on Apache Beam. With existing Spark/Hadoop assets, use Dataproc; for visual low-code builds, Cloud Data Fusion; to manage SQL-based transforms (ELT) inside BigQuery, Dataform. To orchestrate multi-step dependencies, use Cloud Composer (Airflow-based). Map "Spark/Hadoop = Dataproc," "manage SQL ELT = Dataform," "orchestrate workflows = Cloud Composer."
Common: requirement → tool. E.g., "scalable batch/streaming" = Dataflow; "move existing Spark/Hadoop" = Dataproc; "visual low-code ETL" = Cloud Data Fusion; "manage SQL transforms (ELT) in BigQuery" = Dataform; "orchestrate multi-step dependencies" = Cloud Composer.
Watch the mix-ups: (1) Dataflow (Beam, scalable processing) vs Dataproc (Spark/Hadoop). (2) Cloud Composer (Airflow, orchestration) is workflow control, not the transform itself. (3) Dataform manages SQL ELT in BigQuery.
3.1.2Section summary
- Dataflow (Beam, batch/streaming)/Dataproc (Spark/Hadoop)/Cloud Data Fusion (visual)/Dataform (SQL ELT)
- Cloud Composer (Airflow) orchestrates multi-step workflows
- Choose tools by requirements (assets, skills, scale)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To run existing Spark/Hadoop jobs as-is on Google Cloud, which is best?
Q2. To do scalable batch and streaming transformation (Apache Beam-based), which is best?
Q3. To implement SQL-based transforms (ELT) inside BigQuery with version control, which is best?
Q4. To orchestrate a data workflow with multi-step dependencies, which is best?
Q5. To build ETL/ELT pipelines visually with little code, which is best?
Q6. Which best describes the main role of Cloud Composer?