What's changed: Created Associate Data Practitioner Chapter 3 (Domain 3 "Pipeline orchestration": designing pipelines = Dataproc/Dataflow/Cloud Data Fusion/Cloud Composer/Dataform/ELT-ETL; scheduling, automation, monitoring = scheduled queries/Cloud Scheduler/Workflows/Dataproc Workflow Templates/Pub/Sub→BigQuery/Eventarc/Dataflow job UI/Cloud Logging-Monitoring).
3.2Scheduling, automation, and monitoring
Understand automating data processing with scheduled queries and Cloud Scheduler/Cloud Composer, monitoring pipelines via the Dataflow job UI and Cloud Logging/Cloud Monitoring, choosing an orchestration solution (Cloud Composer, Workflows, Dataproc Workflow Templates), and event-driven ingestion from Pub/Sub to BigQuery with Eventarc triggers.
A pipeline becomes operational only when scheduled, automated, and monitored for progress and failures. Choose scheduling and orchestration by requirements.
3.2.1Scheduling and automation
For recurring identical aggregations, BigQuery scheduled queries are simplest. For general time-based triggers use Cloud Scheduler; for multi-step dependencies use Cloud Composer; for lightweight step chaining use Workflows; for Dataproc job chaining use Dataproc Workflow Templates. For event-driven flows, ingest from Pub/Sub to BigQuery or use Eventarc triggers to start pipelines (Dataflow, Cloud Functions, Cloud Run, Cloud Composer). Map "recurring aggregation = scheduled queries" and "multi-step with dependencies = Cloud Composer."
3.2.2Monitoring
Monitor pipeline state. Check Dataflow progress in the Dataflow job UI, and analyze logs and metrics with Cloud Logging and Cloud Monitoring. Use them to find failure causes and detect delays, setting alerts as needed. Map "job progress = Dataflow job UI" and "monitor/diagnose via logs/metrics = Cloud Logging/Monitoring."
Common: requirement → means. E.g., "recurring same aggregation" = scheduled queries; "time-based trigger" = Cloud Scheduler; "multi-step with dependencies" = Cloud Composer; "lightweight step chaining" = Workflows; "event-driven ingestion" = Pub/Sub→BigQuery / Eventarc; "check Dataflow progress" = Dataflow job UI; "monitor via logs/metrics" = Cloud Logging/Monitoring.
Watch the mix-ups: (1) choose scheduled queries (recurring aggregation) vs Cloud Composer (dependency orchestration) by scale/complexity. (2) Use Dataflow job UI (progress) with Cloud Logging/Monitoring (logs/metrics). (3) Event-driven = Pub/Sub→BigQuery / Eventarc.
3.2.3Section summary
- Recurring aggregation = scheduled queries; time trigger = Cloud Scheduler; multi-step dependencies = Cloud Composer; lightweight = Workflows
- Event-driven = Pub/Sub→BigQuery / Eventarc triggers
- Monitoring = Dataflow job UI (progress) + Cloud Logging/Monitoring (logs/metrics)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To auto-run the same aggregation query in BigQuery every morning, which is simplest?
Q2. To check the progress of a Dataflow pipeline, what do you use?
Q3. For event-driven ingestion that loads to BigQuery when events arrive in Pub/Sub, which fits?
Q4. To schedule and orchestrate data processing with multi-step dependencies, which is best?
Q5. To analyze pipeline logs and metrics to find failures or delays, what do you use?
Q6. To trigger pipelines (Dataflow, Cloud Functions, etc.) from diverse event sources, what do you use?