Instiq

3Data pipeline orchestration

Practice questions →Glossary →
  • 3.1Designing data pipelines

    Understand choosing a data transformation tool (Dataproc, Dataflow, Cloud Data Fusion, Cloud Composer, Dataform) by business requirements, evaluating ELT vs ETL use cases, and combining the products needed to implement basic transformation pipelines.

  • 3.2Scheduling, automation, and monitoring

    Understand automating data processing with scheduled queries and Cloud Scheduler/Cloud Composer, monitoring pipelines via the Dataflow job UI and Cloud Logging/Cloud Monitoring, choosing an orchestration solution (Cloud Composer, Workflows, Dataproc Workflow Templates), and event-driven ingestion from Pub/Sub to BigQuery with Eventarc triggers.