What's changed: Created Professional Data Engineer Chapter 2 (Domain 2 "Ingest and process": planning/building = sources/sinks, transforms, batch/streaming/windowing/late data, Dataflow/Beam/Dataproc/Spark/Hadoop/Cloud Data Fusion/BigQuery/Pub/Sub/Kafka; deploy/operationalize = Cloud Composer (DAG)/Workflows, cleansing/AI enrichment, CI/CD).
2.1Planning and building pipelines
Understand defining data sources and sinks, transformation/orchestration logic, batch and streaming (windowing, late data) processing, and choosing the right processing services (Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Spark, Kafka).
The core of a data platform is the pipeline that ingests and processes data. Design sources/sinks, batch vs streaming, and the services to use—by requirements.
2.1.1Sources/sinks and batch/streaming
A pipeline defines data sources (where data comes from) and sinks (where it is written), and the transformation logic between. Two modes: batch (process accumulated data together) and streaming (process arriving data continuously). Streaming requires handling windows (time-based grouping) and late-arriving data. Use Pub/Sub (messaging) for real-time ingestion. Map "continuous/real-time = streaming + Pub/Sub" and "time-based aggregation = windows."
2.1.2Choosing processing services
Choose by characteristics: Dataflow (Apache Beam) for scalable, unified batch/streaming; Dataproc to run existing Spark/Hadoop assets; Cloud Data Fusion for visual low-code; BigQuery for SQL-based large transforms. Messaging = Pub/Sub, and existing Kafka assets can be used. Map "unified Beam batch/stream = Dataflow," "existing Spark/Hadoop = Dataproc," "SQL transform = BigQuery."
Common: requirement → service. E.g., "scalable unified batch + stream" = Dataflow (Beam); "run existing Spark/Hadoop" = Dataproc; "visual low-code" = Cloud Data Fusion; "SQL large transforms" = BigQuery; "real-time ingestion" = Pub/Sub; "time-based aggregation" = windows; "late-arriving data" = watermarks/allowed lateness.
Watch the mix-ups: (1) Do not confuse Dataflow (Beam, unified) and Dataproc (existing Spark/Hadoop). (2) Streaming requires windowing and late-data design. (3) Pub/Sub is typically at-least-once—handle duplicates idempotently.
2.1.3Section summary
- Define sources/sinks/transforms; choose batch vs streaming (windows/late data) by requirements
- Processing = Dataflow (Beam)/Dataproc (Spark/Hadoop)/Cloud Data Fusion (visual)/BigQuery (SQL)
- Real-time ingestion via Pub/Sub (handle at-least-once idempotently)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. For scalable, unified batch and streaming processing, which is best?
Q2. To run existing Spark/Hadoop workloads managed, which is best?
Q3. To ingest events in real time and distribute to multiple services, which is best?
Q4. In streaming, to aggregate per fixed time intervals, which concept?
Q5. To correctly handle late-arriving data in streaming, what should you design?
Q6. Which correctly contrasts Dataflow and Dataproc?