What's changed: Created Professional Data Engineer Chapter 1 (Domain 1 "Design": security/compliance = IAM least privilege/org policies/GMEK-CMEK-Cloud EKM/Sensitive Data Protection/data residency/project-dataset-table design/multi-environment; reliability and migration = Dataform/Dataflow/Cloud Data Fusion, Cloud Composer, DR/fault tolerance/ACID/data validation, BigQuery DTS/Database Migration Service/Datastream/Transfer Appliance).
1.2Designing for reliability, migration, and data methods
Understand data preparation/cleaning (Dataform, Dataflow, Cloud Data Fusion), pipeline monitoring/orchestration, disaster recovery and fault tolerance, ACID vs availability decisions, data validation, and choosing migration tools (BigQuery Data Transfer Service, Database Migration Service, Datastream, Transfer Appliance).
Robust data platforms are designed for reliability (recover from failure, correct data) and migration from existing assets. Choose methods and tools by requirements.
1.2.1Preparation, reliability, and validation
For data prep/cleaning, choose Dataform (manage SQL-based ELT), Dataflow (scalable code-based processing), or Cloud Data Fusion (visual low-code). Operate pipelines with monitoring and orchestration (Cloud Composer). For reliability, design disaster recovery (DR) and fault tolerance (multi-region/zone, retries), and judge ACID vs availability tradeoffs. Ensure quality (missing/inconsistent) with data validation. Map "SQL ELT = Dataform" and "visual ETL = Cloud Data Fusion."
1.2.2Designing data migration
| Tool | Use |
|---|---|
| BigQuery Data Transfer Service | Scheduled loads from SaaS/other DWH into BigQuery |
| Database Migration Service | Migrate databases with minimal downtime |
| Datastream | Change data capture (CDC) streaming replication |
| Transfer Appliance | Physical shipping for huge data on thin networks |
Plan migration by analyzing current stakeholders/processes/technology. Choose tools by requirements: scheduled loads from SaaS/other DWH into BigQuery = BigQuery Data Transfer Service, minimal-downtime DB migration = Database Migration Service, continuous change replication via CDC = Datastream, huge data on thin networks = Transfer Appliance (physical). Map "minimal-downtime DB migration = Database Migration Service" and "continuous change replication = Datastream (CDC)."
Common: requirement → means. E.g., "manage ELT with SQL" = Dataform; "visual low-code ETL" = Cloud Data Fusion; "scalable code processing" = Dataflow; "orchestrate dependent pipelines" = Cloud Composer; "minimal-downtime DB migration" = Database Migration Service; "replicate continuous changes" = Datastream (CDC); "scheduled SaaS-to-BigQuery loads" = BigQuery Data Transfer Service.
Watch the mix-ups: (1) Do not confuse Database Migration Service (DB migration), Datastream (CDC replication), and BigQuery Data Transfer Service (scheduled loads). (2) Choose Dataform (SQL ELT), Dataflow (Beam code), Cloud Data Fusion (visual) by use. (3) Design DR/fault tolerance across multiple regions/zones.
1.2.3Section summary
- Prep/cleaning = Dataform (SQL ELT)/Dataflow (code)/Cloud Data Fusion (visual); operate via Cloud Composer
- Reliability = DR, fault tolerance (multi-region/zone, retries), ACID, data validation
- Migration = BigQuery DTS (scheduled)/Database Migration Service (min downtime)/Datastream (CDC)/Transfer Appliance (physical)
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To implement SQL-based transforms (ELT) in BigQuery with dependencies/version control, which is best?
Q2. To build ETL/ELT pipelines visually with little code, which is best?
Q3. To migrate a database to Google Cloud with minimal downtime, which tool is best?
Q4. To stream-replicate continuous source-DB changes via change data capture (CDC), which is best?
Q5. To migrate hundreds of TB with limited network bandwidth, which is best?
Q6. To schedule/orchestrate multi-step data pipelines with dependencies, which is best?