What's changed: Created Professional Data Engineer Chapter 3 (Domain 3 "Storing": storage choice and DWH = BigQuery/BigLake/AlloyDB/Bigtable/Spanner/Cloud SQL/Cloud Storage/Firestore/Memorystore, normalization/denormalization, partitioning/clustering; data lakes and platforms = data lake (Cloud Storage) discovery/access/cost, Dataplex/Dataplex Catalog, federated governance).
3.2Data lakes and data platforms
Understand managing data lakes (discovery, access, cost controls), building data platforms with Dataplex and Dataplex Catalog, federated governance for distributed data systems, and choosing between lakes and warehouses.
Organizational data spans not only the structured data warehouse but also the data lake that holds diverse raw data. A data platform governs across them.
3.2.1Managing the data lake
A data lake (often on Cloud Storage) cheaply stores diverse raw data. Operationally, set up data discovery (what exists), access control (who can use it), and cost controls (lifecycle). Lake data can be processed/analyzed, and lake+warehouse combinations (lakehouse-like) are possible. Map "cheaply store raw data = data lake (Cloud Storage)" and "manage discovery/access/cost."
3.2.2Dataplex and federated governance
Dataplex is a data platform that centrally manages quality, catalog, and governance across distributed data (lakes/warehouses). Use Dataplex Catalog to discover/organize metadata and understand where data lives and what it means. As organizations grow, federated governance (domains own responsibility while sharing common rules) works better than central control. Map "cross-data governance = Dataplex" and "metadata discovery = Dataplex Catalog."
Common: requirement → means. E.g., "cheaply store diverse raw data" = data lake (Cloud Storage); "govern quality/governance across distributed data" = Dataplex; "discover/organize metadata" = Dataplex Catalog; "distributed governance at scale" = federated governance.
Watch the mix-ups: (1) Data lake (diverse raw) and warehouse (structured analytics) differ (combinable). (2) Do not confuse Dataplex (cross-data governance) and Dataplex Catalog (metadata). (3) Choose central vs federated governance by scale.
3.2.3Section summary
- Data lakes (Cloud Storage) cheaply store raw data; manage discovery/access/cost
- Dataplex governs across distributed data; Dataplex Catalog discovers metadata
- At scale, use federated governance for distributed control
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which is commonly the foundation of a data lake storing diverse raw data cheaply?
Q2. To centrally manage quality/catalog/governance across distributed data (lakes/warehouses), which is best?
Q3. To discover/organize metadata and understand where data lives and what it means, what do you use?
Q4. At scale, to let domains own responsibility while sharing common rules, which approach best fits?
Q5. Which correctly contrasts data lakes and data warehouses?
Q6. Which best lists what to set up when operating a data lake?