What's changed: Created Professional Data Engineer Chapter 4 (Domain 4 "Prepare and use for analysis": visualization prep = Looker/Looker Studio, materialized views/BI Engine, slow-query handling, data masking/IAM/Cloud DLP/column-row access; AI-ML prep and sharing = BigQuery ML/feature engineering, embeddings/RAG, Analytics Hub).
4.2Preparing data for AI/ML and sharing
Understand data prep for feature engineering and training/serving (BigQuery ML), preparing unstructured data for embeddings and retrieval-augmented generation (RAG), and sharing data (rules, publishing datasets, Analytics Hub).
Beyond analytics lie AI/ML and data sharing. Data engineers prepare ML-ready data and share data safely within and across organizations.
4.2.1Preparing data for AI/ML
For structured-data ML, BigQuery ML is convenient—build models with only SQL inside BigQuery (feature engineering, training, prediction in SQL). For generative AI, prepare a foundation by turning unstructured data (documents) into embeddings and using retrieval-augmented generation (RAG) to fetch relevant context for answers. Map "ML with only SQL in BigQuery = BigQuery ML" and "use unstructured data with gen AI = embeddings/RAG."
4.2.2Sharing data
Share by defining sharing rules, publishing datasets, and sharing reports/visualizations. To share BigQuery data without copying with other organizations internally/externally, use Analytics Hub (publishers expose listings; subscribers subscribe and reference). This avoids duplication and sync overhead. Map "copy-free data sharing = Analytics Hub."
Continue reading — free sign-up
You're reading the free preview. Sign up free to read this section in full, plus every chapter (including 4+) and all questions.