What's changed: Created Professional Machine Learning Engineer Chapter 2 (Domain 2 "Collaboration": org data sources (Cloud Storage/BigQuery/Spanner/Cloud SQL/Spark/Hadoop), Vertex AI datasets, preprocessing (Dataflow/TFX/BigQuery), Vertex AI Feature Store, PII/PHI privacy; Jupyter backends (Vertex AI Workbench/Colab Enterprise/Dataproc), security, frameworks (TensorFlow/PyTorch/sklearn/Spark/JAX), Model Garden, experiment tracking (Vertex AI Experiments/Kubeflow Pipelines/TensorBoard), gen AI evaluation).
2.1Exploring and preprocessing organization-wide data
Understand exploring organization-wide data (Cloud Storage, BigQuery, Spanner, Cloud SQL, Apache Spark, Apache Hadoop), organizing data types (tabular/text/speech/image/video), managing datasets in Vertex AI, preprocessing (Dataflow, TensorFlow Extended [TFX], BigQuery), creating/consolidating features in Vertex AI Feature Store, privacy of data usage (PII/PHI), and ingesting data into Vertex AI for inference.
ML quality is decided by data. Explore data scattered across the org, preprocess it efficiently, and shape it into reusable features—considering privacy from the start.
2.1.1Data sources and preprocessing
Data lives in diverse places: Cloud Storage (unstructured/files), BigQuery (analytic tables), Spanner/Cloud SQL (operational DBs), and Apache Spark/Apache Hadoop (existing big data). Organize types (tabular/text/speech/image/video) and manage them as Vertex AI managed datasets. For preprocessing, choose Dataflow (Apache Beam) for large-scale stream/batch transforms, TensorFlow Extended (TFX) for production ML-pipeline preprocessing, and BigQuery for SQL-native aggregation/features. Map "large-scale parallel preprocessing = Dataflow" and "preprocess in SQL = BigQuery."
2.1.2Feature Store and privacy
Consolidate computed features in Vertex AI Feature Store to reuse across teams/models, using the same features for training and serving to prevent duplicate computation and training/serving skew. Consider privacy from the start: when handling PII (personally identifiable information) or PHI (protected health information), protect it with Sensitive Data Protection (DLP) masking/pseudonymization, least privilege, and VPC Service Controls perimeters. Map "feature reuse and consistency = Feature Store" and "protect sensitive data = redact PII/PHI."
Common: requirement → means. E.g., "large-scale stream/batch preprocessing in parallel" = Dataflow; "aggregate to features in SQL" = BigQuery; "preprocessing components in a production ML pipeline" = TFX; "reuse features across teams, consistent in training/serving" = Vertex AI Feature Store; "protect data with PII/PHI" = DLP (Sensitive Data Protection) + least privilege.
Watch the mix-ups: (1) Dataflow (parallel big-data transforms) vs BigQuery (SQL aggregation) are different preprocessing means—choose by scale and shape. (2) Without a Feature Store, features drift between training/serving (a skew source). (3) Redact PII/PHI early—after-the-fact protection is risky.
2.1.3Section summary
- Preprocess = Dataflow (parallel)/BigQuery (SQL)/TFX (pipeline); data = Vertex AI managed datasets
- Feature reuse and training/serving consistency = Vertex AI Feature Store
- Redact PII/PHI early with DLP; protect with least privilege/perimeters
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To preprocess large log data in parallel for both stream and batch (transform/clean), which is best?
Q2. To reuse computed features across teams/models and use the same values in training and serving to prevent skew, which is best?
Q3. To build features by SQL aggregation from tables already in BigQuery, which is best?
Q4. Data with PII/PHI will be used for training. What should you do first?
Q5. To define preprocessing components consistent across training and serving within a production ML pipeline, which is best?