What's changed: Created Professional Machine Learning Engineer Chapter 5 (Domain 5 "Pipelines": data/model validation, consistent preprocessing, MLFlow hosting, components/parameters/triggers/Cloud Build/Cloud Run, orchestration (Kubeflow Pipelines/Gemini Enterprise Agent Platform Pipelines/Cloud Composer), hybrid/multicloud, TFX/Kubeflow DSL; retraining policy, CI/CD (Cloud Build/Jenkins), Experiments on Agent Platform/Vertex ML Metadata, model/data versioning, lineage).
5.1End-to-end ML pipelines
Understand data and model validation, consistent preprocessing between training and serving, hosting third-party pipelines (MLFlow), identifying components/parameters/triggers/compute (Cloud Build, Cloud Run), orchestration frameworks (Kubeflow Pipelines, Gemini Enterprise Agent Platform Pipelines, Cloud Composer), hybrid/multicloud strategies, and system design with TFX components or Kubeflow DSL.
The heart of MLOps is the end-to-end pipeline that removes manual work. Automate data ingestion→validation→training→evaluation→deployment, made reproducible.
5.1.1Orchestration and validation
Automate ML pipelines with an orchestration framework. For Google-managed, serverless ML pipelines, use Gemini Enterprise Agent Platform Pipelines (runs Kubeflow Pipelines / TFX); for general workflow/data orchestration (Airflow-based), use Cloud Composer. Embed data validation (schema/distribution anomalies) and model validation (meets thresholds) to ensure quality. Keeping consistent preprocessing between training and serving is essential to prevent training/serving skew—share preprocessing via TFX Transform. Map "managed ML pipeline = Gemini Enterprise Agent Platform Pipelines" and "general workflow scheduling/dependencies = Cloud Composer (Airflow)."
5.1.2Components and triggers
Design pipelines by decomposing into components (preprocess/train/evaluate/deploy steps), parameters, triggers (schedule/event/code change), and compute needs. Use Cloud Build for CI-style build/containerization and Cloud Run for lightweight steps or serving. Describe system design with TFX components or Kubeflow DSL, and host third-party pipelines like MLFlow on Google Cloud when needed. Adopt hybrid/multicloud strategies per requirements. Map "componentize the pipeline = components/DSL" and "auto-trigger on code change = triggers (Cloud Build)."
Common: requirement → means. E.g., "run ML pipelines managed and serverless" = Gemini Enterprise Agent Platform Pipelines; "run Kubeflow/TFX pipelines" = Gemini Enterprise Agent Platform Pipelines; "schedule/dependency-manage general workflows" = Cloud Composer (Airflow); "consistent preprocessing for training/serving" = share via TFX Transform; "auto-trigger pipeline on code change" = Cloud Build triggers; "keep using MLFlow" = host it on Google Cloud.
Watch the mix-ups: (1) Choose Gemini Enterprise Agent Platform Pipelines (ML-specific, Kubeflow/TFX) vs Cloud Composer (general Airflow) by use. (2) Not sharing preprocessing between training/serving causes skew—unify via TFX Transform. (3) Manual retraining/deployment is the enemy of reproducibility—automate with triggers.
5.1.3Section summary
- Managed ML pipelines = Gemini Enterprise Agent Platform Pipelines (Kubeflow/TFX); general workflows = Cloud Composer
- Embed data/model validation; keep preprocessing consistent across training/serving to prevent skew
- Componentize via DSL; auto-trigger (Cloud Build); MLFlow can be hosted on Google Cloud
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. To run ML pipelines defined in Kubeflow Pipelines or TFX in a managed, serverless way, which is best?
Q2. Preprocessing differs between training and serving, causing training/serving skew. What is the fundamental fix?
Q3. To schedule general (incl. non-ML) workflows with dependencies on an Airflow base, which is best?
Q4. To auto-build and containerize an ML pipeline triggered by a code push to the repo, which is best?
Q5. You already run pipelines with MLFlow and want to keep using it hosted on Google Cloud. Which is correct?
Keep track of your progress
The full study guide is free to read. Sign up free to practice with the question bank, track what you have read, review your mistakes, and highlight passages.