What's changed: Created Professional Machine Learning Engineer Chapter 5 (Domain 5 "Pipelines": data/model validation, consistent preprocessing, MLFlow hosting, components/parameters/triggers/Cloud Build/Cloud Run, orchestration (Kubeflow Pipelines/Vertex AI Pipelines/Cloud Composer), hybrid/multicloud, TFX/Kubeflow DSL; retraining policy, CI/CD (Cloud Build/Jenkins), Vertex AI Experiments/Vertex ML Metadata, model/data versioning, lineage).
5.1End-to-end ML pipelines
Understand data and model validation, consistent preprocessing between training and serving, hosting third-party pipelines (MLFlow), identifying components/parameters/triggers/compute (Cloud Build, Cloud Run), orchestration frameworks (Kubeflow Pipelines, Vertex AI Pipelines, Cloud Composer), hybrid/multicloud strategies, and system design with TFX components or Kubeflow DSL.
The heart of MLOps is the end-to-end pipeline that removes manual work. Automate data ingestion→validation→training→evaluation→deployment, made reproducible.
5.1.1Orchestration and validation
Automate ML pipelines with an orchestration framework. For Google-managed, serverless ML pipelines, use Vertex AI Pipelines (runs Kubeflow Pipelines / TFX); for general workflow/data orchestration (Airflow-based), use Cloud Composer. Embed data validation (schema/distribution anomalies) and model validation (meets thresholds) to ensure quality. Keeping consistent preprocessing between training and serving is essential to prevent training/serving skew—share preprocessing via TFX Transform. Map "managed ML pipeline = Vertex AI Pipelines" and "general workflow scheduling/dependencies = Cloud Composer (Airflow)."
Continue reading — free sign-up
You're reading the free preview. Sign up free to read this section in full, plus every chapter (including 4+) and all questions.