What's changed: In-scope coverage: data prep/ingestion/storage/stores/extraction services
1.3Data Splitting, Quality, and Bias
Understand train/validation/test splitting and preventing data leakage, detecting class imbalance and bias (SageMaker Clarify), and checking data quality.
To evaluate a model correctly, split data into train/validation/test and ensure test information does not leak into training. Checking bias and imbalance also matters.
1.3.1Splitting and bias detection
- Train/validation/test: for fitting, tuning, and final evaluation; split before training.
- Data leakage: when test/future info contaminates training, evaluation is over-optimistic. Prevent it.
- SageMaker Clarify: detects bias in data/models and provides explainability (feature importance).
Common on MLA: over-optimistic eval = data leakage (transform before split / test info contamination), bias/explainability = SageMaker Clarify, split time-series chronologically.
Correct evaluation needs "proper splitting, leakage prevention, and bias checks." Split data into train (fit) / validation (tune) / test (final eval), and split before preprocessing. k-fold cross-validation estimates generalization, but split time-series chronologically (not randomly) with walk-forward validation so you don’t predict the past from the future. Stratified sampling preserves class ratios, useful for imbalanced classification. The biggest pitfall, data leakage, arises from (1) fitting scalers/encoders before the split, (2) features containing future info, or (3) duplicate records spread across train and test—inflating evaluation. SageMaker Clarify detects pre-training data bias (class imbalance, attribute skew) and post-training model bias, and provides SHAP-based explainability (feature importance). Continuously check data quality (missing rates, distributions, schema) with tools like Glue Data Quality. The axes: "over-optimistic = suspect leakage," "fairness/explainability = Clarify," "time-series = chronological split."
| Issue | Approach |
|---|---|
| Estimate generalization | k-fold cross-validation |
| Split time-series | Chronological (walk-forward) |
| Split imbalanced classes | Stratified sampling |
| Bias/explainability | SageMaker Clarify (SHAP) |
Scenario: a demand-forecast model’s evaluation is way off in production. The cause is often data leakage—a random split of time-series let the future bleed into train. Fix with a chronological split + walk-forward validation. Check pre/post-training bias with SageMaker Clarify and explain contributions with SHAP. Also verify duplicate records don’t span train/test.
Q. Main cause of over-optimistic eval? Data leakage. Q. Split time-series? Chronological (walk-forward). Q. Split imbalanced data? Stratified sampling. Q. Bias/explainability? SageMaker Clarify (SHAP). Q. Estimate generalization? k-fold cross-validation.
Watch the mix-ups: (1) Randomly splitting time-series leaks—always chronological. (2) Even in CV, fit preprocessing inside each fold or the estimate is optimistic. (3) Clarify detects/explains but doesn’t auto-fix fairness—remediation is a design choice. (4) A very high validation score can signal overfitting or leakage.
Split time-series chronologically (not randomly) so you do not predict the past from the future; use time-series cross-validation.
1.3.2In-scope services for data prep, ingestion, and storage
ML data preparation spans a wide range of in-scope ingestion, transform, and storage services. For ingestion, receive streams with Amazon Kinesis Data Streams (retention/replay) or Amazon Data Firehose (no-code delivery to S3/OpenSearch), and ingest camera video with Amazon Kinesis Video Streams. Prepare data with AWS Glue DataBrew (no-code cleaning), run large distributed processing with Amazon EMR (Spark/Hadoop), and do real-time stream aggregation with Amazon Managed Service for Apache Flink. Collect from SaaS/external or on-prem via AWS DataSync (NFS/SMB sync). Orchestrate with AWS Step Functions (serverless state machine) or, for existing Airflow assets, Amazon Managed Workflows for Apache Airflow (MWAA). Decouple with Amazon Simple Queue Service (queue) and notify with Amazon Simple Notification Service (fan-out). Govern fine-grained data-lake permissions with AWS Lake Formation.
For storage, the standard for training data/model artifacts is Amazon S3 (low-cost, durable), archival is Amazon S3 Glacier, shared files are Amazon Elastic File System, high-throughput HPC/ML is Amazon FSx (for Lustre, an S3-integrated parallel FS), and on-prem tiering is AWS Storage Gateway. Choose purpose-built data stores: columnar analytics = Amazon Redshift, full-text/vector search = Amazon OpenSearch Service, low-latency cache = Amazon ElastiCache, key-value = Amazon DynamoDB, document = Amazon DocumentDB, graph = Amazon Neptune. To extract from unstructured data, use Amazon Textract for document text, Amazon Transcribe for audio, Amazon Translate for translation, Amazon Comprehend Medical for clinical text, and Amazon Mechanical Turk for large-scale human labeling.
1.3.3Section summary
- Split train/val/test before training; prevent leakage
- Bias/explainability = SageMaker Clarify; split time-series chronologically
- Ingest = Kinesis/Firehose/DataSync; prep = Glue DataBrew/EMR/Flink; storage = S3/EFS/FSx; stores = Redshift/OpenSearch/DynamoDB and more
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. What is a classic cause of over-optimistic model evaluation?
Q2. Which SageMaker feature detects bias and provides explainability (e.g., feature importance)?
Q3. What is the correct way to split time-series data?