What's changed: In-scope coverage: ops platform (ECS/EKS/CloudFormation/CDK/CLI/Code trio/SSM/Managed Grafana/Well-Architected/Cost Explorer/Budgets) + data-prep/ML (Glue DataBrew/SageMaker AI/Bedrock/Kendra)
3.1Operating Pipelines and Data Quality
Understand pipeline operations: monitoring (CloudWatch, job success/duration), automation via schedules and retries, and data-quality checks (Glue Data Quality, DLQ). The starting point for "Data Operations and Support" in DEA-C01.
A pipeline is not done once built—monitoring, auto-handling failures, and keeping data quality are key operations.
3.1.1Monitoring, automation, and quality
- Monitor: watch job success/duration/throughput in CloudWatch and alarm on failure.
- Automate/retry: schedule via EventBridge, retry with Step Functions, and isolate failed records in a DLQ.
- Data quality: validate with rules using Glue Data Quality to catch bad data early.
Common on DEA: detect/notify job failures = CloudWatch alarms, isolate failed records = DLQ, validate data-quality rules = Glue Data Quality, scheduled runs = EventBridge schedules.
Pipeline operations rest on three pillars: "monitoring, automation/retry, and data quality." Monitoring uses CloudWatch to watch job success, duration, and throughput, raising alarms (→ SNS, EventBridge) on failures or delays. Aggregate Glue/EMR/Step Functions logs in CloudWatch Logs and trace causes with Logs Insights. Automation/retry is hardened with EventBridge (schedule/event triggers) and Step Functions (per-step retries, exponential backoff, catch), isolating unprocessable records in a DLQ (dead-letter queue) for later reprocessing. The key to safe re-runs is idempotency (same input yields the same result). Data quality uses Glue Data Quality (define rules in DQDL—not-null/unique/range checks) to catch bad data early and quarantine it, protecting downstream. Beyond failure alerts, design for backfill (reprocessing history) and late-arriving data too.
| Operational need | Use |
|---|---|
| Detect/notify job failure | CloudWatch alarm → SNS |
| Schedule/retry | EventBridge / Step Functions |
| Isolate failed records | DLQ |
| Validate data quality | Glue Data Quality (DQDL) |
Scenario: a nightly ETL fails intermittently and bad data flows downstream. Trigger Step Functions on an EventBridge schedule with retries + exponential backoff per step. Send failed records to a DLQ. Validate not-null/range with Glue Data Quality before loading and quarantine bad rows. Notify job failures via a CloudWatch alarm to SNS and trace causes with Logs Insights.
Q. Notify job failure? CloudWatch alarm → SNS. Q. Isolate failed records? DLQ. Q. Validate data quality? Glue Data Quality (DQDL). Q. Schedule/retry? EventBridge / Step Functions. Q. Key to safe re-runs? Idempotency.
Watch the mix-ups: (1) Unbounded retries of non-idempotent processing cause duplicate inserts/double counting. (2) A DLQ is a holding area—you still need a reprocessing mechanism (monitor and replay). (3) Glue Data Quality "validates"—design the quarantine/notify follow-up for bad data. (4) Misconfigured alarm threshold/period causes false positives or misses.
Idempotent processing (re-runs do not change results) lets you retry/reprocess safely.
3.1.2In-scope services for the operations platform
The execution platform, IaC, CI/CD, and operations are also in scope for DEA-C01. For compute, beyond Amazon EC2, orchestrate containerized ETL with Amazon Elastic Container Service (ECS, AWS-native, minimal ops with Fargate) or Amazon Elastic Kubernetes Service (EKS, Kubernetes-standard). Define infrastructure declaratively from templates with AWS CloudFormation, from a programming language with the AWS Cloud Development Kit (CDK), and operate/automate via the AWS CLI. Automate releases with AWS CodeBuild (build/test) -> AWS CodeDeploy (controlled deploy) -> AWS CodePipeline (stage chaining). Manage fleet config, patching, and parameters with AWS Systems Manager; run OSS observability dashboards with Amazon Managed Grafana; and review architecture systematically with the AWS Well-Architected Tool. Visualize and forecast cost with AWS Cost Explorer and alert on budget thresholds with AWS Budgets.
| Area | Service | Key point |
|---|---|---|
| Container execution | ECS / EKS | AWS-native = ECS; K8s-standard = EKS |
| IaC | CloudFormation / CDK / CLI | Declarative / programmatic / command |
| CI/CD | CodeBuild / CodeDeploy / CodePipeline | Build / deploy / orchestrate |
| Ops & review | Systems Manager / Managed Grafana / Well-Architected Tool | Manage / visualize / review |
| Cost | Cost Explorer / Budgets | Visualize-forecast / budget alerts |
3.1.3Section summary
- Monitor (CloudWatch) / automate-retry (EventBridge, Step Functions, DLQ)
- Validate data quality with Glue Data Quality
- Platform = ECS/EKS; IaC = CloudFormation/CDK; CI/CD = Code trio; cost = Cost Explorer/Budgets
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to detect and be notified when an ETL job fails. What fits best?
Q2. You want to isolate records that failed processing for later inspection/reprocessing. What do you use?
Q3. You want rule-based validation to ensure data quality. Which fits best?