What's changed: In-scope coverage: added ingestion/migration services (AppFlow/Data Exchange/API Gateway/DataSync/Transfer Family/MGN/Discovery/Snow Family)
1.3Orchestration and Stream Processing
Understand pipeline orchestration (Step Functions / Managed Workflows for Apache Airflow), stream processing (Managed Service for Apache Flink), and lightweight transforms with Lambda.
Multiple ingest/transform steps are orchestrated to run in order automatically. Processing streams in flight (aggregations) also matters.
1.3.1Orchestration and processing
- Step Functions: orchestrate pipeline sequence/branch/parallel/retry as a state machine.
- Managed Workflows for Apache Airflow (MWAA): orchestrate pipelines with Airflow DAGs (good for existing Airflow assets).
- Managed Service for Apache Flink: aggregate/transform streams in flight (e.g., windowed aggregations).
- Lambda: for lightweight transforms and event-driven processing (can also transform within Firehose).
Common on DEA: orchestrate pipelines = Step Functions or MWAA (Airflow), real-time stream aggregation = Managed Service for Apache Flink, lightweight transform = Lambda. Choose MWAA if you have existing Airflow assets.
Treat pipelines as "orchestration" vs. "stream processing." For orchestration, Step Functions (a state machine declaring sequence/branch (Choice)/parallel (Map)/retry/error handling, serverless, integrating directly with many AWS services) and MWAA (Managed Workflows for Apache Airflow) (Python DAGs, leveraging existing Airflow assets and community operators); for simple chains Glue workflows, and for event-driven triggering EventBridge (schedule/event rules). To aggregate/transform a stream in flight, use Managed Service for Apache Flink (windowed aggregation, SQL/Java, event-time processing); for lightweight per-record or in-Firehose transforms, Lambda. The axes: "complex branching/multi-service = Step Functions," "Airflow compatibility/existing DAGs = MWAA," "real-time stream aggregation = Flink," "short transform = Lambda." Retry-on-failure and observability (CloudWatch logs/metrics, X-Ray) are also key to operational design.
| Goal | Use |
|---|---|
| Complex branching/multi-service orchestration | Step Functions |
| Orchestrate with existing Airflow DAGs | MWAA |
| Real-time stream aggregation | Managed Service for Apache Flink |
| Lightweight per-record transform | Lambda |
Scenario: automate daily ingest → Glue transform → Redshift load with retries. Trigger Step Functions on an EventBridge schedule and orchestrate Glue job → quality check → Redshift COPY in sequence/branches (retries and error notifications per step). Meanwhile, do real-time aggregation of arriving events with Managed Service for Apache Flink and light pre-delivery shaping with Lambda in Firehose.
Q. Complex branching orchestration? Step Functions. Q. Existing Airflow? MWAA. Q. Stream aggregation? Managed Service for Apache Flink. Q. Short transform? Lambda. Q. Schedule trigger? EventBridge.
Watch the mix-ups: (1) Orchestration (Step Functions/MWAA) ≠ stream processing (Flink)—Flink isn’t a "run things in order" tool. (2) MWAA shines with existing Airflow assets but costs more and has higher ops overhead than Step Functions. (3) Lambda has runtime/payload limits—unfit for large transforms (use Glue/EMR). (4) Glue workflows suit simple chains; complex branching needs Step Functions.
Glue has workflows too, but Step Functions suits complex branching/multi-service orchestration, and MWAA suits Airflow-compatibility requirements.
1.3.2In-scope services for ingestion and migration
DEA-C01 probes a variety of ingestion and migration services as data entry points. For external data, Amazon AppFlow performs no-code integration between SaaS (Salesforce/Zendesk, etc.) and AWS, while AWS Data Exchange subscribes to third-party curated data products for your analytics. App-originated data uses Amazon API Gateway as a managed HTTP/REST entry with throttling and auth, routing to Lambda/Kinesis. For migration from on-premises, the distinctions matter. AWS DataSync syncs NFS/SMB files to S3/EFS/FSx fast, incrementally, on a schedule; AWS Transfer Family accepts SFTP/FTPS/FTP file exchange into S3/EFS without running servers. Whole-server lift-and-shift is handled by AWS Application Migration Service (MGN) via block-level continuous replication, with AWS Application Discovery Service collecting server config, dependencies, and utilization to plan it first. For thin links or petabyte-scale data, AWS Snow Family ships offline via encrypted physical devices, and Snowball Edge can do edge preprocessing.
| Use | Service | When to use |
|---|---|---|
| Ingest from SaaS | Amazon AppFlow | No-code SaaS-to-AWS integration |
| Subscribe to external data | AWS Data Exchange | Third-party data products |
| Fast file sync | AWS DataSync | NFS/SMB to S3/EFS incremental |
| SFTP intake | AWS Transfer Family | SFTP/FTPS, serverless |
| Server migration | Application Migration Service / Discovery Service | MGN migrates; Discovery plans |
| Large offline transfer | AWS Snow Family | Thin links, PB scale |
1.3.3Section summary
- Orchestration = Step Functions / MWAA (Airflow)
- Real-time aggregation = Flink, lightweight transform = Lambda
- Ingest = AppFlow/Data Exchange/API Gateway; migrate = DataSync/Transfer Family/MGN/Snow Family
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. You want to orchestrate pipelines using existing Apache Airflow DAGs. Which service fits best?
Q2. You want real-time windowed aggregation of streaming data. Which service fits best?
Q3. You want to orchestrate a multi-step pipeline with sequence, branching, and retries. Which fits best?