Select the execution engine for the transformation, then coordinate its dependencies and interval-specific runs. The scheduler and transformation project have different responsibilities.
In this chapter
- Airflow, Composer, dbt, and repeatable backfillsSeparate orchestration from distributed execution and preserve interval correctness.
- Dagster, Argo and workload orchestration trade-offsAsset materialization, DAG scheduling, Kubernetes workflows and safe retries.
- Choosing a data transformation tool and execution engineUse data location, state, operation shape and deadlines to choose execution.
For a full recovery procedure, see resumable pipelines, replay and backfills.