An orchestrator decides what should run, when dependencies are satisfied, and how outcomes are recorded. The transformation engine still does the actual compute. An asset-oriented orchestrator, a task scheduler and a Kubernetes job controller may coordinate the same pipeline with different operating costs and data-lineage guarantees.
One pipeline, three orchestration views
Imagine a batch job that ingests daily generation records, normalizes interval granularity, updates a warehouse fact table and publishes an analytical report.
Swipe or scroll sideways to read the full diagram.
Raw source records are validated, transformed into a warehouse fact table, then used for marts and reporting. Dagster can model resulting data assets, Airflow can schedule task dependencies, and Argo can launch containerized steps on Kubernetes.
Dagster models assets as datasets and other materializable outputs. Asset dependencies, partitions, checks and materialization events make it easier to reason about which dataset is stale or should be rebuilt. The trade-off is introducing a platform and execution model oriented around assets.
Airflow schedules directed acyclic graphs (DAGs) of tasks. It has broad integrations and clear time-interval scheduling, while lineage and asset semantics may require additional metadata or provider features. Airflow operators should submit large transformations to Spark, Dataflow, BigQuery or another execution engine rather than run memory-intensive data jobs in scheduler processes.
Argo Workflows coordinates containerized jobs as Kubernetes-native workflows. This can fit ephemeral batch execution and graphics processing unit (GPU) or central processing unit (CPU) tasks already packaged as images. Cluster policy, resource requests and observability become important responsibilities. An Argo Workflows controller is distinct from Argo CD, which manages GitOps application deployment.
How retries affect output correctness
An orchestrator marking an attempt failed says little about whether the underlying sink write committed. A retry can duplicate partitions or reapply stale changes unless the destination supports deterministic MERGE, write-to-staging-plus-swap or stable idempotency keys.
When backfilling historical days, pass the explicit data interval rather than today’s wall clock. Version upstream source snapshots and transformation code where reproducibility matters. An asset can be successfully materialized and still be incomplete; reconciliation and data-quality assertions must inspect its content.
| Decision | Useful signal |
|---|---|
| Strong dataset lineage and partitions | Asset-oriented orchestration |
| Many scheduled API and warehouse tasks | General DAG scheduler |
| Each step is an isolated container on an existing cluster | Kubernetes-native workflow |
| Long-running external transactional workflows | Durable execution system such as Temporal |
| Direct SQL transforms | dbt invoked by an orchestrator or scheduled warehouse |
Operating signals
Track queued versus running tasks, retry counts, resource saturation, missed schedules, source age and downstream data freshness. Recoverability requires knowing which interval failed and which outputs were already committed. An execution graph without lineage or data checks can appear healthy while delivering partial data.