This guide is part of Orchestrating Spatial ETL Pipelines, the reference for turning extract, clean, and load scripts into scheduled, restartable, observable spatial data products.

Choosing an Orchestrator for Spatial ETL

Every team building a production spatial pipeline eventually faces the same fork: Airflow, Prefect, or Dagster. The comparison is usually made on generic data-engineering criteria — UI polish, community size, cloud offering — and those criteria lead spatial teams to the wrong answer. Geospatial ETL has three properties that a warehouse-row pipeline rarely does. The unit of work is a tile, scene, or coordinate window whose count is frequently unknown until a query runs; the outputs are large binary rasters and vector partitions that must never travel through the orchestrator’s own message bus; and reprocessing is spatial as much as temporal — “re-run tile 85283473 for last quarter” is a routine request. An orchestrator that models these three properties naturally saves you thousands of lines of glue; one that fights them turns every backfill into an incident.

This guide compares the three orchestrators strictly through that spatial lens. It gives a real feature matrix, a decision framework that maps workload shape to a recommended tool, and the same reference task expressed three ways so the differences are concrete rather than rhetorical. The deeper mechanics of each live in their own references: building Airflow DAGs for spatial ETL, orchestrating spatial pipelines with Prefect, and managing spatial assets with Dagster.

The Decision, as a Flow

The choice reduces to three questions asked in order. The moment one of them lands, you have a default — you can still override it for organisational reasons, but the workload has spoken. Ask them from the most spatial property inward: runtime cardinality first, then partitioned reprocessing, then cadence.

The rest of this guide backs each branch with evidence: the feature matrix shows why the questions are ordered this way, the implementation section shows the same task in all three tools, and the framework at the end turns specific workload shapes into a recommendation.

Prerequisites & Environment

You do not need all three orchestrators installed to follow the reasoning, but reproducing the reference task in each requires the following. Pin the orchestrator major version — every one of these tools has had breaking API changes across majors.

• Python 3.10+ — required for modern type hints and current releases of all three orchestrators.
• Apache Airflow 2.9+ — for the stable TaskFlow API and dynamic task mapping via .expand().
• Prefect 2.14+ — for the @flow/@task API and .map() fan-out (the 2.x line; 3.x keeps this surface).
• Dagster 1.7+ — for the asset and partition APIs used here.
• Shared spatial stack — geopandas>=0.14, rasterio>=1.3, and pyarrow>=14 for the GeoParquet handoffs every example assumes.

pip install "apache-airflow>=2.9" "prefect>=2.14" "dagster>=1.7" \
    "geopandas>=0.14" "rasterio>=1.3" "pyarrow>=14"

Verify each orchestrator imports cleanly and reports the expected major version before building anything on top of it:

import importlib.metadata as md

for pkg in ("apache-airflow", "prefect", "dagster"):
    try:
        print(pkg, md.version(pkg))
    except md.PackageNotFoundError:
        print(pkg, "not installed")

One environment rule cuts across all three tools: the orchestrator moves references, never rasters. Every example below hands off spatial data as a GeoParquet path or URI in object storage, and only the small identifier — a tile_id or scene_id — flows through the orchestrator’s own state. This is the single most important spatial-orchestration decision, and it is independent of which tool you pick.

Version / Compatibility Matrix

The matrix below is the comparison, scored strictly for spatial ETL. Read it top to bottom as reasons the decision-flow questions are ordered the way they are: runtime fan-out and partitioned backfills are where the tools genuinely diverge, while scheduling and operator breadth are where Airflow’s maturity still leads.

No row makes one tool win outright; the rows are weighted differently by workload. A nightly government-portal ingest weights scheduling and operator ecosystem heavily and lands on Airflow. A query that returns an unpredictable number of Sentinel-2 scenes weights dynamic fan-out and lands on Prefect. A tiled elevation product that gets reprocessed by region weights partitioning and lands on Dagster.

Step-by-Step Implementation

To make the matrix concrete, here is one reference task — discover a set of scene IDs, then process each — expressed in all three tools. The processing function is deliberately identical and pure so the only thing that varies is the orchestration wrapper.

Step 1 — Isolate the spatial logic from the orchestrator

Before any orchestrator enters the picture, the geospatial work lives in plain functions that take and return paths. This is what makes tasks testable and portable across all three tools, and it is the prerequisite the troubleshooting reference below keeps returning to.

from pathlib import Path

import geopandas as gpd


def discover_scene_ids(bbox: tuple[float, float, float, float]) -> list[str]:
    """Return the scene IDs intersecting a bounding box — count unknown until now."""
    # In production this queries a STAC API or manifest; the point is the
    # length of the returned list is a *runtime* value, not a parse-time one.
    return query_manifest(bbox)  # -> ['S2A_2026_07_08_...', ...]


def process_scene(scene_id: str, out_root: Path) -> str:
    """Pure spatial work: read, normalize CRS, write GeoParquet. Returns a URI."""
    gdf: gpd.GeoDataFrame = read_scene_features(scene_id)
    gdf = gdf.to_crs(3857)
    dest = out_root / f"{scene_id}.parquet"
    gdf.to_parquet(dest)  # the raster/vector never touches the orchestrator
    return str(dest)

Step 2 — Express it in Airflow as a mapped TaskFlow DAG

Airflow resolves the fan-out with dynamic task mapping. Note that discover_scene_ids runs as a task and its return value — a list of small strings — is what .expand() maps over. This is safe precisely because the mapped values are IDs, the pattern detailed in passing GeoDataFrames between Airflow tasks.

from datetime import datetime
from pathlib import Path

from airflow.decorators import dag, task


@dag(schedule="0 3 * * *", start_date=datetime(2026, 7, 1), catchup=False)
def scene_pipeline():
    @task
    def discover() -> list[str]:
        return discover_scene_ids(bbox=(-124.5, 32.5, -114.1, 42.0))

    @task
    def process(scene_id: str) -> str:
        return process_scene(scene_id, Path("/data/out"))

    process.expand(scene_id=discover())  # mapped count resolved at run time


scene_pipeline()

Step 3 — Express it in Prefect as a mapped flow

Prefect resolves the same fan-out imperatively, inside the running flow. The unbounded list from discover_scene_ids is an ordinary Python value, so .map() fans out over it with no parse-time graph and no XCom size constraint.

from pathlib import Path

from prefect import flow, task


@task(retries=3, retry_delay_seconds=10)
def process(scene_id: str, out_root: Path) -> str:
    return process_scene(scene_id, out_root)


@flow(name="scene-pipeline")
def scene_pipeline() -> list[str]:
    scene_ids = discover_scene_ids(bbox=(-124.5, 32.5, -114.1, 42.0))
    futures = process.map(scene_ids, out_root=Path("/data/out"))
    return [f.result() for f in futures]  # runtime fan-out, any cardinality

Step 4 — Express it in Dagster as partitioned assets

Dagster models each scene as a partition of one asset. The scene IDs become a dynamic partition set, and a backfill of any subset launches exactly those runs — the property that makes Dagster the cheapest place to do spatial reprocessing.

from pathlib import Path

from dagster import asset, DynamicPartitionsDefinition, AssetExecutionContext

scenes = DynamicPartitionsDefinition(name="scenes")


@asset(partitions_def=scenes)
def scene_features(context: AssetExecutionContext) -> str:
    scene_id = context.partition_key
    return process_scene(scene_id, Path("/data/out"))

Step 5 — Score the three against the workload

With the same task in hand, the choice is no longer abstract. If discover_scene_ids returns a wildly variable count each run, Step 3 was the least friction. If the request “re-materialize the 2026-Q2 scenes for the Sierra tiles” is routine, Step 4 made it a one-click backfill. If the pipeline runs at 03:00 daily and already leans on S3, Postgres, and HTTP operators, Step 2 fit the existing platform. That mapping — from the shape of the work to the tool — is the framework in the next section.

Advanced Patterns & Edge Cases

Very large runtime fan-outs

When a single query yields tens of thousands of tiles, the fan-out mechanism itself becomes the bottleneck. Airflow’s dynamic mapping pushes every mapped value through the metadata database, so a 50,000-way .expand() degrades the scheduler; Prefect keeps the fan-out in-process and bounds it with a task runner’s concurrency limit instead. The concrete throttling and batching pattern for tile-scale fan-out is worked through in parallel tile downloads with Prefect task mapping.

Spatial backfills that must stay cheap

A backfill request in spatial ETL is usually two-dimensional — a time range and a set of tiles — and a monolithic DAG cannot express that without re-running everything. Dagster’s partition selection is the natural fit here: reprocessing is a query over partition keys rather than a full re-run. The mechanics of selecting and launching those partitions are detailed in backfilling satellite scene partitions in Dagster, with the partition scheme itself covered in partitioning spatial assets by tile in Dagster.

Idempotency under retries, regardless of tool

None of the three tools makes your writes idempotent for you — that is your task body’s responsibility. Whichever orchestrator you pick, every scene write must be keyed on its stable identifier and use upsert or atomic-replace semantics so a retry overwrites rather than appends. The Airflow-specific version of this discipline, including atomic renames and delete-then-insert loads, is in writing idempotent spatial ETL tasks in Airflow, and it transfers unchanged to Prefect and Dagster.

Performance Optimization

The orchestrator rarely dominates wall-clock time — the spatial work does — but the fan-out mechanism determines how much of that work runs concurrently and how much the control plane wastes. The table below is a representative measurement of dispatching 5,000 scene tasks (each a no-op stand-in for process_scene) on a single machine with a concurrency limit of 16, isolating scheduling overhead from spatial compute.

The takeaway is not a leaderboard — it is that fan-out cardinality is the axis to benchmark for your own workload. Measure the dispatch cost at your real tile count before committing, using a no-op task body so you separate control-plane overhead from spatial compute:

import time

from prefect import flow, task


@task
def noop(scene_id: str) -> str:
    return scene_id  # stand-in: measures fan-out cost, not spatial compute


@flow
def bench(n: int) -> float:
    start = time.perf_counter()
    futures = noop.map([f"scene_{i}" for i in range(n)])
    [f.result() for f in futures]
    return time.perf_counter() - start


if __name__ == "__main__":
    for n in (500, 5_000, 20_000):
        print(n, f"{bench(n):.1f}s")

Integration into ETL Pipelines

Whichever orchestrator wins, it plugs into the same surrounding contract. Discovery emits a manifest of IDs; every task reads and writes GeoParquet in object storage keyed on those IDs; validation routes failures to a dead-letter store instead of failing the run. The Load stage’s target is a separate decision that the PostGIS versus DuckDB comparison for analytical loads works through, and the scheduling and watermark discipline that keeps incremental runs from re-processing old windows is covered in scheduling and incremental spatial loads. Because the spatial logic lives in pure functions (Step 1), migrating from one orchestrator to another later is a re-wrapping exercise, not a rewrite — a point developed for the two most common candidates in the focused Airflow versus Prefect head-to-head.

Troubleshooting Reference

Related

• Building Airflow DAGs for Spatial ETL — DAG structure, sensors, pools, and idempotent operators for fixed-schedule spatial pipelines
• Orchestrating Spatial Pipelines with Prefect — Python-native flows and runtime fan-out for query-driven spatial work
• Managing Spatial Assets with Dagster — partitioned raster and vector assets with targeted, parallel backfills
• Airflow vs Prefect for Geospatial Workflows — a focused head-to-head on a tile-download, reproject, and load workflow
• Orchestrating Spatial ETL Pipelines — the reference model these orchestrators each express differently