This guide is part of Managing Spatial Assets with Dagster, within the broader Orchestrating Spatial ETL Pipelines reference.

Backfilling Satellite Scene Partitions in Dagster

The Problem: A Full Re-Run Is the Wrong Tool for a Historical Fix

Satellite archives grow one acquisition date at a time, and reprocessing needs are almost always bounded: a cloud-mask algorithm changed, a two-week gap appeared after an outage, or a CRS-registration fix must be applied to last quarter’s scenes. Re-running the entire pipeline to repair a window is slow, expensive, and risks re-touching scenes that were already correct.

Treating history as monolithic breaks scene pipelines in predictable ways:

A backfill is many runs, not one long run A fourteen-day backfill request expands into fourteen partition runs shown as a strip. Twelve complete, one fails and is retried individually, and one is skipped because no scene was acquired that day. Because status is tracked per partition, the operator can see exactly which days are still missing without reading any logs. backfill 03-01 → 03-14 · one run per partition 03-01 03-14 skipped no scene failed retry this one Twelve days are done and stay done. The retry re-runs one partition, not the range, and the status strip is the whole status report. A skipped partition is a first-class outcome — recording "no scene existed" is what stops the same gap being investigated every month.
  • Cost scales with the archive, not the fix. Recomputing three years of daily scenes to correct one month burns compute proportional to the whole history instead of the affected window.
  • No parallelism across dates. A single sequential re-run processes dates one after another, when the dates are independent and could fan out across workers.
  • One bad scene aborts everything. Without per-partition isolation, a corrupt acquisition halfway through kills the run and loses the progress on every prior date.
  • No record of what actually reprocessed. A blanket re-run leaves no per-date status, so you cannot prove which scenes are now current.

Version and Environment Compatibility

Dagster Backfill capability Notes
1.7.x – 1.8.x Range backfills, per-partition status, RetryPolicy, backfill concurrency Recommended; robust missing-only backfills and UI status
1.6.x Range backfills, RetryPolicy Stable backfills; some concurrency controls added later in the series
1.5.x Range backfills Works, but pin exactly; treat asset-check gating as advisory 
pip install "dagster>=1.7,<1.9" dagster-webserver "geopandas>=1.0" "shapely>=2.0" "pyarrow>=14"

How a Range Backfill Recomputes Only the Gaps

Range backfill recomputing only missing scene partitions A daily partition strip from June 1 to June 6 where June 3 and June 4 are unmaterialized. A backfill launched over the whole range skips the four materialized dates and fans out parallel runs for only the two missing dates, each with its own retry policy. DailyPartitionsDefinition — Jun 1 … Jun 6 Jun 1 ok Jun 2 ok Jun 3 missing Jun 4 missing Jun 5 ok Jun 6 ok backfill "missing only" over the range run: Jun 3 RetryPolicy backoff run: Jun 4 RetryPolicy backoff four materialized dates skipped — only the two gaps recompute, in parallel

Recipe: A DailyPartitionsDefinition Scene Asset and a Range Backfill

The asset below is date-partitioned by acquisition day and carries a RetryPolicy, so any single failed scene retries with backoff rather than sinking the backfill. Reading context.partition_key selects exactly one date; the backfill then drives the range.

import logging
from dagster import (
    asset,
    AssetExecutionContext,
    DailyPartitionsDefinition,
    RetryPolicy,
    Backoff,
)
import geopandas as gpd
import shapely

logger = logging.getLogger(__name__)

scene_partitions = DailyPartitionsDefinition(start_date="2022-01-01")


@asset(
    partitions_def=scene_partitions,
    io_manager_key="geoparquet_io",
    group_name="scenes",
    # A failed date retries up to 3 times with exponential backoff, so a
    # transient download 429 does not fail the whole backfill.
    retry_policy=RetryPolicy(max_retries=3, delay=10, backoff=Backoff.EXPONENTIAL),
)
def scene_footprints(context: AssetExecutionContext) -> gpd.GeoDataFrame:
    """Process the satellite scenes for exactly one acquisition date.

    context.partition_key is the ISO date string; the backfill fans out one
    run per date across the requested window, recomputing only missing ones.
    """
    acq_date: str = context.partition_key
    context.log.info("Processing scenes for %s", acq_date)

    scenes = gpd.read_parquet(f"s3://spatial-lake/raw/scenes/{acq_date}.parquet")
    if scenes.crs is None or scenes.crs.to_epsg() != 3857:
        scenes = scenes.to_crs("EPSG:3857")

    # Vectorized validity repair before load (Shapely 2.0 array API).
    invalid = ~shapely.is_valid(scenes.geometry.values)
    if invalid.any():
        logger.warning("Repairing %d invalid footprints on %s", int(invalid.sum()), acq_date)
        scenes.loc[invalid, scenes.geometry.name] = shapely.make_valid(
            scenes.geometry.values[invalid]
        )

    scenes["acq_date"] = acq_date
    context.add_output_metadata({"acq_date": acq_date, "scenes": len(scenes)})
    return scenes

Launch the historical window as a backfill from the command line. Dagster fans out one run per date across the range and, when you choose the missing-only option, skips dates that already materialized successfully:

# Recompute the March 2023 window; --from/--to bound the partition range.
dagster asset backfill --select scene_footprints \
  --partition-range 2023-03-01...2023-03-31

To target only failed and never-run dates programmatically, filter the partition keys against materialization status before submitting the backfill:

from dagster import DagsterInstance

def missing_scene_dates(from_date: str, to_date: str) -> list[str]:
    """Return partition keys in the window that are unmaterialized or failed."""
    instance = DagsterInstance.get()
    all_keys = scene_partitions.get_partition_keys_in_range(
        # PartitionKeyRange is accepted directly by the definition helper
        partition_key_range=(from_date, to_date)  # type: ignore[arg-type]
    )
    materialized = instance.get_materialized_partitions(
        asset_key=scene_footprints.key
    )
    return [key for key in all_keys if key not in materialized]

Key Implementation Notes

  • The RetryPolicy is what makes a wide backfill survivable. Satellite source APIs throttle; a per-partition exponential backoff absorbs transient 429s so one flaky date does not require re-launching the whole window.
  • Missing-only backfills keep cost proportional to the gap. Because Dagster records per-partition materialization, a range backfill over a quarter that has two missing weeks recomputes two weeks of compute — not the quarter.
  • Concurrency should be sized to the slowest dependency. A month-long daily backfill is 30 independent runs; left unbounded they can open 30 simultaneous PostGIS connections or 30 concurrent API pulls. Cap it (below) to the download or write path, not the core count.
  • Failed partitions stay visible and re-targetable. After retries, any still-failed date remains marked failed in the backfill view, so you fix the root cause and re-launch a narrow backfill over just those keys.
  • Reprojection and repair live inside the partition. Doing the to_crs and vectorized make_valid per date keeps each partition self-contained and deterministic, which is the property that lets any single date be recomputed in isolation.
Backfills need their own concurrency cap Two timelines of provider requests. Under the daily schedule one partition runs per day and request pressure is flat. When a fourteen-day backfill launches without a cap, all fourteen partitions start at once and produce a spike far above the daily rate, which is what triggers the quota block. With a cap of three the same backfill spreads the same total work under the ceiling. requests per minute against the provider provider ceiling daily uncapped backfill capped at 3 — same work, longer, allowed The daily run tells you nothing about backfill pressure: one partition per day is a rate, fourteen partitions at once is a burst.

Bound the fan-out with a run-level concurrency cap so the backfill throttles to the write path:

from dagster import define_asset_job

# Benchmarked against one PostGIS instance: 6 concurrent scene dates
# sustained the highest steady throughput; beyond ~10 the write lock
# contended and total wall-clock time increased.
scene_backfill_job = define_asset_job(
    "scene_backfill_job",
    selection="scene_footprints",
    config={"execution": {"config": {"multiprocess": {"max_concurrent": 6}}}},
)

Troubleshooting Scene Backfills

Failure Root Cause Fix
Backfill recomputes dates that already succeeded Launched with “all partitions” instead of “missing only” Re-launch selecting missing/failed partitions, or pre-filter keys against materialized status
Backfill saturates the source API with 429s Unbounded partition concurrency during fan-out Set max_concurrent or a tag-based concurrency limit sized to the API quota
One failed date marks the whole backfill failed No RetryPolicy and downstream treats any failure as fatal Add a RetryPolicy with backoff; failed partitions isolate and stay re-targetable
PostGIS connections exhausted mid-backfill One connection per concurrent partition run Lower max_concurrent to the connection budget; use pool_pre_ping on the engine
get_partition_keys_in_range raises on the boundary date Off-by-one on an inclusive vs exclusive range end Confirm the end date is inside the partition set and after start_date

Integration Note

Backfilling closes the loop on the asset graph in managing spatial assets with Dagster: the same date-partitioned scene_footprints asset that a schedule advances one day at a time is what a backfill sweeps across history when an algorithm changes. Where the tile-partitioned asset recipe defines the spatial dimension of the grid, this date dimension defines the temporal one — and a MultiPartitionsDefinition combining both lets a backfill target a specific tile across a specific window, recomputing one corner of the space-time grid without disturbing the rest.