How to Handle Rate Limits When Downloading OSM Data

This page is part of Fetching OSM Data via Overpass API, which itself sits inside Mastering Geospatial Data Ingestion in Python.

Problem: An automated pipeline that issues Overpass API requests without rate-limit awareness will receive HTTP 429 Too Many Requests responses, accumulate temporary IP bans, and silently return empty or truncated datasets — breaking every downstream join and topology operation that depends on complete OSM coverage.

Why Rate Limit Failures Break Spatial Pipelines

A bare requests.get() loop against the public Overpass endpoint fails in ways that are easy to miss until data quality checks surface the damage:

• Silent data loss: A 429 mid-run means the affected tile or bounding-box slice returns nothing. Unless you validate feature counts after every batch, the gap propagates into your GeoDataFrame and corrupts spatial joins and rasterization steps.
• IP-level bans escalate quickly: Overpass administrators actively block IPs showing bot-like fixed-interval polling. Recovering from a ban can take hours and disrupts every pipeline that shares the IP.
• Cascading timeouts in orchestrated DAGs: When a queued Airflow or Prefect task blocks on a hung HTTP call, downstream tasks wait, SLA windows breach, and the entire pipeline stalls.
• Non-deterministic reruns: Without caching, every pipeline restart re-hits the API, compounding quota consumption and making backfills expensive — exactly the pattern that triggers daily download cap enforcement.

Version and Environment Compatibility

Install the required packages:

pip install requests geopandas shapely

Request Flow: Backoff, Cache, and Fallback

The diagram below shows the decision path a single query takes through the resilient client — from cache lookup through retry escalation to fallback routing.

Production-Ready OSMRateLimitHandler Recipe

The function below is a single copy-pasteable class that covers every step in the flow above. It wraps requests.Session, parses Retry-After, applies exponential backoff with jitter, and writes results to a SHA-256-keyed disk cache. Drop it into any ETL module that calls Fetching OSM Data via Overpass API functions.

import time
import random
import hashlib
import logging
from pathlib import Path
from typing import Optional

import requests

logger = logging.getLogger(__name__)


class OSMRateLimitHandler:
    """
    Resilient Overpass API client with exponential backoff, jitter, and disk caching.

    Args:
        base_url:    Overpass interpreter endpoint (swap to a mirror on sustained 429s).
        cache_dir:   Directory for SHA-256-keyed XML response cache files.
        max_retries: Maximum retry attempts before raising or routing to fallback.
        base_delay:  Initial wait in seconds; doubles on each retry (before jitter).
        contact:     Identifies your pipeline in the User-Agent header (required by
                     Overpass fair-use policy — anonymous agents are throttled first).
    """

    def __init__(
        self,
        base_url: str = "https://overpass-api.de/api/interpreter",
        cache_dir: str = ".osm_cache",
        max_retries: int = 6,
        base_delay: float = 2.0,
        contact: str = "osm-etl@example.com",
    ) -> None:
        self.base_url = base_url
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(parents=True, exist_ok=True)
        self.max_retries = max_retries
        self.base_delay = base_delay

        self.session = requests.Session()
        self.session.headers.update({
            # Descriptive User-Agent is mandatory; generic strings attract preemptive throttling.
            "User-Agent": f"OSM-ETL-Pipeline/1.0 ({contact})",
            "Accept-Encoding": "gzip, deflate",
        })

    # ------------------------------------------------------------------
    # Internal helpers
    # ------------------------------------------------------------------

    def _cache_key(self, query: str) -> str:
        """Return a 20-char hex prefix of the SHA-256 hash of the raw QL query."""
        return hashlib.sha256(query.encode()).hexdigest()[:20]

    def _get_cached(self, key: str) -> Optional[str]:
        cache_path = self.cache_dir / f"{key}.xml"
        if cache_path.exists():
            logger.debug("Cache hit: %s", key)
            return cache_path.read_text(encoding="utf-8")
        return None

    def _save_cache(self, key: str, content: str) -> None:
        (self.cache_dir / f"{key}.xml").write_text(content, encoding="utf-8")
        logger.debug("Cached response: %s", key)

    # ------------------------------------------------------------------
    # Public interface
    # ------------------------------------------------------------------

    def execute_query(self, query: str, timeout: int = 180) -> str:
        """
        POST an Overpass QL query and return the raw XML response text.

        On HTTP 429:
          1. Read Retry-After header; use its value if present.
          2. Otherwise apply exponential backoff with uniform jitter.
          3. After max_retries, raise RuntimeError — caller should route
             to a Geofabrik .osm.pbf fallback (see fetch_with_fallback).
        """
        key = self._cache_key(query)
        cached = self._get_cached(key)
        if cached:
            return cached

        payload = {"data": query}

        for attempt in range(self.max_retries):
            try:
                response = self.session.post(
                    self.base_url, data=payload, timeout=timeout
                )
                response.raise_for_status()

                self._save_cache(key, response.text)
                logger.info(
                    "Overpass query succeeded (attempt %d, %d bytes)",
                    attempt + 1, len(response.content),
                )
                return response.text

            except requests.exceptions.HTTPError as exc:
                status = exc.response.status_code

                if status == 429:
                    # Prefer the server-provided wait; fall back to exponential backoff.
                    retry_after_raw = exc.response.headers.get("Retry-After")
                    if retry_after_raw is not None:
                        wait_time = float(retry_after_raw)
                    else:
                        # Exponential backoff: base_delay * 2^attempt + jitter ∈ [0, 1)
                        wait_time = self.base_delay * (2 ** attempt) + random.uniform(0, 1)

                    logger.warning(
                        "HTTP 429 on attempt %d — backing off %.1fs", attempt + 1, wait_time
                    )
                    time.sleep(wait_time)

                else:
                    # 4xx/5xx errors other than 429 are not retriable here.
                    raise

            except requests.exceptions.RequestException as exc:
                # Network-level errors (DNS, connection reset, timeout) — still retriable.
                wait_time = self.base_delay * (2 ** attempt) + random.uniform(0, 1)
                logger.warning(
                    "Network error on attempt %d: %s — retrying in %.1fs",
                    attempt + 1, exc, wait_time,
                )
                time.sleep(wait_time)

        raise RuntimeError(
            f"Overpass query failed after {self.max_retries} attempts: {query[:80]}..."
        )

    def fetch_with_fallback(
        self,
        query: str,
        pbf_path: Optional[str] = None,
        timeout: int = 180,
    ) -> str:
        """
        Attempt execute_query; on exhausted retries, return a sentinel string that
        the caller can detect to trigger local .osm.pbf parsing via pyosmium.

        Args:
            query:    Overpass QL string.
            pbf_path: Path to a Geofabrik .osm.pbf file used as local fallback.
            timeout:  Per-request HTTP timeout in seconds.

        Returns:
            Raw XML string on success, or "__FALLBACK__:<pbf_path>" on failure.
        """
        try:
            return self.execute_query(query, timeout=timeout)
        except RuntimeError as exc:
            if pbf_path:
                logger.error(
                    "Overpass quota exhausted — routing to local .pbf: %s. Error: %s",
                    pbf_path, exc,
                )
                return f"__FALLBACK__:{pbf_path}"
            raise

Key Implementation Notes

• Retry-After takes priority over local calculations. Overpass sometimes returns a precise server-side cooldown. Ignoring it and using your own timer risks re-hitting the server during a backoff window it has explicitly declared, compounding the ban risk.
• Jitter is not optional. Parallel workers from a single IP that all failed on the same timestamp will converge on the same retry moment without it, replicating the thundering-herd problem that triggered the 429 in the first place. random.uniform(0, 1) adds up to one second of spread per attempt.
• HTTPError and RequestException are caught separately. raise_for_status() raises HTTPError only if a response was received. Lower-level network failures (socket timeouts, connection resets) raise RequestException before a response object exists, so catching only HTTPError would let those propagate without retry.
• SHA-256 cache keys are collision-safe at pipeline scale. A 20-character hex prefix gives 2^80 distinct keys — sufficient for any realistic set of distinct QL queries in a geospatial pipeline, while keeping filenames short and human-scannable.
• The contact header is enforced by fair-use policy. Overpass administrators explicitly throttle generic agents (python-requests/2.31). Include an email or project URL so queries are traceable and your pipeline can be contacted before a hard ban is issued.
• fetch_with_fallback returns a sentinel string rather than raising. This lets orchestrator tasks (Airflow BranchPythonOperator, Prefect conditional) detect the failure mode and dispatch a separate .pbf parse task without catching exceptions in DAG logic.

Query Optimization to Reduce API Pressure

Rate-limit compliance starts before the HTTP request leaves your machine. Leaner payloads consume fewer server slots and reduce your odds of hitting daily quotas.

Tighten bounding boxes. Use precise (south,west,north,east) coordinates instead of country-level polygons. Overpass evaluates all nodes within the bounding box before tag-filtering — a tighter box means less server RAM consumed per query.

Limit output metadata. Use out body or out skel instead of out meta unless you explicitly need versioning, timestamps, or contributor IDs. out meta inflates response size and processing time.

Cache area IDs. area["name"="Berlin"]->.searchArea; resolves the named area server-side on every request. Cache the numeric area ID and substitute it directly in subsequent queries to skip that resolution step.

Chunk large extractions. Break continental or national queries into regional tiles. Parallelize with strict concurrency limits — one request per IP at a time on public endpoints, or two to three workers per IP on mirrors you have permission to hit harder.

Fallback Architecture for Quota Exhaustion

Public Overpass instances are shared resources. Production pipelines that need reliable scheduled downloads should implement at least one fallback tier:

• Geofabrik .osm.pbf regional extracts bypass the API entirely. Download and filter locally with pyosmium. This is the lowest-latency path for historical or bulk data runs, and it integrates cleanly with the sentinel pattern in fetch_with_fallback above.
• Self-hosted Overpass via Docker removes external rate limits. You control the slot count and can run as many concurrent queries as your hardware tolerates. Syncing with a local .pbf gives you the same query interface with predictable latency.
• Mirror rotation. Community mirrors (overpass.kumi.systems, overpass.openstreetmap.fr) share the same Overpass QL interface. Implement health checks and auto-switch base_url when the primary sustains 429s for more than 30 seconds.

For pipelines that also bulk-download satellite imagery, the same backoff pattern applies to USGS and Sentinel endpoints — the retry class above can be subclassed with a different base_url and max_retries budget.

Troubleshooting Reference

Integration Note

Drop OSMRateLimitHandler at the data-extraction step of your Fetching OSM Data via Overpass API workflow, immediately before you parse the returned XML into a GeoDataFrame. The cached .xml files persist across pipeline reruns, so if a downstream step such as schema harmonization fails and you need to re-run from extraction, you pay no additional API quota. Pair the sentinel fallback with a pyosmium parse task that applies the same attribute-extraction logic as your Overpass path, so the two code paths converge on an identical GeoDataFrame schema before any CRS normalization step runs.

For pipelines that paginate large feature sets, apply the same backoff class to STAC catalog requests — see Best Practices for STAC Catalog Pagination in Python for a direct analogue.

Related

• Fetching OSM Data via Overpass API — parent guide covering Overpass QL syntax, bounding box construction, and XML parsing
• Bulk Downloading Satellite Imagery — same backoff patterns applied to USGS EarthExplorer and Sentinel HTTP endpoints
• Automating USGS EarthExplorer Bulk Downloads with Requests — concrete retry implementation for a different rate-limited spatial API
• Best Practices for STAC Catalog Pagination in Python — pagination and throttling patterns for STAC endpoints