[Proposal] Self-Healing Retries

[daniyarndh]

Currently, Airflow task retries are passive and fixed. When a task fails (e.g., retries=3, retry_delay=5m), Airflow re-executes the exact same code with the exact same executor configuration and resource requests.

For transient code bugs, this is fine. However, in modern data engineering, many failures are caused by transient infrastructure and operational bottlenecks:

1. Out-of-Memory Errors: A batch job processes 20% more data than usual, triggering a Kubernetes OOMKilled signal. Retrying with the same memory allocation fails 3 times in a row, paging on-call engineers at 2 AM.
2. API Rate Limits: Hitting an endpoint with standard 5-minute retries often extends rate-limit penalties or results in IP bans.
3. Database Deadlocks / Lock Timeouts: Repeating a task instantly on the same high-concurrency worker queue prolongs resource contention.

Engineers routinely over-provision resources, for instance, requesting 32 GB RAM for a task that needs 4 GB on 29 out of 30 days, solely to prevent edge-case retry failures - wasting substantial compute budget.

Proposed Solution

We propose introducing Self-Healing Retries capability. Instead of treating all retries identically, Airflow should allow tasks to dynamically mutate their executor specs, execution queues, or retry strategies based on the caught exception or exit code. As a result, Airflow can automatically double the memory allocation after an Out-of-Memory crash, back off during API rate limits, or move jobs to a low-concurrency queue during database locks.

Airflow here acts as the decision brain (intercepting error classes and altering retry specs), while the underlying executor (Kubernetes, Celery, ECS) acts as the execution muscle (provisioning the updated container).

[potiuk]

That is - again - discussion that should happen on devlist (and Ideally in human words). Most likely you will also need to come and explain your ideas, consequences, and answer questions of people (after initial discussion) on a dev call - see our wiki for schedule. One of such callsis tomorrow.