docs(adr): defer ADR 0036 — use the real Airflow Task SDK for now#360
Open
neochaotic wants to merge 1 commit into
Open
docs(adr): defer ADR 0036 — use the real Airflow Task SDK for now#360neochaotic wants to merge 1 commit into
neochaotic wants to merge 1 commit into
Conversation
The runtime compatibility shim was fully prototyped on branch vanity/airflow-runtime-shim (it worked; ~260 MB → ~0.16 MB airflow surface). We are not shipping it: growing it provider-by-provider is whack-a-mole with unbounded maintenance debt on our side, while the base image's apache-airflow-task-sdk already makes provider hooks work today (the env-secrets backend reads AIRFLOW_CONN_*) at the cost of image size, not our code. Decision: keep the real Airflow Task SDK in the task image (status quo) — maximum connector compatibility, ~zero maintenance debt. The slim shim is an image-size optimization deferred until the product is mature; the branch + the two planning docs are the blueprint. ADR 0035 (keyless) stays as the direction for future native connectors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the decision to defer the Airflow runtime compatibility shim (ADR 0036) and keep the real Airflow Task SDK in the task image for now.
Why
The shim was fully prototyped on
vanity/airflow-runtime-shim— it worked and shrank the per-task airflow surface from ~260 MB to ~0.16 MB. But growing it provider-by-provider is whack-a-mole: each provider drags moreairflow.*surface and every Airflow minor drifts, so the maintenance debt is ours and unbounded. Meanwhile the base image'sapache-airflow-task-sdkalready makes provider hooks work today (the env-secrets backend readsAIRFLOW_CONN_*), at the cost of image size — not our code.Decision
docs/planning/{airflow-connector-compatibility,connectors-two-tier-model}.mdare the blueprint.Docs-only. The
vanity/airflow-runtime-shimbranch is kept as a reference.