Skip to content

First run of composer-dev start fails due to PGDATA leaking into Airflow container env #142

Description

@tpwo

Hello, I noticed that on the first start with a PostgreSQL database engine, composer-dev start always fails with:

psycopg2.OperationalError: could not translate host name
"composer-local-dev-db-" to address: Name or service not known

Second composer-dev start was then always successful when I tried a few times.

EDIT: at first I was misled and thought the issue was caused by a race condition. But the proposed fix didn't solve the issue, so I dig more to find the real cause, and then updated this issue and PR with a fix.

Root cause

The Airflow container's entrypoint.sh (line 110) runs sudo chmod -R o+xrw $PGDATA when $PGDATA is set in the environment. The problem is that PGDATA was being passed to the Airflow container via **default_db_variables spread in get_default_environment_variables():

"GCP_PROJECT": self.project_id,
"GOOGLE_CLOUD_PROJECT": self.project_id,
**default_db_variables,
**self.get_environment_variables_for_image_version(),

Postgres requires strict permissions (0700 or 0750) on its data directory. The chmod corrupts those permissions, Postgres refuses to start, its container exits, its hostname disappears from Docker DNS -- which produces the misleading DNS resolution error I shared above.

This only happens on the first start because postgresql_data/ does not exist yet. Postgres initializes the data directory during container startup, and the Airflow entrypoint's chmod corrupts permissions in that window. On subsequent starts the data directory is already initialized, so Postgres starts instantly before the chmod can do damage.

Fix

Filter default_db_variables to only pass AIRFLOW__*-prefixed vars to the Airflow container. Postgres-specific vars (PGDATA, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB) are not needed by Airflow.

In composer_local_dev/environment.py (line ~697):

# Before:
**default_db_variables,

# After:
**{k: v for k, v in default_db_variables.items() if k.startswith("AIRFLOW__")},

Steps to reproduce

  1. Fresh environment (empty postgresql_data/ directory)
  2. composer-dev start

Expected behavior

Environment starts successfully on first attempt.

Actual behavior

Environment fails to start.

Traceback:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not
translate host name "composer-local-dev-db-<env-name>" to address:
Name or service not known

Workaround

Run composer-dev start a second time. Subsequent starts succeed because postgresql_data/ already exists and Postgres initialises instantly, so the chmod no longer has time to corrupt permissions before Postgres finishes starting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions