ERROR: You need to initialize the database. Please run airflow db init - when upgrading chart from before 7.14 to 8.8.0

[jstefanska]

EDIT: We fixed it by adding airflow db init to container command, but still if anybody has info why it was removed in the first place I'd appreciate it - from my understanding, it won't work without initializing metadata db, so why was it removed?

Hi, I have some issue with Airflow deployment with Celery execution on GKE cluster. As this is my first time working with Airflow, and we had a very old version that was not very well maintained by different team, I'm having a bit of a struggle with understanding the error I have.

Each of my pods (web, scheduler, worker) have the following error: ERROR: You need to initialize the database. Please run airflow db init. Make sure the command is run using Airflow version 2.6.3. and it's in CrashLoopBackOff because of it. I see that in the old Chart there was airflow init db command in starting script for scheduler. How was it replaced?
I use postgres built-in, as it's our scratch environment. Is it an issue with connecting to the DB?
check-db can connect to postgres built-in.

This is my scheduler manifest:

Init Containers:
  check-db:
    Container ID:  containerd://6d987466ce34ac63badc354fa065c95c3738a55cc77ba142eaf4fbffae7c36ed
    Image:         apache/airflow:2.6.3-python3.9
    Image ID:      docker.io/apache/airflow@sha256:199883a2b0bd67629ae2fb7f708ad12d79be32db0062d8f46821d6f10a58de54
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec timeout 60s airflow db check
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 08 Feb 2024 10:56:50 +0100
      Finished:     Thu, 08 Feb 2024 10:56:54 +0100
    Ready:          True
    Restart Count:  1
    Environment Variables from:
      data-airflow-test-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:                                root
      DATABASE_PASSWORD:                            <set to the key 'postgresql-password' in secret 'data-airflow'>  Optional: false
      CONNECTION_CHECK_MAX_COUNT:                   0
      GOOGLE_APPLICATION_CREDENTIALS:               /etc/google/credentials/service-account.json
      AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT:       False
      AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE:         False
      AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC:  1
      AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC:        1
      AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG:       1
      GOOGLE_APPLICATION_CREDENTIALS:               /etc/google/credentials/service-account.json
      AIRFLOW__WEBSERVER__RBAC:                     False
      AIRFLOW__API__AUTH_BACKEND:                   airflow.api.auth.backend.basic_auth
      AIRFLOW__WEBSERVER__SECRET_KEY:               <set to the key 'airflow-web-secret-key' in secret 'data-airflow'>         Optional: false
    Mounts:
      /etc/google/credentials/ from google-application-credentials (ro)
      /usr/local/airflow/logs from logs-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-q4h8j (ro)
Containers:
  airflow-scheduler:
    Container ID:  containerd://1580d306237819d1d6aabf9d8fca0e9dcc5dae2769f1a4d3189d84f5ab571672
    Image:         apache/airflow:2.6.3-python3.9
    Image ID:      docker.io/apache/airflow@sha256:199883a2b0bd67629ae2fb7f708ad12d79be32db0062d8f46821d6f10a58de54
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec airflow scheduler -n -1
    State:          Running
      Started:      Thu, 08 Feb 2024 10:58:56 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 08 Feb 2024 10:58:04 +0100
      Finished:     Thu, 08 Feb 2024 10:58:13 +0100
    Ready:          True
    Restart Count:  4
    Limits:
      memory:  900Mi
    Requests:
      cpu:     2600m
      memory:  460Mi
    Environment Variables from:
      data-airflow-test-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:                                root
      DATABASE_PASSWORD:                            <set to the key 'postgresql-password' in secret 'data-airflow'>  Optional: false
      CONNECTION_CHECK_MAX_COUNT:                   0
      GOOGLE_APPLICATION_CREDENTIALS:               /etc/google/credentials/service-account.json
      AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT:       False
      AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE:         False
      AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC:  1
      AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC:        1
      AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG:       1
      GOOGLE_APPLICATION_CREDENTIALS:               /etc/google/credentials/service-account.json
      AIRFLOW__WEBSERVER__RBAC:                     False
      AIRFLOW__API__AUTH_BACKEND:                   airflow.api.auth.backend.basic_auth
      AIRFLOW__WEBSERVER__SECRET_KEY:               <set to the key 'airflow-web-secret-key' in secret 'data-airflow'>         Optional: false
    Mounts:
      /etc/google/credentials/ from google-application-credentials (ro)
      /usr/local/airflow/logs from logs-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-q4h8j (ro)