Where is airflow-web pod logging written to?

[stanvv]

We're trying to collect all logging of our airflow pods for future analysis.
For this, we included logs.persistence.enabled=true in our helm command and updated our helm-values.yaml (see relevant settings below)

Subset of helm-values.yaml

airflow:
  config:
    AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
    AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "azure_logging"
    AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "wasb-airflow-logs"
    AIRFLOW__LOGGING__BASE_LOG_FOLDER: "/opt/airflow/logs/dags"
    AIRFLOW__WEBSERVER__ACCESS_LOGFILE: "/opt/airflow/logs/webserver/access.log"
    AIRFLOW__WEBSERVER__ERROR_LOGFILE: "/opt/airflow/logs/webserver/error.log"
    AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION: "/opt/airflow/logs/dags/dag_processor_manager/dag_processor_manager.log"

scheduler:
  logCleanup:
    enabled: False

logs: 
  persistence:
    enabled: True
    storageClass: azurefile

workers:
  logCleanup:
    enabled: False

pgbouncer:
  logDisconnections: 1
  logConnections: 1
  verbose: 1

This leads to pod logging being written to /opt/airflow/logs/.. which is mounted to a PV (Azure File Share). So far so good!

However, when we run kubectl logs airflow-web -n airflow, we see a whole bunch of logs that we don't see in any of our log files in our PV.
Please find the example here:

Snippet of kubectl logs

Hello from custom entrypoint (baked in docker)
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: /opt/airflow/logs/webserver/access.log /opt/airflow/logs/webserver/error.log
Access Logformat: 
=================================================================
/home/airflow/.local/lib/python3.7/site-packages/airflow/plugins_manager.py:258 RemovedInAirflow3Warning: This decorator is deprecated.
.....
[2023-02-01 09:08:18,364] {app.py:1742} ERROR - Exception on /admin/metrics/ [GET]

How can we write those logs to a path?

[thesuperzapper]

See my comment in #691 (comment)