Skip to content

Commit a4a6162

Browse files
author
Matthew Byng-Maddick
committed
feat: add support for .extraEnv (defaulting to []) for the various pod types
As per #629, add support for .extraEnv for the various pod types, including some of the validation and warnings. This is generally better where we either use an environment variable that's something new, or that we are overriding something in airflow.config (eg. setting the log level at DEBUG on workers but INFO everywhere else) Signed-off-by: Matthew Byng-Maddick <[email protected]>
1 parent 420eae2 commit a4a6162

File tree

10 files changed

+72
-21
lines changed

10 files changed

+72
-21
lines changed

‎charts/airflow/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ Parameter | Description | Default
241241
`scheduler.podDisruptionBudget.*` | configs for the PodDisruptionBudget of the scheduler | `<see values.yaml>`
242242
`scheduler.logCleanup.*` | configs for the log-cleanup sidecar of the scheduler | `<see values.yaml>`
243243
`scheduler.numRuns` | the value of the `airflow --num_runs` parameter used to run the airflow scheduler | `-1`
244+
`scheduler.extraEnv` | extra environment to make available in the scheduler Pods | `[]`
244245
`scheduler.extraPipPackages` | extra pip packages to install in the scheduler Pods | `[]`
245246
`scheduler.extraVolumeMounts` | extra VolumeMounts for the scheduler Pods | `[]`
246247
`scheduler.extraVolumes` | extra Volumes for the scheduler Pods | `[]`
@@ -270,6 +271,7 @@ Parameter | Description | Default
270271
`web.service.*` | configs for the Service of the web pods | `<see values.yaml>`
271272
`web.readinessProbe.*` | configs for the web Pods' readiness probe | `<see values.yaml>`
272273
`web.livenessProbe.*` | configs for the web Pods' liveness probe | `<see values.yaml>`
274+
`web.extraEnv` | extra environment to make available in the web Pods | `[]`
273275
`web.extraPipPackages` | extra pip packages to install in the web Pods | `[]`
274276
`web.extraVolumeMounts` | extra VolumeMounts for the web Pods | `[]`
275277
`web.extraVolumes` | extra Volumes for the web Pods | `[]`
@@ -298,6 +300,7 @@ Parameter | Description | Default
298300
`workers.celery.*` | configs for the celery worker Pods | `<see values.yaml>`
299301
`workers.terminationPeriod` | how many seconds to wait after SIGTERM before SIGKILL of the celery worker | `60`
300302
`workers.logCleanup.*` | configs for the log-cleanup sidecar of the worker Pods | `<see values.yaml>`
303+
`workers.extraEnv` | extra environment to make available in the worker Pods | `[]`
301304
`workers.extraPipPackages` | extra pip packages to install in the worker Pods | `[]`
302305
`workers.extraVolumeMounts` | extra VolumeMounts for the worker Pods | `[]`
303306
`workers.extraVolumes` | extra Volumes for the worker Pods | `[]`
@@ -324,6 +327,7 @@ Parameter | Description | Default
324327
`triggerer.podDisruptionBudget.*` | configs for the PodDisruptionBudget of the triggerer Deployment | `<see values.yaml>`
325328
`triggerer.capacity` | maximum number of triggers each triggerer will run at once (sets `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`) | `1000`
326329
`triggerer.livenessProbe.*` | liveness probe for the triggerer Pods | `<see values.yaml>`
330+
`triggerer.extraEnv` | extra environment to make available in the triggerer Pods | `[]`
327331
`triggerer.extraPipPackages` | extra pip packages to install in the triggerer Pods | `[]`
328332
`triggerer.extraVolumeMounts` | extra VolumeMounts for the triggerer Pods | `[]`
329333
`triggerer.extraVolumes` | extra Volumes for the triggerer Pods | `[]`
@@ -350,6 +354,7 @@ Parameter | Description | Default
350354
`flower.basicAuthSecret` | the name of a pre-created secret containing the basic authentication value for flower | `""`
351355
`flower.basicAuthSecretKey` | the key within `flower.basicAuthSecret` containing the basic authentication string | `""`
352356
`flower.service.*` | configs for the Service of the flower Pods | `<see values.yaml>`
357+
`flower.extraEnv` | extra environment to make available in the flower Pod | `[]`
353358
`flower.extraPipPackages` | extra pip packages to install in the flower Pod | `[]`
354359
`flower.extraVolumeMounts` | extra VolumeMounts for the flower Pods | `[]`
355360
`flower.extraVolumes` | extra Volumes for the flower Pods | `[]`

‎charts/airflow/templates/NOTES.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@
1111
{{- $remote_logging_enabled = true }}
1212
{{- end }}
1313
{{- end }}
14+
{{- range $env := .Values.workers.extraEnv }}
15+
{{- if has $env.name $remote_logging_envvars }}
16+
{{- $remote_logging_enabled = true }}
17+
{{- end }}
18+
{{- end }}
1419

1520
{{- /* if an extra volume has been mounted for worker logs */ -}}
1621
{{- $extra_volumes_worker_logs := false }}
@@ -44,6 +49,7 @@
4449
{{- $fernet_key_warning = false }}
4550
{{- end }}
4651
{{- end }}
52+
{{/* we still warn even if we set the fernet key in sub-extraEnvs as they might be different, and that would be bad */}}
4753

4854
{{- /* if we show the webserver secret_key warning */ -}}
4955
{{- $web_secret_warning := true }}
@@ -61,6 +67,11 @@
6167
{{- $web_secret_warning = false }}
6268
{{- end }}
6369
{{- end }}
70+
{{- range $env := .Values.web.extraEnv }}
71+
{{- if has $env.name $web_secret_envvars }}
72+
{{- $web_secret_warning = false }}
73+
{{- end }}
74+
{{- end }}
6475

6576
{{- /* if we show the external database password warning */ -}}
6677
{{- $external_database_password_warning := false }}

‎charts/airflow/templates/_helpers/pods.tpl‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -580,6 +580,9 @@ EXAMPLE USAGE: {{ include "airflow.env" (dict "Release" .Release "Values" .Value
580580
{{- end }}
581581

582582
{{- /* user-defined environment variables */ -}}
583+
{{- if .extraEnv }}
584+
{{ toYaml .extraEnv }}
585+
{{- end }}
583586
{{- if .Values.airflow.extraEnv }}
584587
{{ toYaml .Values.airflow.extraEnv }}
585588
{{- end }}

‎charts/airflow/templates/_helpers/validate-values.tpl‎

Lines changed: 28 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -51,24 +51,36 @@
5151
{{- end }}
5252
{{- end }}
5353

54-
{{/* Checks for `airflow.config` */}}
55-
{{- if .Values.airflow.config.AIRFLOW__CORE__EXECUTOR }}
56-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__EXECUTOR`, it will be automatically set from `airflow.executor`!" nil }}
57-
{{- end }}
58-
{{- if or .Values.airflow.config.AIRFLOW__CORE__DAGS_FOLDER }}
59-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__DAGS_FOLDER`, it will be automatically set from `dags.path`!" nil }}
60-
{{- end }}
61-
{{- if or (.Values.airflow.config.AIRFLOW__CELERY__BROKER_URL) (.Values.airflow.config.AIRFLOW__CELERY__BROKER_URL_CMD) }}
62-
{{ required "Don't define `airflow.config.AIRFLOW__CELERY__BROKER_URL`, it will be automatically set by the chart!" nil }}
63-
{{- end }}
64-
{{- if or (.Values.airflow.config.AIRFLOW__CELERY__RESULT_BACKEND) (.Values.airflow.config.AIRFLOW__CELERY__RESULT_BACKEND_CMD) }}
65-
{{ required "Don't define `airflow.config.AIRFLOW__CELERY__RESULT_BACKEND`, it will be automatically set by the chart!" nil }}
54+
{{/* defines checks for various environment overrides */}}
55+
{{- define "airflow.environment_check" }}
56+
{{- if .Values.AIRFLOW__CORE__EXECUTOR }}
57+
{{ required (printf "Don't define `AIRFLOW__CORE__EXECUTOR` under %s, it will be automatically set from `airflow.executor`!" .TreeName) nil }}
58+
{{- end }}
59+
{{- if .Values.AIRFLOW__CORE__DAGS_FOLDER }}
60+
{{ required (printf "Don't define `AIRFLOW__CORE__DAGS_FOLDER` under %s, it will be automatically set from `dags.path`!" .TreeName) nil }}
61+
{{- end }}
62+
{{- if or (.Values.AIRFLOW__CELERY__BROKER_URL) (.Values.AIRFLOW__CELERY__BROKER_URL_CMD) }}
63+
{{ required (printf "Don't define `AIRFLOW__CELERY__BROKER_URL` under %s, it will be automatically set by the chart!") nil }}
64+
{{- end }}
65+
{{- if or (.Values.AIRFLOW__CELERY__RESULT_BACKEND) (.Values.AIRFLOW__CELERY__RESULT_BACKEND_CMD) }}
66+
{{ required (printf "Don't define `AIRFLOW__CELERY__RESULT_BACKEND` under %s, it will be automatically set by the chart!") nil }}
67+
{{- end }}
68+
{{- if or (.Values.AIRFLOW__CORE__SQL_ALCHEMY_CONN) (.Values.AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD) }}
69+
{{ required (printf "Don't define `AIRFLOW__CORE__SQL_ALCHEMY_CONN` under %s, it will be automatically set by the chart!") nil }}
70+
{{- end }}
71+
{{- if or (.Values.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN) (.Values.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD) }}
72+
{{ required (printf "Don't define `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` under %s, it will be automatically set by the chart!") nil }}
73+
{{- end }}
6674
{{- end }}
67-
{{- if or (.Values.airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN) (.Values.airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD) }}
68-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN`, it will be automatically set by the chart!" nil }}
75+
{{/* Checks for `airflow.config` and the extraEnvs */}}
76+
{{- include "airflow.environment_check" (dict "Values" .Values.airflow.config "TreeName" "`airflow.config`") }}
77+
{{- range $tree := (list "airflow" "flower" "scheduler" "triggerer" "web" "workers") }}
78+
{{- $envDict := (dict) }}
79+
{{- range $env := (index $.Values $tree "extraEnv") }}
80+
{{- $_ := (set $envDict $env.name "1") }}
81+
{{- end }}
82+
{{- include "airflow.environment_check" (dict "Values" $envDict "TreeName" (printf "`%s.extraEnv`" $tree)) }}
6983
{{- end }}
70-
{{- if or (.Values.airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN) (.Values.airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD) }}
71-
{{ required "Don't define `airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`, it will be automatically set by the chart!" nil }}
7284
{{- end }}
7385

7486
{{/* Checks for `scheduler.logCleanup` */}}

‎charts/airflow/templates/flower/flower-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.flower.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.flower.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.flower.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.flower.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.flower.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.flower.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -100,7 +101,7 @@ spec:
100101
envFrom:
101102
{{- include "airflow.envFrom" . | indent 12 }}
102103
env:
103-
{{- include "airflow.env" . | indent 12 }}
104+
{{- $env | indent 12 }}
104105
ports:
105106
- name: flower
106107
containerPort: 5555

‎charts/airflow/templates/scheduler/scheduler-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.scheduler.affinity) }}
33
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.scheduler.tolerations) }}
44
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.scheduler.securityContext) }}
5+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.scheduler.extraEnv) }}
56
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.scheduler.extraPipPackages }}
67
{{- $extraVolumeMounts := .Values.scheduler.extraVolumeMounts }}
78
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -110,7 +111,7 @@ spec:
110111
envFrom:
111112
{{- include "airflow.envFrom" . | indent 12 }}
112113
env:
113-
{{- include "airflow.env" . | indent 12 }}
114+
{{- $env | indent 12 }}
114115
command:
115116
{{- include "airflow.command" . | indent 12 }}
116117
args:

‎charts/airflow/templates/triggerer/triggerer-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.triggerer.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.triggerer.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.triggerer.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.triggerer.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.triggerer.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.triggerer.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -99,7 +100,7 @@ spec:
99100
envFrom:
100101
{{- include "airflow.envFrom" . | indent 12 }}
101102
env:
102-
{{- include "airflow.env" . | indent 12 }}
103+
{{- $env | indent 12 }}
103104
command:
104105
{{- include "airflow.command" . | indent 12 }}
105106
args:

‎charts/airflow/templates/webserver/webserver-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.web.affinity) }}
33
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.web.tolerations) }}
44
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.web.securityContext) }}
5+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.web.extraEnv) }}
56
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.web.extraPipPackages }}
67
{{- $extraVolumeMounts := .Values.web.extraVolumeMounts }}
78
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -103,7 +104,7 @@ spec:
103104
envFrom:
104105
{{- include "airflow.envFrom" . | indent 12 }}
105106
env:
106-
{{- include "airflow.env" . | indent 12 }}
107+
{{- $env | indent 12 }}
107108
command:
108109
{{- include "airflow.command" . | indent 12 }}
109110
args:

‎charts/airflow/templates/worker/worker-statefulset.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.workers.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.workers.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.workers.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.workers.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.workers.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.workers.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -104,7 +105,7 @@ spec:
104105
envFrom:
105106
{{- include "airflow.envFrom" . | indent 12 }}
106107
env:
107-
{{- include "airflow.env" . | indent 12 }}
108+
{{- $env | indent 12 }}
108109
# have dumb-init only send signals to direct child process (needed for celery workers to warm shutdown)
109110
- name: DUMB_INIT_SETSID
110111
value: "0"

‎charts/airflow/values.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -629,6 +629,9 @@ scheduler:
629629
##
630630
schedulerAgeBeforeCheck: 180
631631

632+
## extra environment for just the scheduler Pods
633+
extraEnv: []
634+
632635
## extra pip packages to install in the scheduler Pods
633636
##
634637
## ____ EXAMPLE _______________
@@ -786,6 +789,9 @@ web:
786789
timeoutSeconds: 5
787790
failureThreshold: 6
788791

792+
## extra environment for just the web Pods
793+
extraEnv: []
794+
789795
## extra pip packages to install in the web Pods
790796
##
791797
## ____ EXAMPLE _______________
@@ -955,6 +961,9 @@ workers:
955961
##
956962
intervalSeconds: 900
957963

964+
## extra environment for just the worker Pods
965+
extraEnv: []
966+
958967
## extra pip packages to install in the worker Pod
959968
##
960969
## ____ EXAMPLE _______________
@@ -1068,6 +1077,9 @@ triggerer:
10681077
timeoutSeconds: 60
10691078
failureThreshold: 5
10701079

1080+
## extra environment for just the triggerer Pods
1081+
extraEnv: []
1082+
10711083
## extra pip packages to install in the triggerer Pod
10721084
##
10731085
## ____ EXAMPLE _______________
@@ -1204,6 +1216,9 @@ flower:
12041216
timeoutSeconds: 5
12051217
failureThreshold: 6
12061218

1219+
## extra environment for just the flower Pods
1220+
extraEnv: []
1221+
12071222
## extra pip packages to install in the flower Pod
12081223
##
12091224
## ____ EXAMPLE _______________

0 commit comments

Comments
 (0)