Skip to content

Commit 13215ba

Browse files
author
Matthew Byng-Maddick
committed
feat: add support for .extraEnv (defaulting to []) for the various pod types
As per #629, add support for .extraEnv for the various pod types, including some of the validation and warnings. This is generally better where we either use an environment variable that's something new, or that we are overriding something in airflow.config (eg. setting the log level at DEBUG on workers but INFO everywhere else) Signed-off-by: Matthew Byng-Maddick <[email protected]>
1 parent c3339af commit 13215ba

File tree

10 files changed

+73
-23
lines changed

10 files changed

+73
-23
lines changed

‎charts/airflow/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,7 @@ Parameter | Description | Default
251251
`scheduler.podDisruptionBudget.*` | configs for the PodDisruptionBudget of the scheduler | `<see values.yaml>`
252252
`scheduler.logCleanup.*` | configs for the log-cleanup sidecar of the scheduler | `<see values.yaml>`
253253
`scheduler.numRuns` | the value of the `airflow --num_runs` parameter used to run the airflow scheduler | `-1`
254+
`scheduler.extraEnv` | extra environment to make available in the scheduler Pods | `[]`
254255
`scheduler.extraPipPackages` | extra pip packages to install in the scheduler Pods | `[]`
255256
`scheduler.extraVolumeMounts` | extra VolumeMounts for the scheduler Pods | `[]`
256257
`scheduler.extraVolumes` | extra Volumes for the scheduler Pods | `[]`
@@ -280,6 +281,7 @@ Parameter | Description | Default
280281
`web.service.*` | configs for the Service of the web pods | `<see values.yaml>`
281282
`web.readinessProbe.*` | configs for the web Pods' readiness probe | `<see values.yaml>`
282283
`web.livenessProbe.*` | configs for the web Pods' liveness probe | `<see values.yaml>`
284+
`web.extraEnv` | extra environment to make available in the web Pods | `[]`
283285
`web.extraPipPackages` | extra pip packages to install in the web Pods | `[]`
284286
`web.extraVolumeMounts` | extra VolumeMounts for the web Pods | `[]`
285287
`web.extraVolumes` | extra Volumes for the web Pods | `[]`
@@ -308,6 +310,7 @@ Parameter | Description | Default
308310
`workers.celery.*` | configs for the celery worker Pods | `<see values.yaml>`
309311
`workers.terminationPeriod` | how many seconds to wait after SIGTERM before SIGKILL of the celery worker | `60`
310312
`workers.logCleanup.*` | configs for the log-cleanup sidecar of the worker Pods | `<see values.yaml>`
313+
`workers.extraEnv` | extra environment to make available in the worker Pods | `[]`
311314
`workers.extraPipPackages` | extra pip packages to install in the worker Pods | `[]`
312315
`workers.extraVolumeMounts` | extra VolumeMounts for the worker Pods | `[]`
313316
`workers.extraVolumes` | extra Volumes for the worker Pods | `[]`
@@ -334,6 +337,7 @@ Parameter | Description | Default
334337
`triggerer.podDisruptionBudget.*` | configs for the PodDisruptionBudget of the triggerer Deployment | `<see values.yaml>`
335338
`triggerer.capacity` | maximum number of triggers each triggerer will run at once (sets `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`) | `1000`
336339
`triggerer.livenessProbe.*` | liveness probe for the triggerer Pods | `<see values.yaml>`
340+
`triggerer.extraEnv` | extra environment to make available in the triggerer Pods | `[]`
337341
`triggerer.extraPipPackages` | extra pip packages to install in the triggerer Pods | `[]`
338342
`triggerer.extraVolumeMounts` | extra VolumeMounts for the triggerer Pods | `[]`
339343
`triggerer.extraVolumes` | extra Volumes for the triggerer Pods | `[]`
@@ -360,6 +364,7 @@ Parameter | Description | Default
360364
`flower.basicAuthSecret` | the name of a pre-created secret containing the basic authentication value for flower | `""`
361365
`flower.basicAuthSecretKey` | the key within `flower.basicAuthSecret` containing the basic authentication string | `""`
362366
`flower.service.*` | configs for the Service of the flower Pods | `<see values.yaml>`
367+
`flower.extraEnv` | extra environment to make available in the flower Pod | `[]`
363368
`flower.extraPipPackages` | extra pip packages to install in the flower Pod | `[]`
364369
`flower.extraVolumeMounts` | extra VolumeMounts for the flower Pods | `[]`
365370
`flower.extraVolumes` | extra Volumes for the flower Pods | `[]`

‎charts/airflow/templates/NOTES.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@
1111
{{- $remote_logging_enabled = true }}
1212
{{- end }}
1313
{{- end }}
14+
{{- range $env := .Values.workers.extraEnv }}
15+
{{- if has $env.name $remote_logging_envvars }}
16+
{{- $remote_logging_enabled = true }}
17+
{{- end }}
18+
{{- end }}
1419

1520
{{- /* if an extra volume has been mounted for worker logs */ -}}
1621
{{- $extra_volumes_worker_logs := false }}
@@ -44,6 +49,7 @@
4449
{{- $fernet_key_warning = false }}
4550
{{- end }}
4651
{{- end }}
52+
{{/* we still warn even if we set the fernet key in sub-extraEnvs as they might be different, and that would be bad */}}
4753

4854
{{- /* if we show the webserver secret_key warning */ -}}
4955
{{- $web_secret_warning := true }}
@@ -61,6 +67,11 @@
6167
{{- $web_secret_warning = false }}
6268
{{- end }}
6369
{{- end }}
70+
{{- range $env := .Values.web.extraEnv }}
71+
{{- if has $env.name $web_secret_envvars }}
72+
{{- $web_secret_warning = false }}
73+
{{- end }}
74+
{{- end }}
6475

6576
{{- /* if we show the external database password warning */ -}}
6677
{{- $external_database_password_warning := false }}

‎charts/airflow/templates/_helpers/pods.tpl‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -582,6 +582,9 @@ EXAMPLE USAGE: {{ include "airflow.env" (dict "Release" .Release "Values" .Value
582582
{{- end }}
583583

584584
{{- /* user-defined environment variables */ -}}
585+
{{- if .extraEnv }}
586+
{{ toYaml .extraEnv }}
587+
{{- end }}
585588
{{- if .Values.airflow.extraEnv }}
586589
{{ toYaml .Values.airflow.extraEnv }}
587590
{{- end }}

‎charts/airflow/templates/_helpers/validate-values.tpl‎

Lines changed: 29 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -51,24 +51,35 @@
5151
{{- end }}
5252
{{- end }}
5353

54-
{{/* Checks for `airflow.config` */}}
55-
{{- if .Values.airflow.config.AIRFLOW__CORE__EXECUTOR }}
56-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__EXECUTOR`, it will be automatically set from `airflow.executor`!" nil }}
57-
{{- end }}
58-
{{- if or .Values.airflow.config.AIRFLOW__CORE__DAGS_FOLDER }}
59-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__DAGS_FOLDER`, it will be automatically set from `dags.path`!" nil }}
60-
{{- end }}
61-
{{- if or (.Values.airflow.config.AIRFLOW__CELERY__BROKER_URL) (.Values.airflow.config.AIRFLOW__CELERY__BROKER_URL_CMD) }}
62-
{{ required "Don't define `airflow.config.AIRFLOW__CELERY__BROKER_URL`, it will be automatically set by the chart!" nil }}
63-
{{- end }}
64-
{{- if or (.Values.airflow.config.AIRFLOW__CELERY__RESULT_BACKEND) (.Values.airflow.config.AIRFLOW__CELERY__RESULT_BACKEND_CMD) }}
65-
{{ required "Don't define `airflow.config.AIRFLOW__CELERY__RESULT_BACKEND`, it will be automatically set by the chart!" nil }}
66-
{{- end }}
67-
{{- if or (.Values.airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN) (.Values.airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD) }}
68-
{{ required "Don't define `airflow.config.AIRFLOW__CORE__SQL_ALCHEMY_CONN`, it will be automatically set by the chart!" nil }}
54+
{{/* defines checks for various environment overrides */}}
55+
{{- define "airflow.environment_check" }}
56+
{{- if .Values.AIRFLOW__CORE__EXECUTOR }}
57+
{{ required (printf "Don't define `AIRFLOW__CORE__EXECUTOR` under %s, it will be automatically set from `airflow.executor`!" .TreeName) nil }}
58+
{{- end }}
59+
{{- if .Values.AIRFLOW__CORE__DAGS_FOLDER }}
60+
{{ required (printf "Don't define `AIRFLOW__CORE__DAGS_FOLDER` under %s, it will be automatically set from `dags.path`!" .TreeName) nil }}
61+
{{- end }}
62+
{{- if or (.Values.AIRFLOW__CELERY__BROKER_URL) (.Values.AIRFLOW__CELERY__BROKER_URL_CMD) }}
63+
{{ required (printf "Don't define `AIRFLOW__CELERY__BROKER_URL` under %s, it will be automatically set by the chart!") nil }}
64+
{{- end }}
65+
{{- if or (.Values.AIRFLOW__CELERY__RESULT_BACKEND) (.Values.AIRFLOW__CELERY__RESULT_BACKEND_CMD) }}
66+
{{ required (printf "Don't define `AIRFLOW__CELERY__RESULT_BACKEND` under %s, it will be automatically set by the chart!") nil }}
67+
{{- end }}
68+
{{- if or (.Values.AIRFLOW__CORE__SQL_ALCHEMY_CONN) (.Values.AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD) }}
69+
{{ required (printf "Don't define `AIRFLOW__CORE__SQL_ALCHEMY_CONN` under %s, it will be automatically set by the chart!") nil }}
70+
{{- end }}
71+
{{- if or (.Values.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN) (.Values.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD) }}
72+
{{ required (printf "Don't define `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` under %s, it will be automatically set by the chart!") nil }}
73+
{{- end }}
6974
{{- end }}
70-
{{- if or (.Values.airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN) (.Values.airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD) }}
71-
{{ required "Don't define `airflow.config.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`, it will be automatically set by the chart!" nil }}
75+
{{/* Checks for `airflow.config` and the extraEnvs */}}
76+
{{- include "airflow.environment_check" (dict "Values" .Values.airflow.config "TreeName" "`airflow.config`") }}
77+
{{- range $tree := (list "airflow" "flower" "scheduler" "triggerer" "web" "workers") }}
78+
{{- $envDict := (dict) }}
79+
{{- range $env := (index $.Values $tree "extraEnv") }}
80+
{{- $_ := (set $envDict $env.name "1") }}
81+
{{- end }}
82+
{{- include "airflow.environment_check" (dict "Values" $envDict "TreeName" (printf "`%s.extraEnv`" $tree)) }}
7283
{{- end }}
7384

7485
{{/* Checks for `scheduler.logCleanup` */}}
@@ -220,4 +231,4 @@
220231
{{ required "If `externalRedis.host` is set, then `redis.enabled` should be `false`!" nil }}
221232
{{- end }}
222233
{{- end }}
223-
{{- end }}
234+
{{- end }}

‎charts/airflow/templates/flower/flower-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.flower.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.flower.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.flower.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.flower.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.flower.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.flower.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -100,7 +101,7 @@ spec:
100101
envFrom:
101102
{{- include "airflow.envFrom" . | indent 12 }}
102103
env:
103-
{{- include "airflow.env" . | indent 12 }}
104+
{{- $env | indent 12 }}
104105
ports:
105106
- name: flower
106107
containerPort: 5555

‎charts/airflow/templates/scheduler/scheduler-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.scheduler.affinity) }}
33
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.scheduler.tolerations) }}
44
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.scheduler.securityContext) }}
5+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.scheduler.extraEnv) }}
56
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.scheduler.extraPipPackages }}
67
{{- $extraVolumeMounts := .Values.scheduler.extraVolumeMounts }}
78
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -110,7 +111,7 @@ spec:
110111
envFrom:
111112
{{- include "airflow.envFrom" . | indent 12 }}
112113
env:
113-
{{- include "airflow.env" . | indent 12 }}
114+
{{- $env | indent 12 }}
114115
command:
115116
{{- include "airflow.command" . | indent 12 }}
116117
args:

‎charts/airflow/templates/triggerer/triggerer-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.triggerer.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.triggerer.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.triggerer.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.triggerer.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.triggerer.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.triggerer.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -99,7 +100,7 @@ spec:
99100
envFrom:
100101
{{- include "airflow.envFrom" . | indent 12 }}
101102
env:
102-
{{- include "airflow.env" . | indent 12 }}
103+
{{- $env | indent 12 }}
103104
command:
104105
{{- include "airflow.command" . | indent 12 }}
105106
args:

‎charts/airflow/templates/webserver/webserver-deployment.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.web.affinity) }}
33
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.web.tolerations) }}
44
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.web.securityContext) }}
5+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.web.extraEnv) }}
56
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.web.extraPipPackages }}
67
{{- $extraVolumeMounts := .Values.web.extraVolumeMounts }}
78
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -105,7 +106,7 @@ spec:
105106
envFrom:
106107
{{- include "airflow.envFrom" . | indent 12 }}
107108
env:
108-
{{- include "airflow.env" . | indent 12 }}
109+
{{- $env | indent 12 }}
109110
command:
110111
{{- include "airflow.command" . | indent 12 }}
111112
args:

‎charts/airflow/templates/worker/worker-statefulset.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.workers.affinity) }}
44
{{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.workers.tolerations) }}
55
{{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.workers.securityContext) }}
6+
{{- $env := include "airflow.env" (dict "Release" .Release "Values" .Values "extraEnv" .Values.workers.extraEnv) }}
67
{{- $extraPipPackages := concat .Values.airflow.extraPipPackages .Values.workers.extraPipPackages }}
78
{{- $extraVolumeMounts := .Values.workers.extraVolumeMounts }}
89
{{- $volumeMounts := include "airflow.volumeMounts" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages "extraVolumeMounts" $extraVolumeMounts) }}
@@ -104,7 +105,7 @@ spec:
104105
envFrom:
105106
{{- include "airflow.envFrom" . | indent 12 }}
106107
env:
107-
{{- include "airflow.env" . | indent 12 }}
108+
{{- $env | indent 12 }}
108109
# have dumb-init only send signals to direct child process (needed for celery workers to warm shutdown)
109110
- name: DUMB_INIT_SETSID
110111
value: "0"

‎charts/airflow/values.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -643,6 +643,9 @@ scheduler:
643643
##
644644
schedulerAgeBeforeCheck: 180
645645

646+
## extra environment for just the scheduler Pods
647+
extraEnv: []
648+
646649
## extra pip packages to install in the scheduler Pods
647650
##
648651
## ____ EXAMPLE _______________
@@ -811,6 +814,9 @@ web:
811814
timeoutSeconds: 5
812815
failureThreshold: 6
813816

817+
## extra environment for just the web Pods
818+
extraEnv: []
819+
814820
## extra pip packages to install in the web Pods
815821
##
816822
## ____ EXAMPLE _______________
@@ -992,6 +998,9 @@ workers:
992998
##
993999
intervalSeconds: 900
9941000

1001+
## extra environment for just the worker Pods
1002+
extraEnv: []
1003+
9951004
## extra pip packages to install in the worker Pod
9961005
##
9971006
## ____ EXAMPLE _______________
@@ -1111,6 +1120,9 @@ triggerer:
11111120
timeoutSeconds: 60
11121121
failureThreshold: 5
11131122

1123+
## extra environment for just the triggerer Pods
1124+
extraEnv: []
1125+
11141126
## extra pip packages to install in the triggerer Pod
11151127
##
11161128
## ____ EXAMPLE _______________
@@ -1253,6 +1265,9 @@ flower:
12531265
timeoutSeconds: 5
12541266
failureThreshold: 6
12551267

1268+
## extra environment for just the flower Pods
1269+
extraEnv: []
1270+
12561271
## extra pip packages to install in the flower Pod
12571272
##
12581273
## ____ EXAMPLE _______________

0 commit comments

Comments
 (0)