Skip to content

Commit 236d5e6

Browse files
shakti-garg-saxoshakti-garg
shakti-garg-saxo
and
shakti-garg
authored
feat(kubernetes): Improve the security of the kubernetes/helm charts (#1782)
* 1747 | remove obsolete yaml files * 1747 | remove configmap and its hardcoded references * 1747 | add missing input parameter of neo4j.host * 1747 | remove obsolete secrets and parameterize the rest * 1747 | auto-generate gms secret * 1747 | remove fullName overrides * 1747 | fix parameters in subchart's values.yaml * 1747 | remove hardcoding from parameters for gms host and port * 1747 | upgrade chart version * 1747 | update helm docs * 1747 | add extraEnv, extraVolume and extraMounts * 1747 | Alters pull policy of images to 'always' for ldh Co-authored-by: shakti-garg <[email protected]>
1 parent ece9b82 commit 236d5e6

30 files changed

+252
-502
lines changed

‎contrib/kubernetes/README.md

Lines changed: 4 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -31,48 +31,10 @@ The following table lists the configuration parameters and its default values
3131

3232
| Repository | Name | Version |
3333
|------------|------|---------|
34-
| file://./charts/datahub-frontend | datahub-frontend | 0.1.0 |
35-
| file://./charts/datahub-gms | datahub-gms | 0.1.0 |
36-
| file://./charts/datahub-mae-consumer | datahub-mae-consumer | 0.1.0 |
37-
| file://./charts/datahub-mce-consumer | datahub-mce-consumer | 0.1.0 |
38-
39-
#### Chart Values
40-
41-
| Key | Type | Default | Description |
42-
|-----|------|---------|-------------|
43-
| datahub-frontend.enabled | bool | `true` | |
44-
| datahub-frontend.image.repository | string | `"linkedin/datahub-frontend"` | |
45-
| datahub-frontend.image.tag | string | `"latest"` | |
46-
| datahub-gms.enabled | bool | `true` | |
47-
| datahub-gms.image.repository | string | `"linkedin/datahub-gms"` | |
48-
| datahub-gms.image.tag | string | `"latest"` | |
49-
| datahub-mae-consumer.enabled | bool | `true` | |
50-
| datahub-mae-consumer.image.repository | string | `"linkedin/datahub-mae-consumer"` | |
51-
| datahub-mae-consumer.image.tag | string | `"latest"` | |
52-
| datahub-mce-consumer.enabled | bool | `true` | |
53-
| datahub-mce-consumer.image.repository | string | `"linkedin/datahub-mce-consumer"` | |
54-
| datahub-mce-consumer.image.tag | string | `"latest"` | |
55-
| global.datahub.appVersion | string | `"1.0"` | |
56-
| global.datahub.gms.host | string | `"datahub-gms-deployment"` | |
57-
| global.datahub.gms.port | string | `"8080"` | |
58-
| global.datahub.gms.secret | string | `"YouKnowNothing"` | |
59-
| global.elasticsearch.host | string | `"elasticsearch"` | |
60-
| global.elasticsearch.port | string | `"9200"` | |
61-
| global.hostAliases[0].hostnames[0] | string | `"broker"` | |
62-
| global.hostAliases[0].hostnames[1] | string | `"mysql"` | |
63-
| global.hostAliases[0].hostnames[2] | string | `"elasticsearch"` | |
64-
| global.hostAliases[0].hostnames[3] | string | `"neo4j"` | |
65-
| global.hostAliases[0].ip | string | `"192.168.0.104"` | |
66-
| global.kafka.bootstrap.server | string | `"broker:29092"` | |
67-
| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | |
68-
| global.neo4j.password | string | `"datahub"` | |
69-
| global.neo4j.uri | string | `"bolt://neo4j"` | |
70-
| global.neo4j.username | string | `"neo4j"` | |
71-
| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | |
72-
| global.sql.datasource.host | string | `"mysql"` | |
73-
| global.sql.datasource.password | string | `"datahub"` | |
74-
| global.sql.datasource.url | string | `"jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | |
75-
| global.sql.datasource.username | string | `"datahub"` | |
34+
| file://./charts/datahub-frontend | datahub-frontend | 0.2.0 |
35+
| file://./charts/datahub-gms | datahub-gms | 0.2.0 |
36+
| file://./charts/datahub-mae-consumer | datahub-mae-consumer | 0.2.0 |
37+
| file://./charts/datahub-mce-consumer | datahub-mce-consumer | 0.2.0 |
7638

7739
## Install DataHub
7840
Navigate to the current directory and run the below command. Update the `datahub/values.yaml` file with valid hostname/IP address configuration for elasticsearch, neo4j, schema-registry, broker & mysql.

‎contrib/kubernetes/datahub/Chart.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,24 @@ description: A Helm chart for LinkedIn DataHub
44
type: application
55
# This is the chart version. This version number should be incremented each time you make changes
66
# to the chart and its templates, including the app version.
7-
version: 0.0.1
7+
version: 0.1.0
88
# This is the version number of the application being deployed. This version number should be
99
# incremented each time you make changes to the application.
1010
appVersion: latest #0.3.1
1111
dependencies:
1212
- name: datahub-gms
13-
version: 0.1.0
13+
version: 0.2.0
1414
repository: file://./charts/datahub-gms
1515
condition: datahub-gms.enabled
1616
- name: datahub-frontend
17-
version: 0.1.0
17+
version: 0.2.0
1818
repository: file://./charts/datahub-frontend
1919
condition: datahub-frontend.enabled
2020
- name: datahub-mae-consumer
21-
version: 0.1.0
21+
version: 0.2.0
2222
repository: file://./charts/datahub-mae-consumer
2323
condition: datahub-mae-consumer.enabled
2424
- name: datahub-mce-consumer
25-
version: 0.1.0
25+
version: 0.2.0
2626
repository: file://./charts/datahub-mce-consumer
2727
condition: datahub-mce-consumer.enabled

‎contrib/kubernetes/datahub/README.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ datahub
22
=======
33
A Helm chart for LinkedIn DataHub
44

5-
Current chart version is `0.0.1`
5+
Current chart version is `0.1.0`
66

77
## Chart Requirements
88

@@ -13,7 +13,7 @@ Current chart version is `0.0.1`
1313
| file://./charts/datahub-mae-consumer | datahub-mae-consumer | 0.1.0 |
1414
| file://./charts/datahub-mce-consumer | datahub-mce-consumer | 0.1.0 |
1515

16-
## Chart Values
16+
#### Chart Values
1717

1818
| Key | Type | Default | Description |
1919
|-----|------|---------|-------------|
@@ -30,9 +30,7 @@ Current chart version is `0.0.1`
3030
| datahub-mce-consumer.image.repository | string | `"linkedin/datahub-mce-consumer"` | |
3131
| datahub-mce-consumer.image.tag | string | `"latest"` | |
3232
| global.datahub.appVersion | string | `"1.0"` | |
33-
| global.datahub.gms.host | string | `"datahub-gms-deployment"` | |
3433
| global.datahub.gms.port | string | `"8080"` | |
35-
| global.datahub.gms.secret | string | `"YouKnowNothing"` | |
3634
| global.elasticsearch.host | string | `"elasticsearch"` | |
3735
| global.elasticsearch.port | string | `"9200"` | |
3836
| global.hostAliases[0].hostnames[0] | string | `"broker"` | |
@@ -42,11 +40,14 @@ Current chart version is `0.0.1`
4240
| global.hostAliases[0].ip | string | `"192.168.0.104"` | |
4341
| global.kafka.bootstrap.server | string | `"broker:29092"` | |
4442
| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | |
45-
| global.neo4j.password | string | `"datahub"` | |
43+
| global.neo4j.host | string | `"neo4j:7474"` | |
4644
| global.neo4j.uri | string | `"bolt://neo4j"` | |
4745
| global.neo4j.username | string | `"neo4j"` | |
46+
| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | |
47+
| global.neo4j.password.secretKey | string | `"neo4j-password"` | |
4848
| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | |
4949
| global.sql.datasource.host | string | `"mysql"` | |
50-
| global.sql.datasource.password | string | `"datahub"` | |
5150
| global.sql.datasource.url | string | `"jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | |
5251
| global.sql.datasource.username | string | `"datahub"` | |
52+
| global.sql.datasource.password.secretRef | string | `"mysql-secrets"` | |
53+
| global.sql.datasource.password.secretKey | string | `"mysql-password"` | |

‎contrib/kubernetes/datahub/charts/datahub-frontend/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ type: application
1414

1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
17-
version: 0.1.0
17+
version: 0.2.0
1818

1919
# This is the version number of the application being deployed. This version number should be
2020
# incremented each time you make changes to the application.

‎contrib/kubernetes/datahub/charts/datahub-frontend/README.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,19 @@ datahub-frontend
22
================
33
A Helm chart for datahub-frontend
44

5-
Current chart version is `0.1.0`
5+
Current chart version is `0.2.0`
66

77
## Chart Values
88

99
| Key | Type | Default | Description |
1010
|-----|------|---------|-------------|
1111
| affinity | object | `{}` | |
1212
| datahub.play.mem.buffer.size | string | `"10MB"` | |
13+
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
14+
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
15+
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
1316
| fullnameOverride | string | `"datahub-frontend"` | |
14-
| global.datahub.gms.host | string | `"datahub-gms-deployment"` | |
1517
| global.datahub.gms.port | string | `"8080"` | |
16-
| global.datahub.gms.secret | string | `"YouKnowNothing"` | |
1718
| image.pullPolicy | string | `"IfNotPresent"` | |
1819
| image.repository | string | `"linkedin/datahub-frontend"` | |
1920
| image.tag | string | `"latest"` | |

‎contrib/kubernetes/datahub/charts/datahub-frontend/templates/deployment.yaml

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ spec:
2121
serviceAccountName: {{ include "datahub-frontend.serviceAccountName" . }}
2222
securityContext:
2323
{{- toYaml .Values.podSecurityContext | nindent 8 }}
24+
volumes:
25+
{{- if .Values.extraVolumes }}
26+
{{ toYaml .Values.extraVolumes | indent 8 }}
27+
{{- end }}
2428
containers:
2529
- name: {{ .Chart.Name }}
2630
securityContext:
@@ -41,18 +45,25 @@ spec:
4145
port: http
4246
env:
4347
- name: DATAHUB_GMS_HOST
44-
value: "{{ .Values.global.datahub.gms.host }}"
48+
value: {{ printf "%s-%s" .Release.Name "datahub-gms" }}
4549
- name: DATAHUB_GMS_PORT
4650
value: "{{ .Values.global.datahub.gms.port }}"
4751
- name: DATAHUB_SECRET
4852
valueFrom:
4953
secretKeyRef:
50-
name: {{ include "datahub-frontend.fullname" . }}-secret
54+
name: {{ printf "%s-gms-secret" .Release.Name }}
5155
key: datahub.gms.secret
5256
- name: DATAHUB_APP_VERSION
5357
value: "{{ .Values.global.datahub.appVersion }}"
5458
- name: DATAHUB_PLAY_MEM_BUFFER_SIZE
5559
value: "{{ .Values.datahub.play.mem.buffer.size }}"
60+
{{- if .Values.extraEnvs }}
61+
{{ toYaml .Values.extraEnvs | indent 10 }}
62+
{{- end }}
63+
volumeMounts:
64+
{{- if .Values.extraVolumeMounts }}
65+
{{ toYaml .Values.extraVolumeMounts | indent 10 }}
66+
{{- end }}
5667
resources:
5768
{{- toYaml .Values.resources | nindent 12 }}
5869
{{- with .Values.nodeSelector }}

‎contrib/kubernetes/datahub/charts/datahub-frontend/templates/secrets.yaml

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎contrib/kubernetes/datahub/charts/datahub-frontend/values.yaml

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,11 @@ replicaCount: 1
77
image:
88
repository: linkedin/datahub-frontend
99
tag: "latest"
10-
pullPolicy: IfNotPresent
10+
pullPolicy: Always
1111

1212
imagePullSecrets: []
1313
nameOverride: ""
14-
fullnameOverride: "datahub-frontend"
14+
fullnameOverride: ""
1515

1616
serviceAccount:
1717
# Specifies whether a service account should be created
@@ -50,6 +50,22 @@ ingress:
5050
# hosts:
5151
# - chart-example.local
5252

53+
# Extra environment variables
54+
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
55+
# syntax here
56+
extraEnvs: []
57+
# - name: MY_ENVIRONMENT_VAR
58+
# value: the_value_goes_here
59+
60+
extraVolumes: []
61+
# - name: extras
62+
# emptyDir: {}
63+
64+
extraVolumeMounts: []
65+
# - name: extras
66+
# mountPath: /usr/share/extras
67+
# readOnly: true
68+
5369
resources: {}
5470
# We usually recommend not to specify default resources and to leave this as a conscious
5571
# choice for the user. This also increases chances charts run on environments with little
@@ -76,6 +92,5 @@ datahub:
7692
global:
7793
datahub:
7894
gms:
79-
host: "datahub-gms-deployment"
8095
port: "8080"
81-
secret: "YouKnowNothing"
96+
appVersion: "1.0"

‎contrib/kubernetes/datahub/charts/datahub-gms/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ type: application
1414

1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
17-
version: 0.1.0
17+
version: 0.2.0
1818

1919
# This is the version number of the application being deployed. This version number should be
2020
# incremented each time you make changes to the application.

‎contrib/kubernetes/datahub/charts/datahub-gms/README.md

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,32 +2,39 @@ datahub-gms
22
===========
33
A Helm chart for LinkedIn DataHub's datahub-gms component
44

5-
Current chart version is `0.1.0`
5+
Current chart version is `0.2.0`
66

77
## Chart Values
88

99
| Key | Type | Default | Description |
1010
|-----|------|---------|-------------|
1111
| affinity | object | `{}` | |
12+
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
13+
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
14+
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
1215
| fullnameOverride | string | `"datahub-gms-deployment"` | |
1316
| global.datahub.appVersion | string | `"1.0"` | |
14-
| global.datahub.gms.host | string | `"datahub-gms-service"` | |
1517
| global.datahub.gms.port | string | `"8080"` | |
16-
| global.datahub.gms.secret | string | `"YouKnowNothing"` | |
17-
| global.elasticsearch.host | string | `"192.168.0.104"` | |
18+
| global.elasticsearch.host | string | `"elasticsearch"` | |
1819
| global.elasticsearch.port | string | `"9200"` | |
1920
| global.hostAliases[0].hostnames[0] | string | `"broker"` | |
21+
| global.hostAliases[0].hostnames[1] | string | `"mysql"` | |
22+
| global.hostAliases[0].hostnames[2] | string | `"elasticsearch"` | |
23+
| global.hostAliases[0].hostnames[3] | string | `"neo4j"` | |
2024
| global.hostAliases[0].ip | string | `"192.168.0.104"` | |
21-
| global.kafka.bootstrap.server | string | `"192.168.0.104:29092"` | |
22-
| global.kafka.schemaregistry.url | string | `"http://192.168.0.104:8081"` | |
23-
| global.neo4j.password | string | `"datahub"` | |
24-
| global.neo4j.uri | string | `"bolt://192.168.0.104"` | |
25+
| global.kafka.bootstrap.server | string | `"broker:29092"` | |
26+
| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | |
27+
| global.neo4j.host | string | `"neo4j:7474"` | |
28+
| global.neo4j.uri | string | `"bolt://neo4j"` | |
2529
| global.neo4j.username | string | `"neo4j"` | |
30+
| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | |
31+
| global.neo4j.password.secretKey | string | `"neo4j-password"` | |
2632
| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | |
27-
| global.sql.datasource.host | string | `"192.168.0.104:3306"` | |
28-
| global.sql.datasource.password | string | `"datahub"` | |
29-
| global.sql.datasource.url | string | `"jdbc:mysql://192.168.0.104:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | |
33+
| global.sql.datasource.host | string | `"mysql"` | |
34+
| global.sql.datasource.url | string | `"jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | |
3035
| global.sql.datasource.username | string | `"datahub"` | |
36+
| global.sql.datasource.password.secretRef | string | `"mysql-secrets"` | |
37+
| global.sql.datasource.password.secretKey | string | `"mysql-password"` | |
3138
| image.pullPolicy | string | `"IfNotPresent"` | |
3239
| image.repository | string | `"linkedin/datahub-gms"` | |
3340
| image.tag | string | `"latest"` | |

0 commit comments

Comments
 (0)