Skip to content
This repository was archived by the owner on Jan 11, 2023. It is now read-only.

Commit b473029

Browse files
ritazhjackfrancis
ritazh
authored and
jackfrancis
committed
Add keyvault-flexvolume addon (#3498)
1 parent 6cd2a0f commit b473029

File tree

18 files changed

+340
-0
lines changed

18 files changed

+340
-0
lines changed

‎docs/clusterdefinition.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ Here are the valid values for the orchestrator types:
7373
| [cluster-autoscaler](../examples/addons/cluster-autoscaler/README.md) | false | 1 | Delivers the Kubernetes cluster autoscaler component. See https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/azure for more info |
7474
| [nvidia-device-plugin](../examples/addons/nvidia-device-plugin/README.md) | true if using a Kubernetes cluster (v1.10+) with an N-series agent pool | 1 | Delivers the Kubernetes NVIDIA device plugin component. See https://github.com/NVIDIA/k8s-device-plugin for more info |
7575
| container-monitoring | false | 1 | Delivers the Kubernetes container monitoring component |
76+
| [keyvault-flexvolume](../examples/addons/keyvault-flexvolume/README.md) | false | as many as linux agent nodes | Access secrets, keys, and certs in Azure Key Vault from pods |
7677

7778
To give a bit more info on the `addons` property: We've tried to expose the basic bits of data that allow useful configuration of these cluster features. Here are some example usage patterns that will unpack what `addons` provide:
7879

‎examples/addons/keyvault-flexvolume/README.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Azure Key Vault FlexVolume Add-on
2+
3+
[The Azure Key Vault FlexVolume](https://github.com/Azure/kubernetes-keyvault-flexvol) integrates Azure Key Vault with Kubernetes via a FlexVolume.
4+
5+
With the Azure Key Vault FlexVolume, developers can access application-specific secrets, keys, and certs stored in Azure Key Vault directly from their pods.
6+
7+
Add this add-on to your apimodel as shown below to automatically enable Key Vault FlexVolume in your new Kubernetes cluster.
8+
9+
```json
10+
{
11+
"apiVersion": "vlabs",
12+
"properties": {
13+
"orchestratorProfile": {
14+
"orchestratorType": "Kubernetes",
15+
"kubernetesConfig": {
16+
"addons": [
17+
{
18+
"name": "keyvault-flexvolume",
19+
"enabled" : true
20+
}
21+
]
22+
}
23+
},
24+
"masterProfile": {
25+
"count": 1,
26+
"dnsPrefix": "",
27+
"vmSize": "Standard_DS2_v2",
28+
},
29+
"agentPoolProfiles": [
30+
{
31+
"name": "agentpool",
32+
"count": 3,
33+
"vmSize": "Standard_DS2_v2",
34+
"availabilityProfile": "VirtualMachineScaleSets"
35+
}
36+
],
37+
"linuxProfile": {
38+
"adminUsername": "azureuser",
39+
"ssh": {
40+
"publicKeys": [
41+
{
42+
"keyData": ""
43+
}
44+
]
45+
}
46+
},
47+
"servicePrincipalProfile": {
48+
"clientId": "",
49+
"secret": ""
50+
}
51+
}
52+
}
53+
54+
```
55+
56+
To validate the add-on is running as expected, run the following commands:
57+
58+
You should see the keyvault flexvolume installer pods running on each agent node:
59+
60+
```bash
61+
kubectl get pods -n kv
62+
63+
keyvault-flexvolume-f7bx8 1/1 Running 0 3m
64+
keyvault-flexvolume-rcxbl 1/1 Running 0 3m
65+
keyvault-flexvolume-z6jm6 1/1 Running 0 3m
66+
```
67+
68+
Follow the README at https://github.com/Azure/kubernetes-keyvault-flexvol for get started steps.
69+
70+
##
71+
To update resources:
72+
73+
```json
74+
"kubernetesConfig": {
75+
"addons": [
76+
{
77+
"name": "keyvault-flexvolume",
78+
"enabled": true,
79+
"containers": [
80+
{
81+
"name": "keyvault-flexvolume",
82+
"image": "ritazh/kv-flexvol-installer:v0.0.3",
83+
"cpuRequests": "100m",
84+
"memoryRequests": "300Mi",
85+
"cpuLimits": "100m",
86+
"memoryLimits": "300Mi"
87+
}
88+
]
89+
}
90+
]
91+
}
92+
```
93+
94+
## Supported Orchestrators
95+
96+
Kubernetes

‎examples/addons/keyvault-flexvolume/kubernetes-keyvault-flexvolume.json‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"apiVersion": "vlabs",
3+
"properties": {
4+
"orchestratorProfile": {
5+
"orchestratorType": "Kubernetes",
6+
"kubernetesConfig": {
7+
"addons": [
8+
{
9+
"name": "keyvault-flexvolume",
10+
"enabled": true
11+
}
12+
]
13+
}
14+
},
15+
"masterProfile": {
16+
"count": 1,
17+
"dnsPrefix": "",
18+
"vmSize": "Standard_DS2_v2"
19+
},
20+
"agentPoolProfiles": [
21+
{
22+
"name": "agentpool",
23+
"count": 3,
24+
"vmSize": "Standard_DS2_v2",
25+
"availabilityProfile": "VirtualMachineScaleSets"
26+
}
27+
],
28+
"linuxProfile": {
29+
"adminUsername": "azureuser",
30+
"ssh": {
31+
"publicKeys": [
32+
{
33+
"keyData": ""
34+
}
35+
]
36+
}
37+
},
38+
"servicePrincipalProfile": {
39+
"clientId": "",
40+
"secret": ""
41+
}
42+
}
43+
}

‎examples/e2e-tests/kubernetes/kubernetes-config/addons-disabled.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@
1313
"name": "aci-connector",
1414
"enabled": false
1515
},
16+
{
17+
"name": "keyvault-flexvolume",
18+
"enabled": false
19+
},
1620
{
1721
"name": "kubernetes-dashboard",
1822
"enabled": false

‎examples/e2e-tests/kubernetes/kubernetes-config/addons-enabled.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@
1414
"name": "aci-connector",
1515
"enabled": true
1616
},
17+
{
18+
"name": "keyvault-flexvolume",
19+
"enabled": true
20+
},
1721
{
1822
"name": "kubernetes-dashboard",
1923
"enabled": true

‎parts/k8s/addons/kubernetesmasteraddons-keyvault-flexvolume-installer.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
labels:
5+
kubernetes.io/cluster-service: "true"
6+
addonmanager.kubernetes.io/mode: "EnsureExists"
7+
name: kv
8+
---
9+
apiVersion: extensions/v1beta1
10+
kind: DaemonSet
11+
metadata:
12+
labels:
13+
app: keyvault-flexvolume
14+
kubernetes.io/cluster-service: "true"
15+
addonmanager.kubernetes.io/mode: "EnsureExists"
16+
name: keyvault-flexvolume
17+
namespace: kv
18+
spec:
19+
template:
20+
metadata:
21+
labels:
22+
app: keyvault-flexvolume
23+
kubernetes.io/cluster-service: "true"
24+
addonmanager.kubernetes.io/mode: "EnsureExists"
25+
spec:
26+
tolerations:
27+
containers:
28+
- name: keyvault-flexvolume
29+
image: "ritazh/kv-flexvol-installer:v0.0.3"
30+
imagePullPolicy: Always
31+
resources:
32+
requests:
33+
cpu: <kubernetesKeyVaultFlexVolumeInstallerCPURequests>
34+
memory: <kubernetesKeyVaultFlexVolumeInstallerMemoryRequests>
35+
limits:
36+
cpu: <kubernetesKeyVaultFlexVolumeInstallerCPULimit>
37+
memory: <kubernetesKeyVaultFlexVolumeInstallerMemoryLimit>
38+
env:
39+
- name: TARGET_DIR
40+
value: "/etc/kubernetes/volumeplugins"
41+
volumeMounts:
42+
- mountPath: "/etc/kubernetes/volumeplugins"
43+
name: volplugins
44+
volumes:
45+
- hostPath:
46+
path: "/etc/kubernetes/volumeplugins"
47+
name: volplugins
48+
nodeSelector:
49+
beta.kubernetes.io/os: linux

‎parts/k8s/kubernetesmastercustomdata.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,14 @@ MASTER_ARTIFACTS_CONFIG_PLACEHOLDER
240240
sed -i "s|<kubernetesClusterAutoscalerUseManagedIdentity>|{{WrapAsVariable "kubernetesClusterAutoscalerUseManagedIdentity"}}|g" "/etc/kubernetes/addons/cluster-autoscaler-deployment.yaml"
241241
{{end}}
242242

243+
{{if .OrchestratorProfile.KubernetesConfig.IsKeyVaultFlexVolumeEnabled}}
244+
sed -i "s|<kubernetesKeyVaultFlexVolumeInstallerCPURequests>|{{WrapAsVariable "kubernetesKeyVaultFlexVolumeInstallerCPURequests"}}|g" "/etc/kubernetes/addons/keyvault-flexvolume-installer.yaml"
245+
sed -i "s|<kubernetesKeyVaultFlexVolumeInstallerMemoryRequests>|{{WrapAsVariable "kubernetesKeyVaultFlexVolumeInstallerMemoryRequests"}}|g" "/etc/kubernetes/addons/keyvault-flexvolume-installer.yaml"
246+
sed -i "s|<kubernetesKeyVaultFlexVolumeInstallerCPULimit>|{{WrapAsVariable "kubernetesKeyVaultFlexVolumeInstallerCPULimit"}}|g" "/etc/kubernetes/addons/keyvault-flexvolume-installer.yaml"
247+
sed -i "s|<kubernetesKeyVaultFlexVolumeInstallerMemoryLimit>|{{WrapAsVariable "kubernetesKeyVaultFlexVolumeInstallerMemoryLimit"}}|g" "/etc/kubernetes/addons/keyvault-flexvolume-installer.yaml"
248+
249+
250+
{{end}}
243251
{{if .OrchestratorProfile.KubernetesConfig.IsReschedulerEnabled}}
244252
sed -i "s|<kubernetesReschedulerSpec>|{{WrapAsVariable "kubernetesReschedulerSpec"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
245253
sed -i "s|<kubernetesReschedulerCPURequests>|{{WrapAsVariable "kubernetesReschedulerCPURequests"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"

‎parts/k8s/kubernetesmastervars.t‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,10 @@
127127
"kubernetesClusterAutoscalerMaxNodes": "[parameters('kubernetesClusterAutoscalerMaxNodes')]",
128128
"kubernetesClusterAutoscalerEnabled": "[parameters('kubernetesClusterAutoscalerEnabled')]",
129129
"kubernetesClusterAutoscalerUseManagedIdentity": "[parameters('kubernetesClusterAutoscalerUseManagedIdentity')]",
130+
"kubernetesKeyVaultFlexVolumeInstallerCPURequests": "[parameters('kubernetesKeyVaultFlexVolumeInstallerCPURequests')]",
131+
"kubernetesKeyVaultFlexVolumeInstallerMemoryRequests": "[parameters('kubernetesKeyVaultFlexVolumeInstallerMemoryRequests')]",
132+
"kubernetesKeyVaultFlexVolumeInstallerCPULimit": "[parameters('kubernetesKeyVaultFlexVolumeInstallerCPULimit')]",
133+
"kubernetesKeyVaultFlexVolumeInstallerMemoryLimit": "[parameters('kubernetesKeyVaultFlexVolumeInstallerMemoryLimit')]",
130134
"kubernetesReschedulerSpec": "[parameters('kubernetesReschedulerSpec')]",
131135
"kubernetesReschedulerCPURequests": "[parameters('kubernetesReschedulerCPURequests')]",
132136
"kubernetesReschedulerMemoryRequests": "[parameters('kubernetesReschedulerMemoryRequests')]",

‎parts/k8s/kubernetesparams.t‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -521,6 +521,34 @@
521521
},
522522
"type": "string"
523523
},
524+
"kubernetesKeyVaultFlexVolumeInstallerCPURequests": {
525+
{{PopulateClassicModeDefaultValue "kubernetesKeyVaultFlexVolumeInstallerCPURequests"}}
526+
"metadata": {
527+
"description": "Key Vault FlexVolume Installer CPU Requests"
528+
},
529+
"type": "string"
530+
},
531+
"kubernetesKeyVaultFlexVolumeInstallerMemoryRequests": {
532+
{{PopulateClassicModeDefaultValue "kubernetesKeyVaultFlexVolumeInstallerMemoryRequests"}}
533+
"metadata": {
534+
"description": "Key Vault FlexVolume Installer Memory Requests"
535+
},
536+
"type": "string"
537+
},
538+
"kubernetesKeyVaultFlexVolumeInstallerCPULimit": {
539+
{{PopulateClassicModeDefaultValue "kubernetesKeyVaultFlexVolumeInstallerCPULimit"}}
540+
"metadata": {
541+
"description": "Key Vault FlexVolume Installer CPU Limit"
542+
},
543+
"type": "string"
544+
},
545+
"kubernetesKeyVaultFlexVolumeInstallerMemoryLimit": {
546+
{{PopulateClassicModeDefaultValue "kubernetesKeyVaultFlexVolumeInstallerMemoryLimit"}}
547+
"metadata": {
548+
"description": "Key Vault FlexVolume Installer Memory Limit"
549+
},
550+
"type": "string"
551+
},
524552
"kubernetesReschedulerSpec": {
525553
{{PopulateClassicModeDefaultValue "kubernetesReschedulerSpec"}}
526554
"metadata": {

‎pkg/acsengine/addons.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,11 @@ func kubernetesAddonSettingsInit(profile *api.Properties) []kubernetesFeatureSet
122122
"audit-policy.yaml",
123123
common.IsKubernetesVersionGe(profile.OrchestratorProfile.OrchestratorVersion, "1.8.0"),
124124
},
125+
{
126+
"kubernetesmasteraddons-keyvault-flexvolume-installer.yaml",
127+
"keyvault-flexvolume-installer.yaml",
128+
profile.OrchestratorProfile.KubernetesConfig.IsKeyVaultFlexVolumeEnabled(),
129+
},
125130
}
126131
}
127132

0 commit comments

Comments
 (0)