There are 2 different Network Policy options :
- Calico
- Cilium
Before enabling Calico policy for the AKS-Engine cluster, you must first decide which network plugin to use for the cluster: Azure or Kubenet. The difference between Azure and Kubenet lies in the way IP addresses get allocated; Azure uses Azure-native IPs while Kubenet does static IP assignment based on the node's pod CIDR. If you're not sure which one to go with, we recommend going with Azure.
The kubernetes-calico deployment template enables Calico networking and policies for the AKS-Engine cluster via "networkPolicy": "calico" being present inside the kubernetesConfig.
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"kubernetesConfig": {
"networkPolicy": "calico",
"networkPlugin": "azure|kubenet"
}This template will deploy the Kubernetes Datastore backed version of Calico with user-supplied networking which supports kubernetes ingress policies.
If deploying on a K8s 1.8 or later cluster, then egress policies are also supported!
To understand how to deploy this template, please read the baseline Kubernetes document, and use the appropriate kubernetes-calico-[azure|kubenet].json example file in this folder as an api model reference.
Once the template has been successfully deployed, following the simple policy tutorial or the advanced policy tutorial will help to understand calico networking.
Note:
ping(ICMP) traffic is blocked on the cluster by default. Whereverpingis used in any tutorial substitute testing access with something likewget -q --timeout=5 google.com -O -instead.
Clusters deployed with calico networkPolicy enabled prior to 0.17.0 had calico 2.6.3 deployed, and a daemonset with an updateStrategy of Ondelete.
AKS-Engine releases starting with 0.17.0 now produce an addon manifest for calico in /etc/kubernetes/addons/calico-daemonset.yaml contaning calico 3.1.x, and an updateStrategy of RollingUpdate. Due to breaking changes introduced by calico 3, one must first migrate through calico 2.6.5 or a later 2.6.x release in order to migrate to calico 3.1.x. as described in the calico kubernetes upgrade documentation. The AKS-Engine manifest for calico uses the kubernetes API datastore, policy-only setup.
-
To update to
2.6.5+in preparation of an upgrade to 3.1.x as specified, edit/etc/kubernetes/addons/calico-daemonset.yamlon a master node, replacingcalico/node:v3.1.1withcalico/node:v2.6.10andcalico/cni:v3.1.1withcalico/cni:v2.0.6. Runkubectl apply -f /etc/kubernetes/addons/calico-daemonset.yaml.Wait until all the pods in the daemonset get rotated and come up up-to-date, healthy and ready:
YYYY-MM-DD HH:MM:SS.FFF [INFO][n] health.go 150: Overall health summary=&health.HealthReport{Live:true, Ready:true} -
To complete the upgrade to 3.1.x, edit
/etc/kubernetes/addons/calico-daemonset.yamlon the master node again, replacingcalico/node:v2.6.10withcalico/node:v3.1.1andcalico/cni:v2.0.6withcalico/cni:v3.1.1. Runkubectl apply -f /etc/kubernetes/addons/calico-daemonset.yaml.Propagate this updated manifest to all master nodes in the cluster.
-
Confirm that all the pods in the daemonset get rotated and come up healthy after finishing tasks logged by migrate.go:
YYYY-MM-DD HH:MM:SS.FFF [INFO][n] startup.go 1044: Running migration YYYY-MM-DD HH:MM:SS.FFF [INFO][n] migrate.go 842: Querying current v1 snapshot and converting to v3 [xxx] YYYY-MM-DD HH:MM:SS.FFF [INFO][n] migrate.go 851: continue by upgrading your calico/node versions to Calico v3.1.x YYYY-MM-DD HH:MM:SS.FFF [INFO][n] startup.go 1048: Migration successful
If you have any customized calico resource manifests, you must also follow the conversion guide for these.
The kubernetes-cilium deployment template enables Cilium networking and policies for the AKS-Engine cluster via "networkPolicy": "cilium" or "networkPlugin": "cilium" being present inside the kubernetesConfig.
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"kubernetesConfig": {
"networkPolicy": "cilium"
}Note: To execute the
ciliumcommand that is running inside of the pods, you will need remove theDenyEscalatingExecwhen specifying the Admission Control Values. If running Kubernetes with theorchestratorReleasenewer than 1.9 use--enable-admission-pluginsinstead of--admission-controlas illustrated below:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.10",
"kubernetesConfig": {
"networkPlugin": "cilium",
"networkPolicy": "cilium",
"apiServerConfig": {
"--enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,AlwaysPullImages"
},Once the template has been successfully deployed, following the deploy the demo application tutorial will provide a good foundation for how to do L3/4 policy as well as more advanced Layer 7 inspection and routing. If you have Istio you can try this tutorial where cilium is used to side by side with Istio to enforce security policies in a Kubernetes deployment.
For the latest documentation on Cilium (including BPF and XDP reference guides), please refer to this