added the documentation for workflow with Windows prefix delegation (#263)

Prefix delegation for Windows nodes is a new feature wherein instead of allocating secondary IPv4 addresses, controller would allocate IPv4 prefixes. This commit adds the documentation related to the same including HLD, troubleshooting guide and various configuration options.

Author: rawahars

README.md

@@ -26,8 +26,23 @@ Note: The SecurityGroupPolicy CRD only supports up to 5 security groups per cust
 
 The controller manages the IPv4 Addresses for all the Windows Node in EKS Cluster and allocates IPv4 Address to Windows Pods. The Networking on the host is setup by [amazon-vpc-cni-plugins](https://github.com/aws/amazon-vpc-cni-plugins).
 
+The controller supports the following modes for IPv4 address management on Windows-
+- **Secondary IPv4 address mode** → Secondary private IPv4 addresses are assigned to the primary instance ENI and the same are allocated to the Windows pods.
+  <br/><br/>
+  For more details about the high level workflow, please visit our documentation [here](docs/windows/secondary_ip_mode_workflow.md).
+
+
+- **Prefix delegation mode** &rarr; /28 IPv4 prefixes are assigned to the primary instance ENI and the IP addresses from the prefix are allocated to the Windows pods.
+  <br/><br/>
+  For more details about the configuration options with *prefix delegation*, please visit our documentation [here](docs/windows/prefix_delegation_config_options.md).
+  
+  For more details about the high level workflow, please visit our documentation [here](docs/windows/prefix_delegation_hld_workflow.md).
+
 Please follow this [guide](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html) for enabling Windows Support on your EKS cluster.
 
+## Troubleshooting
+For troubleshooting issues related to Security group for pods or Windows IPv4 address management, please visit our troubleshooting guide [here](docs/troubleshooting.md).
+
 ## License
 
 This library is licensed under the Apache 2.0 License. 

docs/images/windows-prefix-delegation-node-create-events.jpg

@@ -1,3 +1,33 @@
+# Troubleshooting Guide
+
+## Table of Contents
+- [Troubleshooting Windows](#troubleshooting-windows)
+  - [Verify if your EKS Cluster is on the required Platform Version](#verify-if-your-eks-cluster-is-on-the-required-platform-version)
+  - [Verify Windows IPAM is enabled in the ConfigMap](#verify-windows-ipam-is-enabled-in-the-configmap)
+  - [Verify Node has the Resource Capacity](#verify-node-has-the-resource-capacity)
+  - [Verify Pod has the resource limits](#verify-pod-has-the-resource-limit)
+  - [Verify Pod has the IPv4 Address Annotation](#verify-pod-has-the-ipv4-address-annotation)
+  - [Look for Issues on the Windows Host](#look-for-issues-on-the-windows-host)
+- [Troubleshooting Security Group for Pods](#troubleshooting-security-group-for-pods)
+  - [Verify ENI Trunking is Enabled](#verify-eni-trunking-is-enabled)
+  - [Verify Trunk ENI is created](#verify-trunk-eni-is-created)
+  - [Verify Pod has the resource limit](#verify-pod-has-the-resource-limit)
+  - [Verify Pod has the pod-eni annotation](#verify-pod-has-the-pod-eni-annotation)
+  - [Check Issues with VPC CNI](#check-issues-with-vpc-cni)
+- [Troubleshooting Prefix Delegation for Windows](#troubleshooting-prefix-delegation-for-windows)
+  - [Verify Windows prefix delegation is enabled in the ConfigMap](#verify-windows-prefix-delegation-is-enabled-in-the-configmap)
+  - [Check both pod events and node events for any specific error](#check-both-pod-events-and-node-events-for-any-specific-error)
+  - [Verify Node has the required Resource Capacity](#verify-node-has-the-required-resource-capacity)
+  - [Verify Pod has the required resource limits](#verify-pod-has-the-required-resource-limits)
+  - [Verify Pod has the required IPv4 Address Annotation](#verify-pod-has-the-required-ipv4-address-annotation)
+  - [Verify the configuration options set for windows prefix delegation](#verify-the-configuration-options-set-for-windows-prefix-delegation)
+  - [Look for networking issues on the Windows Host](#look-for-networking-issues-on-the-windows-host)
+- [List of Common Issues](#list-of-common-issues)
+  - [PSP Blocking Controller Annotations](#psp-blocking-controller-annotations)
+  - [Missing IAM Permissions on the Cluster Role](#missing-iam-permissions-on-the-cluster-role)
+  - [ENI/IP Exhaustion](#eniip-exhaustion)
+  - [Disable prefix delegation feature for Windows](#disable-prefix-delegation-feature-for-windows)
+
 ## Troubleshooting Windows
 
 Please follow the troubleshooting guide in the chronological order to debug issues with Windows Node and Pods.
@@ -227,6 +257,80 @@ If the Pod is still stuck in `ContainerCreating` you can,
 - Check the CNI Logs from the collected logs.
 - Open an [Issue](https://github.com/aws/amazon-vpc-resource-controller-k8s/issues/new/choose) in this repository if the problem still persists.
 
+## Troubleshooting Prefix Delegation for Windows
+Please follow the troubleshooting steps here for issues with Windows Node and Pods when using `prefix delegation` mode.
+
+The following steps should be checked in chronological order to find out any issues with the workflow.
+### Verify Windows prefix delegation is enabled in the ConfigMap
+
+To get the ConfigMap and the data field
+
+```bash
+kubectl get configmaps -n kube-system amazon-vpc-cni -o custom-columns=":data"
+```
+
+You should have the ConfigMap with the following data in the string,
+```
+enable-windows-ipam:true enable-windows-prefix-delegation:true
+```
+
+**Resolution**
+
+If the ConfigMap is missing or doesn't have the above field, you can create or update the `amazon-vpc-cni` ConfigMap with the required fields-
+```
+enable-windows-ipam: "true"
+enable-windows-prefix-delegation: "true"
+```
+
+**Note**: Windows IPAM needs to be enabled in order to use windows prefix delegation feature.
+
+### Check both pod events and node events for any specific error
+In case the controller encounters any error during it's prefix delegation workflow which needs to be acted upon by the customer, it will emit the errors as pod events and/or node events. Therefore, checking the same can be a good starting point to root cause the issue.
+
+You can obtain the pod events using the following command.
+```bash
+kubectl get events --all-namespaces
+```
+
+In case there is any explicit error, the same needs to be looked into.
+
+For example, if the error states that there are insufficient space in the subnet to carve a /28 prefix, then the subnet needs to be looked into to ensure that /28 ranges are available which can be allocated as prefixes.
+
+### Verify Node has the required Resource Capacity
+Same as [Verify Node has the Resource Capacity](#verify-node-has-the-resource-capacity)
+
+### Verify Pod has the required resource limits
+Same as [Verify Pod has the resource limits](#verify-pod-has-the-resource-limit)
+
+### Verify Pod has the required IPv4 Address Annotation
+Same as [Verify Pod has the IPv4 Address Annotation](#verify-pod-has-the-ipv4-address-annotation)
+
+### Verify the configuration options set for windows prefix delegation
+Configuration options can be used to fine-tune the behaviour of prefix delegation on Windows. The details about the options are available [here](windows/prefix_delegation_config_options.md).
+
+To get the ConfigMap and the data field
+
+```bash
+kubectl get configmaps -n kube-system amazon-vpc-cni -o custom-columns=":data"
+```
+
+If you see any of the following keys in the data-
+```
+minimum-ip-target
+warm-ip-target
+warm-prefix-target
+```
+Then the configuration options have been set.
+
+**Resolution**
+
+Verify if the configuration is correct as mentioned in the [documentation](windows/prefix_delegation_config_options.md).
+
+Alternatively, to isolate the issue, try removing the above keys from the config map.
+
+### Look for networking issues on the Windows Host
+Same as [Look for Issues on the Windows Host](#look-for-issues-on-the-windows-host)
+
 ## List of Common Issues  
 
 ### PSP Blocking Controller Annotations
@@ -265,3 +369,21 @@ aws ec2 describe-subnets --subnet-id subnet-id-here
 ```
 
 From the response you can look for how many IPv4 address are available in the Subnet from the field `AvailableIpAddressCount`
+
+### Disable prefix delegation feature for Windows
+
+You should check if the feature is enabled via ConfigMap. To get the ConfigMap and the data field
+
+```bash
+kubectl get configmaps -n kube-system amazon-vpc-cni -o custom-columns=":data"
+```
+
+If have the ConfigMap with the following data in the string,
+```
+enable-windows-prefix-delegation:true
+```
+then the feature is enabled.
+
+**Resolution**
+
+You can disable the feature by editing your config map and setting `enable-windows-prefix-delegation` as `"false"`.

docs/images/windows-prefix-delegation-pod-create-events.jpg

@@ -0,0 +1,58 @@
+# Configuration options with Prefix Delegation mode on Windows
+
+We provide multiple configurations which allow you to fine tune the pre-scaling and dynamic scaling behaviour. These configuration options can be set in the `amazon-vpc-cni` config map.
+
+* **warm-ip-target** → The number of IP addresses to be allocated in excess of current need. When used with prefix delegation, the controller allocates a new prefix to the ENI if the number of free IP addresses from the existing prefixes is less than this value on the node.
+
+   For example, consider that we set warm-ip-target to 15. Initially when the node starts, the ENI has 1 prefix i.e. 16 IP addresses allocated to it. When we launch 2 pods, then the number of available IP addresses becomes 14 and therefore, a new prefix will be allocated to the ENI, which brings the total count of available IP addresses to 30.
+
+
+* **warm-prefix-target** → The number of prefixes to be allocated in excess of current need. This will cause a new prefix (/28) to be allocated even if a single IP from the existing prefix is used. Therefore, use this configuration only if needed after careful consideration. A good use case is a scenario where there can be sudden spikes leading to scheduling of substantially high number of pods.
+
+   For example, consider that we set warm-prefix-target to 2. Initially when the node starts, 2 prefixes will be allocated to the ENI. Since there won’t be any running pods, the current need would be 0 and therefore, both the prefixes would be unused. If we run even a single pod then the current need would be 1 IP address which would come from 1 prefix. Therefore, only 1 prefix would be in excess of the current need. This would lead to 1 more prefix being allocated to the ENI bringing the total count of prefixes on the ENI to 3.
+
+
+* **minimum-ip-target** → The minimum number of IP addresses to be available at any time. This behaves identically to warm-ip-target except that instead of setting a target number of free IP addresses to keep available at all times, it sets a target number for a floor on how many total IP addresses are allocated.
+
+   For example, consider that we set minimum-ip-target to 20. This means that the total number of IP addresses (free and allocated to pods) should be at least 20. Therefore, even before the pods are scheduled, there should be at least 20 IP addresses available. Since 1 prefix has 16 IP addresses, the controller would allocate 2 prefixes bringing the total count of available IP address on the node to 32 which is greater than the set value of 20.
+
+### Considerations while using the above configuration options
+- These configuration options work only with the prefix delegation mode.
+- The settings for these values would depend upon your use case. If set, `warm-ip-target` and/or `minimum-ip-target` will take precedence over `warm-prefix-target`.
+- The default values used by the controller when these configurations have not been explicitly specified are-
+  ```
+  warm-ip-target: "1"
+  minimum-ip-target: "3"
+  ```
+- Setting either `warm-prefix-target` or both `warm-ip-target` and `minimum-ip-target` to zero/negative values is not supported. In such cases, default values as above will be used by the controller.
+- If only `minimum-ip-target` is set, `warm-ip-target` defaults to 1. If only `warm-ip-target` is set, `minimum-ip-target` defaults to 3.
+- If the values of `warm-prefix-target`, `warm-ip-target` or `minimum-ip-target` are set such that the max node IPv4 capacity is exceeded, then the maximum allocated IP addresses would be limited to the max node IPv4 capacity. For example, if a node has 14 secondary IP slots and we set `warm-prefix-target` to 20, then only 14 prefixes will be allocated on node startup.
+
+### Examples
+The following table shows the various parameters when the configuration options for prefix delegation are set. You can fine-tune these values as per your expected workload.
+
+|warm-prefix-target|warm-ip-target|minimum-ip-target|Pods|Attached Prefixes|Pod per Prefixes|Unused Prefixes|Unused IPs|
+|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+|-|-|-|0|1|0|1|16|
+|-|-|-|15|1|15|0|1|
+|-|-|-|16|2|16,0|1|16|
+|1|-|-|0|1|0|1|16|
+|3|-|-|0|3|0|3|48|
+|1|-|-|5|2|5|1|27|
+|1|-|-|17|3|16,1|1|31|
+|-|-|1|0|1|0|1|16|
+|-|-|1|5|1|5|0|11|
+|-|-|1|15|1|15|0|1|
+|-|-|1|16|2|16,0|1|16|
+|-|5|-|0|1|0|1|16|
+|-|5|-|5|1|5|0|11|
+|-|5|-|11|1|11|0|5|
+|-|5|-|12|2|12,0|1|20|
+|-|1|1|0|1|0|1|16|
+|-|1|1|5|1|5|0|11|
+|-|1|1|17|2|16,1|0|15|
+|-|3|1|14|2|14,0|1|18|
+|-|7|20|0|2|0,0|2|32|
+|-|7|20|5|2|5,0|1|27|
+|-|7|20|17|2|16,1|0|15|
+|-|7|20|26|3|16,10,0|1|22|

docs/troubleshooting.md

@@ -0,0 +1,31 @@
+# Windows Event Workflows in IPv4 Prefix Delegation mode
+This document presents high level workflow diagram for Events associated with Windows Nodes and Pods when using the IPv4 prefix delegation mode.
+
+## Adding a Windows Node to the Cluster
+
+<img alt="New Windows Node Create Event Diagram" height="50%" src="../images/windows-prefix-delegation-node-create-events.jpg" width="50%"/>
+
+1. Controller watches for Node Event from the Kube API server.
+2. User Adds a Windows Node to the Cluster with the label `kubernetes.io/os: windows`.
+3. Resource controller would start managing an IP address warm pool for the Windows node. It would invoke EC2 APIs on behalf of the customer to allocate /28 prefixes to the primary ENI. Internally, it would deconstruct the prefix into IP addresses and the pods would later be assigned one of the IP address from the prefix range. 
+  
+   In order to reduce latency after pod creation, controller would warm up a prefix beforehand. Customer can control the pre-scaling/warm settings using configuration options as specified [here](prefix_delegation_config_options.md).
+4. Controller updates the resource capacity on this node to `vpc.amazonaws.com/PrivateIPv4Address: # (Secondary IP per interface -1)*16`. This limits the Number of Windows Pod that can be scheduled on Windows Node based on the number of available IPv4 addresses.
+
+## Creating a new Windows Pod
+
+<img alt="New Windows Pod Create Event Diagram" height="50%" src="../images/windows-prefix-delegation-pod-create-events.jpg" width="50%"/>
+
+1. User Creates a new Windows Pod with the nodeSelector `kubernetes.io/os: windows`.
+2. Webhook mutates the Create Pod request by adding the following resource limit and capacity `vpc.amazonaws.com/PrivateIPv4Address: 1`. This tells the scheduler that the Pod has to be scheduled on a Node with 1 available IPv4 Address.
+3. Controller receives the Pod Create event and allocates a IPv4 address from the Prefix Warm Pool. The IP address assigned to the pod would be from the range of one of the prefixes assigned to the primary ENI on the node.
+
+   It is worthwhile to note that the controller would assign the IP address to the pods such that the prefix with the fewest remaining IP addresses would be consumed first. This means that if there are 2 prefixes on the node such that 10 IP addresses from the second prefix are yet to be allocated and 5 from the first, then newer pods will be allocated the IP addresses from the first prefix while it has unassigned IP addresses.
+
+4. Controller annotates the Pod with `vpc.amazonaws.com/PrivateIPv4Address: IPv4 Address`.
+5. VPC CNI Plugin Binary on the Windows host reads the IPv4 address present in the annotation from API Server and sets up the Networking for the Pod
+
+
+## Delete events
+
+When the pods are terminated, the IP addresses are released back into the warmpool. If the available IP addresses in the warmpool are greater than the required number, then controller will release the free prefixes. This essentially means that a prefix is released back only if all the IP addresses from its range are unallocated from the pods.

docs/windows/prefix_delegation_config_options.md

@@ -1,5 +1,5 @@
-# Windows Event Workflows
-This document presents high level workflow diagram for Events associated with Windows Nodes and Pods.
+# Windows Event Workflows in Secondary IPv4 address mode
+This document presents high level workflow diagram for Events associated with Windows Nodes and Pods when using the secondary IPv4 address mode.
 
 ## Adding a Windows Node to the Cluster
 ![New Windows Node Create Event Diagram](../images/windows-node-create-event.png)