-
Notifications
You must be signed in to change notification settings - Fork 169
Description
Describe the bug
using topologySpreadConstraints on a n pod setup with n zones will likely result in an unbalanced distribution after a redeploy/restart. Problem seems to be that the Update strategy (+1, -1, ...) ends with removing a last pod so this can lead to a unbalanced result. from the kubernetes documentation https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ it looks like the expected setup would be to use matchLabelKeys with generated "pod-template-hash" which seemed to solve the issue in our setup.
With matchLabelKeys, you don't need to update the pod.spec between different revisions. The controller/operator just needs to set different values to the same label key for different revisions. The scheduler will assume the values automatically based on matchLabelKeys. For example, if you are configuring a Deployment, you can use the label keyed with pod-template-hash, which is added automatically by the Deployment controller, to distinguish between different revisions in a single Deployment.
To Reproduce
Steps to reproduce the behavior:
- deploy a 3 node redis-ha-haproxy setup on a 3 zones cluster
- "restart" or redeploy the setup
- check node assigments (in our case >50% ended in unbalanced assignment)
Expected behavior
all Pods will be (optimal) balanced accross available zones
Additional context
due to the persitentVolume claim binding as well as other Update mechanisms probably not an issue on the redis nodes
potential change:
`{{- if .Values.topologySpreadConstraints.enabled }}
topologySpreadConstraints:
- maxSkew: {{ .Values.topologySpreadConstraints.maxSkew | default 1 }}
topologyKey: {{ .Values.topologySpreadConstraints.topologyKey | default "topology.kubernetes.io/zone" }}
whenUnsatisfiable: {{ .Values.topologySpreadConstraints.whenUnsatisfiable | default "ScheduleAnyway" }}
labelSelector:
matchLabels:
app: {{ template "redis-ha.name" . }}-haproxy
release: {{ .Release.Name }}
matchLabelKeys:- pod-template-hash
{{- end }}`
- pod-template-hash