Skip to content

[chart/redis-ha][BUG] topologySpreadConstraints not working as expected on updates #333

@pkollmer

Description

@pkollmer

Describe the bug
using topologySpreadConstraints on a n pod setup with n zones will likely result in an unbalanced distribution after a redeploy/restart. Problem seems to be that the Update strategy (+1, -1, ...) ends with removing a last pod so this can lead to a unbalanced result. from the kubernetes documentation https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ it looks like the expected setup would be to use matchLabelKeys with generated "pod-template-hash" which seemed to solve the issue in our setup.

With matchLabelKeys, you don't need to update the pod.spec between different revisions. The controller/operator just needs to set different values to the same label key for different revisions. The scheduler will assume the values automatically based on matchLabelKeys. For example, if you are configuring a Deployment, you can use the label keyed with pod-template-hash, which is added automatically by the Deployment controller, to distinguish between different revisions in a single Deployment.

To Reproduce
Steps to reproduce the behavior:

  1. deploy a 3 node redis-ha-haproxy setup on a 3 zones cluster
  2. "restart" or redeploy the setup
  3. check node assigments (in our case >50% ended in unbalanced assignment)

Expected behavior
all Pods will be (optimal) balanced accross available zones

Additional context
due to the persitentVolume claim binding as well as other Update mechanisms probably not an issue on the redis nodes

potential change:
`{{- if .Values.topologySpreadConstraints.enabled }}
topologySpreadConstraints:

  • maxSkew: {{ .Values.topologySpreadConstraints.maxSkew | default 1 }}
    topologyKey: {{ .Values.topologySpreadConstraints.topologyKey | default "topology.kubernetes.io/zone" }}
    whenUnsatisfiable: {{ .Values.topologySpreadConstraints.whenUnsatisfiable | default "ScheduleAnyway" }}
    labelSelector:
    matchLabels:
    app: {{ template "redis-ha.name" . }}-haproxy
    release: {{ .Release.Name }}
    matchLabelKeys:
    • pod-template-hash
      {{- end }}`

Metadata

Metadata

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions