KEP-5471 Extended Toleration Operators for Threshold-Based Placement

[helayoty]

• One-line PR description: Add numeric comparison operators (Lt, Gt) to Tolerations for SLA-based scheduling with threshold-based taint matching.

• Issue link: Extended Toleration Operators for Threshold-Based Placement #5471

• Other comments: cc @kubernetes/sig-scheduling-misc @kubernetes/sig-apps-misc

[SergeyKanzhelev]

can we include the scenario of adding tolerations on the semantic version comparison? It will likely require either a new operator or some way to express that a taint's string needs to be parsed as a semver.

[helayoty]

The current scope for this KEP is integer-only support for SLA/failure-probability because it’s the simplest and safest to implement (avoids floating-point parsing and complex type semantics).

I agree that semver comparisons are a valid and important future use case (e.g., kubelet version taints, device firmware versions). To keep this KEP narrowly scoped and implementable, I propose to document the semantic version comparison in a new Future Work section. And also consider if we would add this to the node affinity as well. wdyt?

[SergeyKanzhelev]

One consideration here is what is the delta here. It is a lot of red tape to run thru scenarios and API reviews. Making two changes together may be much easier than splitting into separate KEPs.

I am not objecting to this scope though. This is not blocking proposal, but I believe it will be very useful.

BTW, we may also want to support semver in CEL as well =). So potentially another scope increase

[helayoty]

Updated the scope to include both semver and CEL.

[helayoty]

@sanposhiho fyi for semver addition.

[macsko]

What about the scoring for taints with PreferNoSchedule?

[helayoty]

Great Question. For PreferNoSchedule taints, numeric operators only determine whether the taint is considered as tolerated for scoring, that maintains consistent soft-preference behavior while enabling threshold-based SLA matching. I've updateed the KEP with more details:
https://github.com/kubernetes/enhancements/pull/5473/files#diff-91266bbd6ed6b175ba3fea357b964cc03e25c3f95351ba69b8f63711d9875ac7R450

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: helayoty
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric, sanposhiho for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[macsko]

/cc @dom4ha @sanposhiho

[jackfrancis]

Is it fair to add a goal to ensure that the addition of new operators has zero operational effect on existing pod scheduling outcomes using the Equality or Exists operators? I'm thinking of the various switch toleration.Operator bits in source and assuming that adding more operators should have zero cost to existing scenarios.

[jackfrancis]

nit: maybe we add "interruptible batch workloads" or checkpoint-able or other similar language to recognize that there is such a thing as a "critical batch workload" that would not be able to benefit from running on interruptible hardware.

[jackfrancis]

Something to consider: for every scenario that is (strictly speaking) possible using existing semantics (e.g., NodeAffinity) it might be helpful to include that scenario here. A before/after type picture. Not wanting to add too much information to the KEP, but that would help folks to assess the ergonomic improvements.

[jackfrancis]

Should this be Ge?

[jackfrancis]

Maybe we can also inclue an example pod spec that will explicitly not get scheduled onto a spot node, e.g.,

---
# Critical workload will not be scheduled until a suitable high reliability node has capacity
apiVersion: v1
kind: Pod
metadata:
  name: critical-workload
spec:
  tolerations:
  - key: node.kubernetes.io/sla
    operator: Gt
    value: "950"
    effect: NoSchedule

[jackfrancis]

Let's add the ResourceSlice example that includes the taint that ResourceClaim "gpu-claim-high-sla" will tolerate.

[jackfrancis]

Nit: Let's call this a "Short-lived batch training workload" to distinguish from other (very normal) batch workloads that would not want to be interrupted in an hour.

[jackfrancis]

Nit: to make this more current, let's add 1.33/1.34 as our two example versions here

[jackfrancis]

SemverGe for this and below in this example?

[jackfrancis]

Nit: the pre-release and build metadata points are part of semver, so we could drop them. Or if being super explicit is helpful, let's add the "per semver specification" suffix to the Pre-release versions line item as well.

[jackfrancis]

This implies I could have two nodes on my cluster like this:

apiVersion: v1
kind: Node
metadata:
  name: node-1
spec:
  taints:
  - key: node.kubernetes.io/foo
    value: "100"
    effect: NoExecute
---
apiVersion: v1
kind: Node
metadata:
  name: node-2
spec:
  taints:
  - key: node.kubernetes.io/foo
    value: "v1.34.0"
    effect: NoExecute

And that the scheduler would simply schedule workloads with proper tolerations using either of the above semantics (int comparison or semver comparison) to the right node.

Also, apiserver has no preference between either of the below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  template:
    spec:
      tolerations:
      - key: node.kubernetes.io/sla
        operator: Gt
        value: "950"
        effect: NoExecute
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  template:
    spec:
      tolerations:
      - key: node.kubernetes.io/sla
        operator: Gt
        value: "v1.33.0"
        effect: NoExecute

During scheduling, if we encounter a matching node taint whose value is of a different type than the workload toleration (int vs semver) then we simply continue and rule out that node as a candidate for scheduling.

Just thinking out loud here, did I get that right?

[jackfrancis]

Is this observation worth including as a Risk? E.g.:

**Risk**: Invalid taints meant to be used with the new comparison operators (e.g., `node.kubernetes.io/sla=95.5` and `node.kubernetes.io/version=1`) are not detected at admission time

[sanposhiho]

I want to ensure critical workloads only run on nodes with Kubernetes version >= 1.20.0 due to specific API features they require, while allowing development workloads to tolerate older versions.

Why try adding SemverGt to the toleration, not nodeaffinity? I liked your argument on Why not NodeAffinity alone? section, but I'm not sure that reasoning would justify this semver addition.

[sanposhiho]

I just saw you added this semver thing at the last commit along with CEL, which grew the scope of this KEP. I prefer to have another KEP to discuss them.