diff --git a/keps/sig-node/5425-automatic-swap-node-labels/README.md b/keps/sig-node/5425-automatic-swap-node-labels/README.md new file mode 100644 index 00000000000..1ada4e5a431 --- /dev/null +++ b/keps/sig-node/5425-automatic-swap-node-labels/README.md @@ -0,0 +1,761 @@ +# KEP-5425: Automatic Swap Node Labels + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Background](#background) + - [NodeFeatureDiscovery Limitations](#nodefeaturediscovery-limitations) +- [Proposal](#proposal) + - [User Stories](#user-stories) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + +This KEP proposes that kubelet automatically set well-known node labels based on swap configuration to enable workload isolation and scheduling decisions for swap-enabled nodes. + +## Motivation + +### Goals + +- Enable workload isolation between swap-enabled and swap-disabled nodes. +- Support nodeSelector and nodeAffinity based node selection for swap use-cases without requiring external tools. + +### Non-Goals + +- Implementing API-based swap limits (separate KEP). +- Modifying existing Swap modes or functionality at node. +- Providing swap capacity information for quantitative scheduling. + +## Background + +Currently, Kubernetes supports swap through node-level configuration (`NoSwap`, `LimitedSwap`), but workloads have no way to discover or request specific swap behavior from nodes. This KEP proposes Kubelet adding a new node-label for swap-enabled nodes during bootstrap. + +This will help with below scenarios: + +1. Enable targeted scheduling for swap demanding pods to swap enabled nodes. +2. Guard workloads that will have concerns about performance inference from accidentally placed on a swap enabled node. +2. Users missing to manually label nodes for swap-based scheduling. +3. Avoid external tooling dependency (NodeFeatureDiscovery) for swap feature discovery. + +### NodeFeatureDiscovery Limitations + +While NFD can detect swap devices on nodes, it has some limitations: +- NFD labels reflect hardware state (swap device presence), not Kubernetes configuration state +- It cannot distinguish between `NoSwap`, `LimitedSwap`, or future swap modes +- NFD is not an in-tree solution, requiring additional deployments for working with swap. + +## Proposal + +### API Changes + +Kubelet will automatically set the following well-known node labels based on swap configuration: + +```yaml +metadata: + labels: + # Indicates the kubelet swap behavior mode + node.kubernetes.io/swap-behavior: "NoSwap" | "LimitedSwap" +``` + +### Label Semantics + +#### `node.kubernetes.io/swap-behavior` +- **`"NoSwap"`**: Workloads will not use swap (disabled by configuration). +- **`"LimitedSwap"`**: Node-level swap with implicit calculation + +#### Integration Point + +There is precedent in Kubernetes for auto-labeling of nodes based on hardware / os features / cloud-topology. + +eg: below labels are set by Kubelet +```bash +kubernetes.io/arch - CPU architecture +kubernetes.io/hostname - Node hostname +kubernetes.io/os - Operating system +node.kubernetes.io/instance-type - Instance type from cloud provider +topology.kubernetes.io/region - Cloud region +topology.kubernetes.io/zone - Cloud availability zone +``` + +Swap labels will also be set during initial node status update, following the same pattern as existing kubelet-managed labels. + +### Implementation + +#### Kubelet Changes + +```go +// api/core/v1/well_known_labels.go +const LabelLinuxSwapBehavior = "node.kubernetes.io/swap-behavior" + +// pkg/kubelet/kubelet_node_status_others.go +func (kl *Kubelet) getNodeSwapLabels(node *v1.Node) (map[string]string, error) { + swapBehavior := kl.kubeletConfiguration.MemorySwap.SwapBehavior + + // Set node label based on swap-presence and configuration + found, err := swapDetected() + if err != nil { + return nil, err + } + + if swapBehavior == "NoSwap" { + if found { + klog.Warning("Swap is detected at node, but swap disabled for workloads because configuration is NoSwap.") + } + } else { + if !found { + klog.Warningf("Swap configured(%v) but swap device not detected at node.", swapBehavior) + } + } + + return map[string]string{v1.LabelLinuxSwapBehavior: string(swapBehavior)}, nil +} +``` + +### User Stories + +#### Story 1: Workload Isolation +As a cluster administrator, I want to isolate swap-sensitive workloads from swap-enabled nodes to prevent performance degradation. + +```yaml +# Non-swap workload +spec: + nodeSelector: + node.kubernetes.io/swap-behavior: "NoSwap" +``` + +#### Story 2: Swap-Required Workloads +As a developer, I want my memory-intensive batch jobs to run only on swap-enabled nodes for better resource utilization. + +```yaml +# workload desiring swap +spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node.kubernetes.io/swap-behavior + operator: NotIn + values: ["NoSwap"] +``` + +#### Story 3: Migration Planning +As a cluster operator, I want to identify which nodes have which swap configuration during cluster migration. + +```bash +kubectl get nodes -l node.kubernetes.io/swap-behavior=LimitedSwap +``` + +## Design Details + +### Label Lifecycle + +- Labels are set during kubelet startup and initial node registration / reconciliation. +- Labels are updated when kubelet configuration changes +- Labels persist through node restarts + +### Backwards Compatibility + +- New labels are additive. There will be no breaking changes +- Existing swap functionality is unchanged with the new label proposed +- Nodes without explicit swap configuration will get a new `node.kubernetes.io/swap-behavior: "NoSwap"` label on kubelet / node restart + +### Validation + +- Kubelet validates that swap configuration matches detected system state. This will raise kubelet warning logs if swap behavior is `LimitedSwap` but no swap is detected. + +### Test Plan + +[X] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + +- kubelet + + - Test `getNodeSwapLabels()` function with different swap configurations + - Test label updates when swap configuration changes + - Test feature gate enabled/disabled scenarios + +##### Integration tests + +- `TestKubeletSwapLabels` - kubelet startup with swap configuration + - Test kubelet sets correct labels on node registration + - Test label updates when kubelet configuration is updated +- `TestSwapLabelsPersistence` - Validate label persistence across kubelet restarts +- `TestSwapLabelsValidation` - Test validation of swap configuration vs system state + +##### e2e tests + +- E2E test to validate workload scheduling based on swap labels +- In a cluster with swap-enabled nodes, node-query with labels should filter the right set of nodes. + +### Graduation Criteria + + + +**Alpha** + +- Feature is implemented behind the feature gate `AutomaticSwapNodeLabels` +- Basic unit tests are implemented and passing +- Initial integration tests are completed +- Swap Documentation is updated with automatic node-label enhancement details. +- Manual verification of swap node labels for different swap configurations is completed. + +**Beta** +- Feature gate is enabled by default +- Added comprehensive test coverage (unit, integration, e2e) + +### Upgrade / Downgrade Strategy + + + +**Upgrade** + +Users can enable the `AutomaticSwapNodeLables` feature gate on Kubelet after upgrading to a Kubernetes version that supports it. On Kubelet startup, new swap labels will be automatically added to nodes + +**Downgrade** + +Downgrading kubelet to a version without `AutomaticSwapNodeLabels` feature will not remove the pre-created labels; this is also desirable to ensure scheduling decisions will continue to respect the node-isolation. For a clean downgrade, users will have to manually delete the labels if they are no longer desired after downgrade as below: + +```bash +kubectl label nodes --all node.kubernetes.io/swap-behavior- +``` + +### Version Skew Strategy + + + +Nodes with older kubelets won't have swap labels, but functionality is not impacted. Node selection with swap behavior labels will only include swap-enabled nodes in newer versions. This would have implication that workloads using swap labels may not find suitable nodes during mixed-version periods. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: `AutomaticSwapNodeLabels` + - Components depending on the feature gate: `kubelet` +- [X] Other + - Describe the mechanism: Kubelet add preset labels for swap detection and mode on startup. + - Will enabling / disabling the feature require downtime of the control + plane? No + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? No + +###### Does enabling the feature change any default behavior? + + + +Yes, but minimally. New swap relevant node-labels will appear on the node objects / `kubectl describe node` output after upgrade. No functional behavior changes to existing workloads or scheduling. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes. Users could set feature gate `AutomaticSwapNodeLabels=false`. During rollback, the nodes with the feature disabled would stop reporting the node label `node.kubernetes.io/swap-behavior`. Manual action will be required if user intended to remove the labels. If swap configuration at the node is changed, labels that were not cleaned-up will reflect stale configuration. If labels were removed, workloads using the label-based node-swap filtering will no longer follow the swap-node-isolation. + +###### What happens if we reenable the feature if it was previously rolled back? + +If the feature is re-enabled, swap-behavior labels will be re-added based on current swap configuration. + +###### Are there any tests for feature enablement/disablement? + +Yes, feature enablement/disablement tests will be added along with alpha implementation. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +###### Will enabling / using this feature result in introducing new API types? + + + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + +* 2025-06-19: [Proposal](https://github.com/kubernetes/kubernetes/issues/132416) and discussion. + +## Drawbacks + + + +## Alternatives + +### Manual Labeling Only +**Not Preferred**: Requires manual cluster management and doesn't scale. Since swap is considered a high-risk configuratio, it is preferrable to include this node metadata as a built-in solution. + +### Use NodeFeatureDiscovery +**Not Preferred**: +- NFD detects swap presence in the machine, not kubelet configuration +- External dependency for core functionality +- Cannot distinguish between swap modes + +### Automatic Swap Taints +**Not Preferred**: Taints are more disruptive for swap enabled nodes to be added at bootstrap. This will require admins add tolerations to node-management pods and other critical daemon-sets for swap nodes, requiring to manage a second set of yamls which is undesirable. Auto-Labels will help users intend to manage swap with taints with selection queries such as: + +`kubectl taint nodes -l node.kubernetes.io/swap-behavior=LimitedSwap swap-enabled=true:NoSchedule` + +### Extending NodeCapabilities API for swap +**Early In Design**: Labels provide better integration with existing node-selection mechanisms (nodeSelector, affinity, etc.), and applicable for more use-cases such as monitoring. NodeCapabilities API (as proposed) in early design stages and is considered only for Kubelet feature-discovery currently. NodeCapabilities for swap could help with complex swap-aware scheduling needs, if integrated in the future. If swap-capability is exported by node in the future, it can co-exist with labels which are generic node metadata that can help with other filtering needs. + +### Include Swap Capacity Information +**Not Preffered**: +- Conflicts with API / adds complexity without clear use case +- Capacity can change dynamically, making labels stale + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-node/5425-automatic-swap-node-labels/kep.yaml b/keps/sig-node/5425-automatic-swap-node-labels/kep.yaml new file mode 100644 index 00000000000..71bdfcf6672 --- /dev/null +++ b/keps/sig-node/5425-automatic-swap-node-labels/kep.yaml @@ -0,0 +1,28 @@ +ititle: AutomaticSwapLabels +kep-number: 5425 +authors: + - "@ajaysundark" +owning-sig: sig-node +participating-sigs: + - sig-node +creation-date: "2025-06-19" +reviewers: + - '@tallclair' + - '@SergeyKanzhelev' + - TBD +approvers: +- TBD + +status: provisional +stage: alpha +latest-milestone: "v1.34" +milestone: + alpha: "v1.34" + +feature-gates: + - name: AutomaticSwapLabels + components: + - kubelet +disable-supported: true + +metrics: []