Setting option for node draining by external controllers #952

heyvister1 · 2025-10-09T09:43:07Z

SRIOV OP internal drain controller can be disabled, through USE_EXTERNAL_DRAINER. For example draining can be performed by external NVIDIA maintenance operator

github-actions · 2025-10-09T09:43:16Z

Thanks for your PR,
To run vendors CIs, Maintainers can use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs, Maintainers can use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

coveralls · 2025-10-09T10:04:01Z

Pull Request Test Coverage Report for Build 19669073220

Details

14 of 21 (66.67%) changed or added relevant lines in 4 files are covered.
15 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-0.04%) to 61.986%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/daemon/daemon.go	10	13	76.92%
controllers/drain_controller.go	1	5	20.0%

Files with Coverage Reduction	New Missed Lines	%
controllers/helper.go	2	70.61%
pkg/utils/cluster.go	2	85.57%
pkg/daemon/daemon.go	11	44.16%

Totals
Change from base Build 19668853053:	-0.04%
Covered Lines:	8763
Relevant Lines:	14137

💛 - Coveralls

SchSeba

In general I am fine with having this here. but can we please have a more generic name for the variable?

SchSeba · 2025-10-15T17:32:40Z

controllers/helper.go


+// UseMaintenanceOperatorDrainer indicates if internal drain controller is disabled
+// and draining will be done by external NVIDIA maintenance operator
+func UseMaintenanceOperatorDrainer() bool {


can we please have this in the vars and consts folder and not here.

heyvister1 · 2025-10-19T06:16:52Z

/test-all

adrianchiris · 2025-10-19T08:50:06Z

pkg/daemon/daemon.go


+	// add external drainer annotation if enabled
+	if vars.UseExternalDrainer {
+		if err := utils.AnnotateNode(ctx,


can we annotate nodeState instead ?

as we store related drain state in nodeState obj

adrianchiris · 2025-10-19T08:54:07Z

main.go

 	}
 }

+func setupDrainController(mgr ctrl.Manager, restConfig *rest.Config,


Perhaps we should take a bit different approach.

we could always start the drain controller and then skip drain requests if use-external-drainer annotation is set.

that way, if there are any "in-flight" drains they will complete even if we switched on the external-drainer functionality.

WDYT ?

what about using the same annotations as the regular drain flow?

the daemon would set the sriovnetworknodestate.DesiredState == "Drain_Required"

the in-operator drainer would do nothing

the external drainer would drain the node using its own logic, then set the node.CurrentState=DrainComplete

would it be a cleaner implementation?

e0ne

LGTM

adrianchiris · 2025-10-26T08:03:21Z

/test-all

adrianchiris · 2025-11-09T14:33:44Z

doc/design/parallel-node-config.md


 *NOTE:* In the future we are going to drop the node annotation and only use the SriovNetworkNodeState

+*NOTE:* Internal drain controller can be disabled by exposing the following `USE_EXTERNAL_DRAINER` env variable. This means that drain operations will be done externally, for example by utilizing [NVIDIA maintenance OP](https://github.com/Mellanox/maintenance-operator). In addition, `SriovNetworkPoolConfig` will not take any effect during drain procedure, since the maintenance operator will be in charge of parallel node operations.


i dont think we ever talked about changes to existing design docs if new addition changes that feature's behaviour.
i guess its ok but it might pose unmanageable in the future.

@SchSeba @zeeke thoughts ? should we have this addition here or just in main README ?

I am fine adding that change also here not only in the README and update the updatedDate in the doc.

add the link to this PR (Example https://github.com/k8snetworkplumbingwg/sriov-network-operator/pull/917/files#diff-dc6ff1ecffa429d5635307cc009fe47ad0d51f41d549dbf88f67752c7cfbebc6R9)

adrianchiris · 2025-11-09T14:35:58Z

main.go

+func setupDrainController(mgr ctrl.Manager, restConfig *rest.Config,
+	platformsHelper platforms.Interface, scheme *runtime.Scheme) error {
+	if vars.UseExternalDrainer {
+		setupLog.Info("'UseExternalDrainer' is set, draining will be done externally")


please add a note on why we still setup the drain controller here even if UseExternalDrainer is set.

It's more complicated.

if we are in the middle, and we update the config-daemon yaml it will start new pods that will add the label.
in this case we will be in the middle of configuration with the 2 labels in parallel

adrianchiris

LGTM added minor nit to clarify why we need to start drain controller when external drainer is used.

zeeke · 2025-11-11T14:41:03Z

controllers/drain_controller_test.go

 }

-func createNode(ctx context.Context, nodeName string) (*corev1.Node, *sriovnetworkv1.SriovNetworkNodeState) {
+func createNode(ctx context.Context, nodeName string, useExternalDrainer bool) (*corev1.Node, *sriovnetworkv1.SriovNetworkNodeState) {


please, avoid the boolean parameter as it reduce readability.
as alternatives:
a. change the parameter to additionalAnnotation map[string]string
b. add a tweak parameter like

tweak func(*corev1.Node, *sriovnetworkv1.SriovNetworkNodeState) so that a function can customize the creation of the node objects

c. remove the parameter and update the k8s objects after the creation

I prefer b, but the others would work too

zeeke · 2025-11-11T14:41:39Z

controllers/drain_controller_test.go

 			Name:      nodeName,
 			Namespace: vars.Namespace,
-			Labels: map[string]string{
+			Annotations: map[string]string{


Any idea about how these tests were working before this line change?

if I remember right the controller adds them if they don't exist.

SchSeba

I must say I don't follow this sorry.

the daemon will still request drain and this check https://github.com/k8snetworkplumbingwg/sriov-network-operator/pull/952/files#diff-a53b7b593d3d778e62eaeeafa40088656f9212bfa2c2b7991df15fa78e60b0f0R256
will never pass.

also you can have a race as we do the annotation update twice in

	// add external drainer nodestate annotation if flag is enabled
	if vars.UseExternalDrainer {
		err := utils.AnnotateObject(ctx, desiredNodeState,
			consts.NodeStateExternalDrainerAnnotation, "true", dn.client)
		if err != nil {
			funcLog.Error(err, "failed to add nodestate external drainer annotation")
			return false, err
		}
	}

	// annotate both node and node state with drain or reboot
	annotation := consts.DrainRequired
	if reqReboot {
		annotation = consts.RebootRequired
	}
	return true, dn.annotate(ctx, desiredNodeState, annotation)
}

and I don't understand how the daemon will know that he is able to continue with the configuration because the drain was done.

SchSeba · 2025-11-16T13:01:48Z

controllers/drain_controller_test.go

 			Name:      nodeName,
 			Namespace: vars.Namespace,
-			Labels: map[string]string{
+			Annotations: map[string]string{


if I remember right the controller adds them if they don't exist.

SchSeba · 2025-11-16T13:06:27Z

pkg/daemon/daemon.go


+	// remove external drainer nodestate annotation if exists
+	annotations := desiredNodeState.GetAnnotations()
+	if _, ok := annotations[consts.NodeStateExternalDrainerAnnotation]; ok {


can you explain why we need this here?

SchSeba · 2025-11-16T13:12:46Z

main.go

+func setupDrainController(mgr ctrl.Manager, restConfig *rest.Config,
+	platformsHelper platforms.Interface, scheme *runtime.Scheme) error {
+	if vars.UseExternalDrainer {
+		setupLog.Info("'UseExternalDrainer' is set, draining will be done externally")


It's more complicated.

if we are in the middle, and we update the config-daemon yaml it will start new pods that will add the label.
in this case we will be in the middle of configuration with the 2 labels in parallel

…sable SRIOV OP drain controller, in favor of using maintenance OP to drive node drain aspects Signed-off-by: Ido Heyvi <[email protected]>

…rnal-drainer=true' in case exteranl drainer is enabled The motivation is for external drainer verification, that SRIOV operator is set with external drainer Signed-off-by: Ido Heyvi <[email protected]>

github-actions bot added the docs label Oct 9, 2025

heyvister1 force-pushed the disable-drain-controller branch 2 times, most recently from 30699af to cc4a257 Compare October 9, 2025 09:52

heyvister1 mentioned this pull request Oct 9, 2025

Support node draining by external NVIDIA maintenance OP Mellanox/sriov-network-operator#84

Merged

almaslennikov approved these changes Oct 9, 2025

View reviewed changes

SchSeba reviewed Oct 15, 2025

View reviewed changes

heyvister1 force-pushed the disable-drain-controller branch from cc4a257 to 62632d8 Compare October 16, 2025 12:19

github-actions bot added ci tests labels Oct 16, 2025

heyvister1 force-pushed the disable-drain-controller branch from 62632d8 to 237e08b Compare October 16, 2025 12:25

github-actions bot removed ci tests labels Oct 16, 2025

heyvister1 requested review from SchSeba and almaslennikov October 16, 2025 12:26

heyvister1 force-pushed the disable-drain-controller branch 2 times, most recently from 16474e5 to cd01dcf Compare October 16, 2025 14:52

heyvister1 changed the title ~~Support node draining by external NVIDIA maintenance OP~~ Setting option for node draining by external controllers Oct 16, 2025

github-actions bot added the tests label Oct 17, 2025

heyvister1 force-pushed the disable-drain-controller branch from a7fbd43 to c3d7edf Compare October 17, 2025 08:23

almaslennikov approved these changes Oct 17, 2025

View reviewed changes

heyvister1 force-pushed the disable-drain-controller branch 4 times, most recently from daaf0f6 to 7fed536 Compare October 17, 2025 09:48

adrianchiris reviewed Oct 19, 2025

View reviewed changes

heyvister1 force-pushed the disable-drain-controller branch from 7fed536 to ade293d Compare October 19, 2025 10:13

github-actions bot added the ci label Oct 20, 2025

heyvister1 force-pushed the disable-drain-controller branch from b2877d5 to 40509de Compare October 20, 2025 08:53

github-actions bot removed the ci label Oct 20, 2025

heyvister1 force-pushed the disable-drain-controller branch 3 times, most recently from 62ddbb0 to 24a2bfe Compare October 20, 2025 10:02

heyvister1 mentioned this pull request Oct 20, 2025

Disable drain controller forked Mellanox/sriov-network-operator#89

Merged

e0ne approved these changes Oct 21, 2025

View reviewed changes

heyvister1 force-pushed the disable-drain-controller branch 6 times, most recently from b8c3b8c to 721a627 Compare October 24, 2025 06:04

heyvister1 force-pushed the disable-drain-controller branch 3 times, most recently from 914739d to 30a21f9 Compare October 28, 2025 12:40

adrianchiris reviewed Nov 9, 2025

View reviewed changes

adrianchiris approved these changes Nov 9, 2025

View reviewed changes

heyvister1 force-pushed the disable-drain-controller branch from 30a21f9 to fd9607c Compare November 10, 2025 06:51

heyvister1 requested a review from adrianchiris November 10, 2025 06:52

adrianchiris approved these changes Nov 10, 2025

View reviewed changes

zeeke reviewed Nov 11, 2025

View reviewed changes

SchSeba reviewed Nov 16, 2025

View reviewed changes

heyvister1 added 2 commits November 25, 2025 12:10

drain: adding 'USE_EXTERNAL_DRAINER' provoding an option to enable/di…

7f0548e

…sable SRIOV OP drain controller, in favor of using maintenance OP to drive node drain aspects Signed-off-by: Ido Heyvi <[email protected]>

setting nodestate annotation with 'sriovnetwork.openshift.io/use-exte…

c050234

…rnal-drainer=true' in case exteranl drainer is enabled The motivation is for external drainer verification, that SRIOV operator is set with external drainer Signed-off-by: Ido Heyvi <[email protected]>

heyvister1 force-pushed the disable-drain-controller branch from fd9607c to c050234 Compare November 25, 2025 12:11


		NOTE: In the future we are going to drop the node annotation and only use the SriovNetworkNodeState

		NOTE: Internal drain controller can be disabled by exposing the following `USE_EXTERNAL_DRAINER` env variable. This means that drain operations will be done externally, for example by utilizing [NVIDIA maintenance OP](https://github.com/Mellanox/maintenance-operator). In addition, `SriovNetworkPoolConfig` will not take any effect during drain procedure, since the maintenance operator will be in charge of parallel node operations.

Setting option for node draining by external controllers #952

Are you sure you want to change the base?

Setting option for node draining by external controllers #952

Conversation

heyvister1 commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 9, 2025

Uh oh!

coveralls commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 19669073220

Details

💛 - Coveralls

Uh oh!

SchSeba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

heyvister1 commented Oct 19, 2025

Uh oh!

adrianchiris Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

e0ne left a comment

Choose a reason for hiding this comment

Uh oh!

adrianchiris commented Oct 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrianchiris left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SchSeba left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

heyvister1 commented Oct 9, 2025 •

edited

Loading

coveralls commented Oct 9, 2025 •

edited

Loading

adrianchiris Oct 19, 2025 •

edited

Loading

SchSeba left a comment •

edited

Loading