Skip to content

test(evals): add hotplug eval tasks and multi-worker kind config - #1309

Open
lyarwood wants to merge 1 commit into
containers:mainfrom
lyarwood:cnv-87970-hotplug-evals
Open

test(evals): add hotplug eval tasks and multi-worker kind config#1309
lyarwood wants to merge 1 commit into
containers:mainfrom
lyarwood:cnv-87970-hotplug-evals

Conversation

@lyarwood

@lyarwood lyarwood commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Add mcpchecker eval tasks for KubeVirt hotplug use cases, a multi-worker Kind cluster configuration to support live migration, and CI workflow support for running them via /run-mcpchecker kubevirt-multinode or /run-mcpchecker all.

Eval Tasks

5 eval tasks covering hotplug operations using existing core resource tools (declarative VM spec edits):

  • hotplug-cpu-sockets — increase CPU sockets within maxSockets (requires live update)
  • hotplug-memory — increase guest memory within maxGuest (requires live update)
  • hotplug-instancetype — change instancetype on a running VM (requires live update)
  • hotplug-volume-add — attach a PVC with hotpluggable: true
  • hotplug-interface-add — add a Multus secondary network interface

All tasks use suite: kubevirt-multinode and requires: kubevirt-multinode labels, and follow the standard evals/tasks/kubevirt/<task-name>/task.yaml directory layout.

Test Results

Tested against a live OpenShift 4.22 cluster with OpenShift Virtualization 4.22.6 using claude-sonnet-4-6 via Vertex AI:

  • All agent assertions passed (15/15, 100%) — the model correctly performed declarative hotplug operations without requiring dedicated hotplug tools.

Infrastructure & Configuration

  • Added dev/config/kind/cluster-kubevirt-multinode.yaml with 2 worker nodes for live migration
  • Made KIND_CLUSTER_CONFIG configurable in build/kind.mk
  • Configured KubeVirt with LiveUpdate vmRolloutStrategy and LiveMigrate workloadUpdateStrategy in build/kubevirt.mk

CI Changes

  • Updated .github/workflows/mcpchecker.yaml matrix generation to support multi-node Kind cluster configs
  • /run-mcpchecker kubevirt-multinode and /run-mcpchecker all automatically configure the multi-node Kind cluster
  • Updated regex parsing order so kubevirt-multinode matches before kubevirt

Related

@Cali0707

Cali0707 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

mcpchecker kubernetes extension k8s.create silently fails for CRD resources (VirtualMachine) — setup steps may need converting to script-based kubectl apply

@lyarwood mind opening an issue on https://github.com/mcpchecker/kubernetes-extension/issues for this so that we can fix it?

@lyarwood

lyarwood commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

mcpchecker kubernetes extension k8s.create silently fails for CRD resources (VirtualMachine) — setup steps may need converting to script-based kubectl apply

@lyarwood mind opening an issue on https://github.com/mcpchecker/kubernetes-extension/issues for this so that we can fix it?

ACK apologies for not doing this at the time, I'll reproduce today and get something documented.

@lyarwood lyarwood closed this Jul 24, 2026
@lyarwood lyarwood reopened this Jul 24, 2026
@lyarwood
lyarwood force-pushed the cnv-87970-hotplug-evals branch 4 times, most recently from 1b4e803 to 7a34cc6 Compare July 24, 2026 11:49
@lyarwood
lyarwood marked this pull request as ready for review July 24, 2026 11:50
@lyarwood
lyarwood requested review from Cali0707 and manusa as code owners July 24, 2026 11:50
@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

👋 Heads up — this pull request changes files owned by @ksimon1.

You are listed as an owner of one or more of the changed areas in .github/CODEOWNERS. GitHub cannot auto-request review from owners without write access, so this comment is the notification instead. A review when you have a moment would be appreciated 🙏

ksimon1
ksimon1 reviewed Jul 24, 2026
View reviewed changes
Comment thread dev/config/kind/cluster-kubevirt-multinode.yaml Outdated
ksimon1
ksimon1 reviewed Jul 24, 2026
View reviewed changes
Comment thread evals/tasks/kubevirt/helpers/verify-vm.sh Outdated
ksimon1
ksimon1 reviewed Jul 24, 2026
View reviewed changes
Comment thread .github/workflows/mcpchecker.yaml
@lyarwood
lyarwood force-pushed the cnv-87970-hotplug-evals branch from 7a34cc6 to f92ee6e Compare July 24, 2026 13:24
ksimon1
ksimon1 reviewed Jul 28, 2026
View reviewed changes
Comment thread build/kubevirt.mk
ksimon1
ksimon1 reviewed Jul 28, 2026
View reviewed changes
Comment thread .github/workflows/mcpchecker.yaml
@lyarwood
CNV-87970: Add hotplug eval tasks for kubevirt toolset
96518b5 
Add 5 mcpchecker v1alpha2 eval tasks covering hotplug use cases:
- volume add (PVC with hotpluggable: true)
- interface add (Multus secondary NIC)
- CPU socket increase (within maxSockets)
- memory increase (within maxGuest)
- instancetype live-update

These tasks validate whether the LLM can perform hotplug operations
using existing core resource tools (declarative VM spec edits) before
deciding if dedicated hotplug tools are needed.

Also includes:
- Multi-node Kind cluster configuration in dev/config/kind/cluster-kubevirt-multinode.yaml
- LiveUpdate rollout and LiveMigrate update strategy in build/kubevirt.mk
- Updates to mcpchecker workflow for multi-node eval support
- Helper verification functions in evals/tasks/kubevirt/helpers/verify-vm.sh

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
@lyarwood
lyarwood force-pushed the cnv-87970-hotplug-evals branch from f92ee6e to 96518b5 Compare August 14, 2026 12:21
@ksimon1

ksimon1 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@lyarwood can you please rebase this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@manusa manusa Awaiting requested review from manusa manusa is a code owner
@Cali0707 Cali0707 Awaiting requested review from Cali0707 Cali0707 is a code owner
1 more reviewer
@ksimon1 ksimon1 ksimon1 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

toolset/kubevirt

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lyarwood @Cali0707 @ksimon1