bug: Helm uninstall leaves behind scheduler pods due to "kept" SchedulingShard

[googs1025]

➜  dynamo kubectl get pods -nkai-scheduler
NAME                                     READY   STATUS    RESTARTS        AGE
kai-operator-6c7598cd96-zqwp2            1/1     Running   5 (2m14s ago)   13m
kai-scheduler-default-7b9fbfbc97-vsftq   1/1     Running   0               34m
➜  dynamo kubectl get pods -nkai-scheduler
NAME                                     READY   STATUS             RESTARTS        AGE
kai-operator-6c7598cd96-zqwp2            0/1     CrashLoopBackOff   6 (3m48s ago)   23m
kai-scheduler-default-7b9fbfbc97-vsftq   1/1     Running            0               44m
➜  dynamo helm delete -nkai-scheduler kai-scheduler
These resources were kept due to the resource policy:
[Queue] default-parent-queue
[Queue] default-queue
[SchedulingShard] default

release "kai-scheduler" uninstalled
➜  dynamo kubectl get pods -nkai-scheduler
NAME                                     READY   STATUS    RESTARTS   AGE
kai-scheduler-default-7b9fbfbc97-vsftq   1/1     Running   0          44m

the kai-scheduler-default-xxxxx pod continues running, because:

• The SchedulingShard/default CR is not deleted (due to helm.sh/resource-policy: keep)
• The operator (or built-in controller) continues to reconcile it and maintain the Deployment/Pod

This leads to unexpected leftover workloads after helm uninstall, which is confusing

Expected Behavior:
helm delete should fully clean up all scheduler components (including SchedulingShard), unless explicitly opted out.

[enoodle]

Hello @googs1025
I don't think that the helm chart should delete the scheduling shard - It is a default one created for convenience and the users can create more of them, but the helm chart can't delete the CRDs or their instances so it would be weird if only the default scheduling shard is deleted from the cluster.
I think that we do need to have a post delete hook to clean up all the leftover deployments in the namespace. It could be a very wonderful contribution to the project.

[srujanreddya]

Hi @enoodle , Is that fine if I can take this up as a new contributor?

[enoodle]

@srujanreddya That would be great

[srujanreddya]

@enoodle Thanks and please assign the issue to me.

[enoodle]

With @srujanreddya 's PR merged I think that we can close this one.