TaskRuns stuck on deletion after Chains update

[Liu-ko]

Summary

TaskRuns started with Chains v0.24.0 are stuck unable to be deleted after update to Chains v0.27.x. Since Chains v0.27.0 started to use service-side-apply method to manipulate finalizers and chains.tekton.dev was applied using patch-merge method, the operator cannot remove it. As a result, old pre-update started TaskRuns cannot get deleted without human intervention.

Expected Behavior

TaskRuns started on earlier releases of Chains controller than v0.27.x should get deleted successfully when requested.

Actual Behavior

TaskRuns starter on Chains v0.24.0 are stuck and cannot be deleted due to presence of the old chains.tekton.dev finalizer after the update to Chains v0.27.x

Steps to Reproduce the Problem

1. Deploy Chains v0.24.0
2. Start a TaskRun . Check that the TaskRun gets chains.tekton.dev finalizer in its metadata added.
3. Deploy Chains v0.27.x
4. Wait a little and describe the TaskRun. Check that a new finalizer chains.tekton.dev/taskrun is added to the metadata. At this point, there should be 2 finalizers listed on the TaskRun: chains.tekton.dev and chains.tekton.dev/taskrun
5. Run kubectl delete tr <name> to delete the TaskRun. Witness that the deletion is stuck.
6. Describe the stuck TaskRun. It will have only chains.tekton.dev finalizer listed, since the newer one will get removed.

Additional Info

• Kubernetes version:

Client Version: v1.35.6
Kustomize Version: v5.7.1
Server Version: v1.28.0

• Tekton Pipeline version:
  version: v1.12.0

[Liu-ko]

/assign

[enarha]

That was fixed in v0.27.1 https://github.com/tektoncd/chains/releases/tag/v0.27.1. Obviously in main and v0.28.0 as well.
EDIT: Oh, this is from an older version with the old finalizer name. Let me have another look.

[Liu-ko]

That was fixed in v0.27.1 https://github.com/tektoncd/chains/releases/tag/v0.27.1. Obviously in main and v0.28.0 as well. EDIT: Oh, this is from an older version with the old finalizer name. Let me have another look.

yes, it is for older Runs. The v0.27.1 was quite helpful to identify the root cause of this specific issue, but it was not enough to unblock deletion of TaskRuns that were started with Chains v0.24.0 (before the finalizer name was changed as well).

I just assumed that the jump from v0.24.0 to v0.27.x+ should be ok.