Fix finished Workload GC with finalizers

[ShaanveerS]

What type of PR is this?

/kind bug

What this PR does / why we need it:

When finished Workload retention expires, if the Workload still has the resource-in-use finalizer attached, Kubernetes marks it for deletion but won't actually remove it.

This was already called out as a caveat in KEP-1618:

The deletion mechanism assumes that finished Workload objects do not have any finalizers attached to them. After an investigation, the exact place in the code was found that removes the resource-in-use finalizer when the Workload state transitions to finished. If that behavior ever changes, or if there is an external source of a finalizer being attached to the Workload, it will be marked for deletion by the Kubernetes client but will not actually be deleted until the finalizer is removed.

WorkloadSlices were added after the KEP.
Replaced WorkloadSlices are marked finished by the scheduler in replaceWorkloadSlice, and that path only calls workload.Finish, which sets the finished condition but does not remove the finalizer.

The fix is to remove it in the finished Workload GC path, so the Workload can actually be removed once retention has elapsed.

Which issue(s) this PR fixes:

Fixes #11130

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix a bug where finished Workloads could remain stuck after object retention deletion if they still had Kueue's resource-in-use finalizer.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	2b33d90
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6a0691aa4b2df50008b9ec1f

[ShaanveerS]

the failed test seems another recurrence of #10298

[mimowo]

nit, but let's move the record under if deleteRequested, becasuse the return statements don't differ

[mimowo]

Analogously as it is done here: https://github.com/kubernetes-sigs/kueue/pull/11181/changes#diff-552ab657667de70900b502e2617bcb405fbf4203ff08b2a35d7c87f2d637a75bR976-R980

[ShaanveerS]

done

[mimowo]

/lgtm
/approve
/cherrypick release-0.17
/cherrypick release-0.16
Thank you for fixing the issue, and also commonizing the code to make the code more resilient in the future.

[k8s-ci-robot]

LGTM label has been added.

Details

Git tree hash: ac6f38781f40487b2a44ff932637513762c01874

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mimowo, ShaanveerS

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mimowo]

/cherrypick release-0.17
/cherrypick release-0.16

[k8s-infra-cherrypick-robot]

@mimowo: #11181 failed to apply on top of branch "release-0.16":

Applying: Fix finished Workload GC with finalizers
Using index info to reconstruct a base tree...
M	pkg/controller/core/workload_controller.go
M	pkg/controller/core/workload_controller_test.go
M	pkg/controller/jobframework/reconciler.go
M	pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
M	pkg/workload/workload.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/workload/workload.go
Auto-merging pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
Auto-merging pkg/controller/jobframework/reconciler.go
CONFLICT (content): Merge conflict in pkg/controller/jobframework/reconciler.go
Auto-merging pkg/controller/core/workload_controller_test.go
Auto-merging pkg/controller/core/workload_controller.go
CONFLICT (content): Merge conflict in pkg/controller/core/workload_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Fix finished Workload GC with finalizers

Details

In response to this:

/cherrypick release-0.17
/cherrypick release-0.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@mimowo: #11181 failed to apply on top of branch "release-0.17":

Applying: Fix finished Workload GC with finalizers
Using index info to reconstruct a base tree...
M	pkg/controller/core/workload_controller.go
M	pkg/controller/core/workload_controller_test.go
M	pkg/controller/jobframework/reconciler.go
M	pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
M	pkg/workload/workload.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/workload/workload.go
Auto-merging pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
Auto-merging pkg/controller/jobframework/reconciler.go
Auto-merging pkg/controller/core/workload_controller_test.go
Auto-merging pkg/controller/core/workload_controller.go
CONFLICT (content): Merge conflict in pkg/controller/core/workload_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Fix finished Workload GC with finalizers

Details

In response to this:

/cherrypick release-0.17
/cherrypick release-0.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mimowo]

@ShaanveerS please prepare CPs manually