Exclude kube-system from webhooks targeting common resources by dkaluza · Pull Request #11192 · kubernetes-sigs/kueue

dkaluza · 2026-05-14T14:16:06Z

What type of PR is this?

/kind feature
/area integrations

What this PR does / why we need it:

Adds kube-system and kueue-system (or other release namespace in case of helm installation) namespace excludes to webhooks targeting common resource types.
(Custom resources that are not added by Kueue)

This makes sure Kueue deployment does not suspend crucial system components, e.g. jobs run by kubeadm.

Which issue(s) this PR fixes:

Fixes #11006

Special notes for your reviewer:

I haven't modified webhook selectors for Kueue's own resources as those shouldn't be run by other crucial system components. But if you think it is worth to also exclude those, just let me know.

Prepared with Antigravity, but carefully reviewed to not miss any resource.

Does this PR introduce a user-facing change?

Kueue integration webhooks now consistently exclude `kube-system` and the Kueue
installation namespace. For manifest-based installs this is `kueue-system`; for Helm
installs this is the Helm release namespace.

Previously, Pod, Deployment, and StatefulSet integrations already excluded these
namespaces, while other workload integrations did not.

ACTION REQUIRED: If you run Kueue-managed workloads in `kube-system` or in the namespace
where Kueue is installed, move them to another namespace before upgrading.

If this is not possible, update `managedJobsNamespaceSelector` and the relevant webhook
`namespaceSelector`s to include those namespaces. This setup is discouraged, as it may
affect system workloads and disrupt cluster upgrades.

…e resources

netlify · 2026-05-14T14:16:13Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`3bf8e90`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6a0b3182ef6feb00084c6a40

k8s-ci-robot · 2026-05-14T14:16:17Z

Hi @dkaluza. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mbobrovskyi · 2026-05-14T14:21:05Z

/ok-to-test

dkaluza · 2026-05-14T14:20:51Z

            {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") }}
-            namespaceSelector:
              {{- toYaml $managerConfig.managedJobsNamespaceSelector | nindent 6 }}
+            {{- else }}


I do not like the fact that we have to modify 200+ lines to make such small change, but it looks like we want to be really explicit about the changes here, so didn't want to change this.

dkaluza · 2026-05-14T14:22:58Z

+        - webhooks.[name=mstatefulset.kb.io].namespaceSelector
+        - webhooks.[name=mtfjob.kb.io].namespaceSelector
+        - webhooks.[name=mtrainjob.kb.io].namespaceSelector
+        - webhooks.[name=mxgboostjob.kb.io].namespaceSelector


This unfortunately still requires registration of new webhook here when there is a new integration.

This is not the case if we just do * but then we will also target Kueue crds... This can be solved by splitting those in some way, but I'm not convinced this is worth it right now.

Cool, do I understand that this change only then impacts all workloads other than Pod, Deployment, StatefulSet? Because Pod, Deployment, StatefulSet already had the exclusions in the deleted code here:

- patch: |- apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: mutating-webhook-configuration webhooks: - name: mpod.kb.io namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - kueue-system - name: mdeployment.kb.io namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - kueue-system - name: mstatefulset.kb.io namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - kueue-system

That's correct.

….yaml Co-authored-by: Olek Zabłocki <olekz@google.com>

mimowo · 2026-05-15T14:50:09Z

/test pull-kueue-test-e2e-multikueue-main
/test pull-kueue-test-e2e-main-1-34
look like infra, we are fixing currently similar issues by retry: #11238 (comment)

mimowo · 2026-05-15T14:52:10Z

          namespaceSelector:
-            {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") -}}
-              {{- toYaml $managerConfig.managedJobsNamespaceSelector | nindent 6 -}}
+            {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") }}


this par looks duplicated for all crds:

namespaceSelector: {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") -}} {{- toYaml $managerConfig.managedJobsNamespaceSelector | nindent 6 -}} {{- else }} matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - '{{ .Release.Namespace }}' {{- end }}

i'm wondering if there would be a way to introduce some kind of placeholder for that in processing-plain.yaml, wdyt?

This is not a blocker, but it would be good to research this topic on this occasion so that we can open follow up issue to maybe introduce that capability if not existing currently.

Would named templates help?

I believe it should be generally doable with just a better yq selector(condition here), no need for named template yet, but generally I agree named templates also look like a viable way to achieve deduplication in similar cases.

I got some strange behavior of our yaml processor with new condition(not pushed yet), I will investigate on Monday.

Done, needed a small hotfix to make this feasible, added a test for that.

olekzabl · 2026-05-15T21:05:18Z

+        - webhooks.[name=vleaderworkerset.kb.io].namespaceSelector
+        - webhooks.[name=vmpijob.kb.io].namespaceSelector
+        - webhooks.[name=vpaddlejob.kb.io].namespaceSelector
+        - webhooks.[name=vpod.kb.io].namespaceSelector


FYI this target path is exactly the source path.

An AI bot told me it'll still work, even with create: true (just silently ignoring this target).
So maybe this is desired, for the completeness of the target list.
Or maybe it could be considered confusing/misleading.

I'm leaving the choice to you; I just wanted to make sure you're aware of this.

Thanks for pointing this out.

Generally I consider this beneficial, as it provides mentioned completeness. In this way the number of webhooks in both selects is the same. Also they can be easier matched against expected number, you do not have to always remember this special case.

An AI bot told me it'll still work, even with create: true (just silently ignoring this target).

According to kustomize docs create: true just means to create the target path if it does not exists instead of throwing an error.
Generally this is a replacement of the same content with the same content, so I wouldn't expect any problems here.

olekzabl · 2026-05-15T21:05:46Z

/lgtm

k8s-ci-robot · 2026-05-15T21:05:54Z

LGTM label has been added.

Details

Git tree hash: 56e78ee8647edbf06877089eb5a2a3c0437e523c

mimowo · 2026-05-18T08:13:02Z

/release-note-edit

Kueue integration webhooks will no longer target kube-system or kueue-system namespace.

ACTION REQUIRED: If you are running Kueue-managed workloads in kube-system namespace or
namespace in which  Kueue is installed please move those workloads to a different namespace. 

If  this is not feasible then you may need to adjust your `managedJobsNamespaceSelector` and
webhook configuration to also cover these namespaces. However such a deployment is
discouraged as may result in disruptions during cluster upgrades.

mimowo · 2026-05-18T08:17:02Z

        path: /mutate-workload-codeflare-dev-v1beta2-appwrapper
    failurePolicy: Fail
    name: mappwrapper.kb.io
-    {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") }}


qq: does this change:

{{- if (hasKey $managerConfig "managedJobsNamespaceSelector") }} namespaceSelector: {{- toYaml $managerConfig.managedJobsNamespaceSelector | nindent 6 }} {{- end }}

to

namespaceSelector: {{- if (hasKey $managerConfig "managedJobsNamespaceSelector") }} {{- toYaml $managerConfig.managedJobsNamespaceSelector | nindent 6 }} {{- else }} matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: - kube-system - '{{ .Release.Namespace }}' {{- end }}

impact the default configuration by Helm? IIUC it does because by default we would keep managedJobsNamespaceSelector as nil, and so by default Helm would cover kueue-system and kube-system, but I want to double check.

Right now default helm deployment has namespace excludes only for the mentioned:

statefulsets

pods

deployments

After this PR the namespaces will be excluded for all of the workload source types by default . Which makes those consistent across types and aligned with the release manifests.

Cool, let's capture that in the release note

…r in processing plan

dkaluza · 2026-05-18T14:35:00Z

 		buffer.WriteString(line + "\n")

-		if slices.Contains(keyLines, i+offset) {
+		if slices.Contains(keyLines, i) {


As keyLines are from the original yaml, we shouldn't be skipping lines if we add a new snippet.

mimowo · 2026-05-18T17:10:19Z

/release-note-edit

Kueue integration webhooks now consistently exclude `kube-system` and the Kueue
installation namespace. For manifest-based installs this is `kueue-system`; for Helm
installs this is the Helm release namespace.

Previously, Pod, Deployment, and StatefulSet integrations already excluded these
namespaces, while other workload integrations did not.

ACTION REQUIRED: If you run Kueue-managed workloads in `kube-system` or in the namespace
where Kueue is installed, move them to another namespace before upgrading.

If this is not possible, update `managedJobsNamespaceSelector` and the relevant webhook
`namespaceSelector`s to include those namespaces. This setup is discouraged, as it may
affect system workloads and disrupt cluster upgrades.

mimowo · 2026-05-18T17:12:35Z

/lgtm
/approve
Thank you, nice to see closing the inconsistencies between configuration for different Jobs 👍 I'm even wondering if we should cherrypick this as a consistency bug.

Any opinion @tenzen-y ?

k8s-ci-robot · 2026-05-18T17:12:44Z

LGTM label has been added.

Details

Git tree hash: e3a7c6d93ff7bd83f4bd9fcb5369d24c3c3adb80

k8s-ci-robot · 2026-05-18T17:12:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkaluza, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2026-05-18T18:28:01Z

Thank you, nice to see closing the inconsistencies between configuration for different Jobs 👍 I'm even wondering if we should cherrypick this as a consistency bug.
Any opinion @tenzen-y ?

I think we can skip the cherrypick as of now because no user requested the change as a bugfix for now. We will reconsider when this is needed to CP another bugfix, or if a user requests.

dkaluza added 4 commits May 14, 2026 12:37

Add kube-system namespace exlusions to all webhooks targeting job-lik…

6114825

…e resources

Revert unnecessary change of quotes

91c4f2d

Fix the chart and the manifests kustomize

3aeaf4a

Remove no longer relevant comment

3aaff02

k8s-ci-robot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. kind/feature Categorizes issue or PR as related to a new feature. area/integrations Workload integrations labels May 14, 2026

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 14, 2026

k8s-ci-robot requested review from olekzabl and sohankunkerkar May 14, 2026 14:16

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label May 14, 2026

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 14, 2026

dkaluza commented May 14, 2026

View reviewed changes

dkaluza changed the title ~~Webhooks exclude kube system~~ Exclude kube-system from webhooks targeting common resources May 14, 2026

olekzabl reviewed May 15, 2026

View reviewed changes

Comment thread config/components/webhook/kustomization.yaml Outdated

Remove unnecessary space from config/components/webhook/kustomization…

d921226

….yaml Co-authored-by: Olek Zabłocki <olekz@google.com>

mimowo reviewed May 15, 2026

View reviewed changes

olekzabl reviewed May 15, 2026

View reviewed changes

k8s-ci-robot assigned olekzabl May 15, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2026

mimowo reviewed May 18, 2026

View reviewed changes

Add condition matching multiple entries in array for namespaceSelecto…

5cb85d1

…r in processing plan

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 18, 2026

k8s-ci-robot requested a review from olekzabl May 18, 2026 14:28

dkaluza commented May 18, 2026

View reviewed changes

dkaluza added 2 commits May 18, 2026 14:36

Move inserts in processing plan to reduce diff

345f2d5

Fix formatting

3bf8e90

k8s-ci-robot assigned mimowo May 18, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 18, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 18, 2026

k8s-ci-robot merged commit 03754ac into kubernetes-sigs:main May 18, 2026
37 checks passed

k8s-ci-robot added this to the v0.18 milestone May 18, 2026

Conversation

dkaluza commented May 14, 2026 • edited by k8s-ci-robot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

netlify Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

k8s-ci-robot commented May 14, 2026

Uh oh!

mbobrovskyi commented May 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mimowo commented May 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkaluza May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olekzabl commented May 15, 2026

Uh oh!

k8s-ci-robot commented May 15, 2026

Uh oh!

mimowo commented May 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkaluza May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo commented May 18, 2026

Uh oh!

mimowo commented May 18, 2026

Uh oh!

k8s-ci-robot commented May 18, 2026

Uh oh!

k8s-ci-robot commented May 18, 2026

Uh oh!

Uh oh!

mimowo commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dkaluza commented May 14, 2026 •

edited by k8s-ci-robot

Loading

netlify Bot commented May 14, 2026 •

edited

Loading

dkaluza May 15, 2026 •

edited

Loading

dkaluza May 18, 2026 •

edited

Loading