KEP-9270: Introduce configuration for the MultiKueue Incremental Dispatcher

[Mostafahassen1]

What type of PR is this?

/kind kep
/area multikueue

What this PR does / why we need it:

This PR proposes a KEP to introduce the stepSize configuration parameter for the MultiKueue Incremental Dispatcher.

Currently, the Incremental Dispatcher can either query worker clusters one-by-one (which is slow) or all-at-once (which creates a heavy API load and risks race conditions). This KEP outlines a design to allow administrators to batch these queries by specifying a stepSize, finding the optimal balance between dispatch speed and system stability.

Which issue(s) this PR fixes:

Fixes #9270

Special notes for your reviewer:

This PR contains the initial KEP document (README.md and kep.yaml) for the feature proposed in #9270.

Once this KEP is reviewed and approved (status: implementable), I will open follow-up PRs containing the API additions to MultiKueueConfig and the core logic updates to the Workload Controller.

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	ece1621
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/69fe6800f259fa000845ddc5

[Mostafahassen1]

CC / @mimowo , @olekzabl

[olekzabl]

Hi, thank you for your PR!

This is my first round of comments; I plan to come back soon.

[olekzabl]

1. Once the plan gets approved (which I hope to happen soon), we'll probably need to update the "main" KEP for MultiKueue, KEP-693. (Search for "incremental" there, and you'll see).
   OTOH, AFAIU, the good practice is to keep such old KEPs universally valid for all versions - which is achieved by saying "until v..." or "since v..." (for inspiration, search for "since v" in KEP-693, or e.g. in KEP-2724).

2. Given the above - my apologies for coming to this thought so late! - I'm now not sure if this new KEP should even exist.
   So far, this feature feels quite simple, and not involving any major controversy. So maybe just updating KEP-693 could be enough.
   OTOH, Kueue already has some tiny KEPs (e.g. 2936, or 3122). So perhaps having this new one won't hurt :)

@mimowo You know the customs better; what would you say?
Should KEP-693 be updated? Should it happen in this PR? Should we still keep this new KEP?

(P.S. I think a separate KEP makes much sense if we want to have a new dedicated feature gate - but do we? see below)

[Mostafahassen1]

@olekzabl

Given the above - my apologies for coming to this thought so late! - I'm now not sure if this new KEP should even exist.
So far, this feature feels quite simple,

No problem at all! I’m completely open to following the Kueue standards here. If you think it’s better to update KEP-693 instead of having a new one

[olekzabl]

@olekzabl

Given the above - my apologies for coming to this thought so late! - I'm now not sure if this new KEP should even exist.
So far, this feature feels quite simple,

No problem at all! I’m completely open to following the Kueue standards here. If you think it’s better to update KEP-693 instead of having a new one

I'm sorry, I'm puzzled by this question (i.e. "update old or write new or both").
I'm increasingly feeling that the answer depends on whether we want to have a dedicated feature gate for this - which is also not obvious. (After all, your only API change is introducing a new field, which does not change much in the implementation when set, and should change nothing in it when unset).

For me, right now, this feels a delicate judgement call... so for now I'd keep it as is and wait to see what the other reviewers will say.

[olekzabl]

LGTM; I just have one more round of tiny comments.

Now I'm feeling we're just blocked by this thread.

[tenzen-y]

/release-note-none

[mimowo]

/retitle KEP-9270: Introduce configuration for the MultiKueue Incremental Dispatcher

[olekzabl]

LGTM!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Mostafahassen1, olekzabl
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[olekzabl]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Details

Git tree hash: 4d86aed56a9e407dda87bb843a6dcc364bf223f4