KEP-9270: Introduce configuration for the MultiKueue Incremental Dispatcher

keps/9270-multikueue-incremental-step-size/README.md

@@ -0,0 +1,128 @@
+
+# KEP-9270: MultiKueue Incremental Dispatcher Step Size
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API Changes](#api-changes)
+  - [Execution Loop](#execution-loop)
+  - [Test Plan](#test-plan)
+    - [Prerequisite testing updates](#prerequisite-testing-updates)
+    - [Unit tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+    - [e2e tests](#e2e-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+<!-- /toc -->
+
+## Summary
+
+This proposal introduces a new configuration field called `stepSize` to the MultiKueue Incremental Dispatcher. This setting will control the number of worker clusters that the Manager Cluster will attempt to dispatch a workload to concurrently.
+
+## Motivation
+
+Currently, the MultiKueue Incremental Dispatcher has a hardcoded batch size `batchSize := 3` when attempting to schedule a workload across multiple worker clusters. The primary limitation is that this value is not configurable. For administrators managing a massive fleet of worker clusters, a fixed batch size of 3 is too slow and delays job execution. Conversely, if administrators were to theoretically increase this to query all clusters simultaneously to speed things up, it would generate heavy API traffic and increase the risk of race conditions. By making this step size configurable, administrators can find the optimal balance between dispatch speed and Manager Cluster resource usage for their specific environment.
+Introducing a `stepSize` allows cluster administrators to configure a "batch size" for this process, finding the perfect balance between speed and system stability.
+
+### Goals
+
+* Add a `stepSize` parameter to the `MultiKueueConfig` API.
+* Update the Workload Controller's dispatching loop to process worker clusters in batches defined by the `stepSize`.
+
+### Non-Goals
+
+* We will not change the way worker clusters are scored or prioritized.
+* We will not alter the `AllAtOnce` dispatcher behavior, as this proposal strictly targets the Incremental Dispatcher.
+
+## Proposal
+
+We propose updating the `MultiKueueConfig` API to accept an optional integer for the step size, and updating the Workload Controller to respect this batch limit when iterating through the list of available worker clusters.
+
+
+### Risks and Mitigations
+
+**Risk:** An administrator might set the `stepSize` to an extremely high number (e.g., equal to the total number of clusters), which effectively recreates the heavy load issues of the `AllAtOnce` dispatcher and increases the chance of race conditions where multiple clusters admit the job at the exact same time.
+
+**Mitigation:** Provide clear API documentation regarding best practices. The Workload Controller's conflict resolution logic will naturally handle race conditions, just as it already does for the `AllAtOnce` strategy.
+
+## Design Details
+
+### API Changes
+
+We will update the global `Configuration` API (specifically the `MultiKueue` struct) to include the new field, rather than placing it in `MultiKueueConfigSpec`. This keeps the step size configuration co-located with the `DispatcherName` selection.
+
+The field will be named `IncrementalDispatcherStepSize` for clarity. To ensure backwards compatibility and preserve existing system behavior, the default value will be `3`.
+
+We will also use Kubebuilder validation markers (e.g., CEL rules) to enforce that this field can only be populated if DispatcherName is set to the incremental dispatcher.
+
+```go
+type MultiKueue struct {
+    // Other existing fields...
+    // DispatcherName defines the type of dispatcher to use.
+    DispatcherName *string `json:"dispatcherName,omitempty"`
+
+    // IncrementalDispatcherStepSize defines the number of worker clusters the Incremental 
+    // Dispatcher will query simultaneously. 
+    // This field is only valid when DispatcherName is set to the incremental dispatcher.
+	// Note: This field is going to be ignored when the MultiKueueStepSize feature gate is disabled.
+    // Minimum value is 1. If not set, it defaults to 3.
+    // +optional
+    // +kubebuilder:default=3
+    // +kubebuilder:validation:Minimum=1
+    IncrementalDispatcherStepSize *int32 `json:"incrementalDispatcherStepSize,omitempty"`
+}
+```
+
+### Execution Loop
+The core logic change is minimal because the batching mechanism is already implemented in the Incremental Dispatcher (`pkg/controller/workloaddispatcher/incrementaldispatcher.go`).
+
+The update will simply modify the existing logic to:
+1. Read the `stepSize` value from the `MultiKueueConfig`.
+2. Replace the hardcoded `batchSize := 3` variable with this dynamically configured `stepSize` parameter.
+
+The existing loop and conflict resolution mechanisms will naturally handle the rest of the execution based on this new size, meaning no major architectural changes are required to the dispatching loop itself.
+
+### Test Plan
+
+ I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.
+
+#### Prerequisite testing updates
+
+No major prerequisite testing updates are required. Current mock worker cluster frameworks in EnvTest are sufficient.
+
+#### Unit tests
+
+- `pkg/controller/core/workload_controller_test.go`: Verify that the worker cluster list is correctly divided into chunks matching the `stepSize`.
+- `pkg/controller/core/workload_controller_test.go`: Verify that the default value remains `3` if the user does not specify a `stepSize`.
+
+#### Integration tests
+
+- `TestIncrementalDispatcherStepSize`: Simulate a Manager Cluster with 5 Worker Clusters. Set `stepSize` to `2`. Submit a workload and verify via the API server that exactly 2 proxy workloads are created in the first step. Simulate a rejection on the first 2 clusters, and verify that the controller correctly creates the next 2 proxy workloads in the second step.
+
+#### e2e tests
+
+Standard E2E multi-cluster tests will be updated to include an iteration where `MultiKueueConfig` is deployed with a `stepSize > 1` to ensure workloads execute successfully end-to-end without duplication.
+
+### Graduation Criteria
+
+* **Alpha:**
+    * API fields added.
+    * Workload Controller updated to support batching.
+    * Unit tests and EnvTest integration tests passing.
+* **Beta:** * Gather feedback from the community.
+    * E2E tests added and passing consistently.
+* **Stable:**
+    * Feature has been in Beta for at least one release cycle with no major bugs reported.
+
+## Implementation History
+
+- 2026-05-01: KEP proposed and initial draft created.
+
+## Drawbacks
+
+This adds slight code complexity to the existing Incremental Dispatcher loop and introduces one additional configuration parameter for administrators to manage and tune.

keps/9270-multikueue-incremental-step-size/kep.yaml

@@ -0,0 +1,32 @@
+title: MultiKueue Incremental Dispatcher Step Size
+kep-number: 9270
+authors:
+  - "@Mostafahassen1"
+status: provisional
+creation-date: "2026-05-01"
+reviewers:
+  - olekzabl
+approvers:
+  - TBD
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. Kueue typically uses 0.x versioning.
+latest-milestone: "v0.17"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+  alpha: "v0.19"
+  beta: "v0.20"
+  stable: "v0.21"
+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+  - name: MultiKueueStepSize
+    components:
+      - kueue-controller-manager
+disable-supported: true
+