feat: increase Workload PodSet limit to 10

[yuluo-yx]

What type of PR is this?

/kind feature

Which issue(s) this PR fixes:

Fixes #11379

Does this PR introduce a user-facing change?

Increased the maximum number of PodSets per Workload from 8 to 16. As a result, RayCluster, RayJob, and RayService integrations now allow up to 15 worker groups, since one PodSet is reserved for the Ray head group.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yuluo-yx
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @yuluo-yx. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	39fb219
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6a1112d82b30a50008f5486c
😎 Deploy Preview	https://deploy-preview-11388--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[tenzen-y]

/hold

I'd like to figure out if increasing the number of PodSets degrades TAS performance.
Because we can not revert this change afterward.

[yuluo-yx]

/hold

I'd like to figure out if increasing the number of PodSets degrades TAS performance. Because we can not revert this change afterward.

Agree. 👀

[mimowo]

/ok-to-test

[mimowo]

I'd like to figure out if increasing the number of PodSets degrades TAS performance.
Because we can not revert this change afterward.

The performance of TAS (or general scheduler) does not degrade unless a user requests to have more PodSets. Also, users can scale up Kueue (add more CPU/mem to the kueue deployment), but they cannot workaround validation except for forking Kueue.

Maybe to mitigate the issue we can start with some smaller number like 10 or 12, at the same time to mitigate the additional "technical" PodSets for Ray: #11260 - we want to introduce there a new PodSet for RedisCleanup job and that would mean one less PodSet for "Workers".

wdyt?

[tenzen-y]

I'd like to figure out if increasing the number of PodSets degrades TAS performance.
Because we can not revert this change afterward.

The performance of TAS (or general scheduler) does not degrade unless a user requests to have more PodSets. Also, users can scale up Kueue (add more CPU/mem to the kueue deployment), but they cannot workaround validation except for forking Kueue.

Maybe to mitigate the issue we can start with some smaller number like 10 or 12, at the same time to mitigate the additional "technical" PodSets for Ray: #11260 - we want to introduce there a new PodSet for RedisCleanup job and that would mean one less PodSet for "Workers".

wdyt?

The computation costs are N × M in flavor assignment at worst, where N is the number of flavors and M is the number of PodSets. And in the TAS calculation pays more computation costs because it considers the tree topology structure.

But, I am assuming that 10 has less impact.
Surely, it will degrade TAS performance by increasing the number of pattern matches, but 10 sounds like a reasonable balance between performance and user capabilities.

[mimowo]

@yuluo-yx could you adjust the PR to 10?

[yuluo-yx]

@yuluo-yx could you adjust the PR to 10?

got it, I'll update tonight.

[mimowo]

/retitle feat: increase Workload PodSet limit to 10
As per discussion

[mimowo]

/unhold
as per #11388 (comment)

[k8s-ci-robot]

@yuluo-yx: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-test-e2e-main-1-36	4c65448	link	true	/test pull-kueue-test-e2e-main-1-36

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.