From 74345386420ef40ef39b85c0f02449c5abab4195 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Wed, 13 May 2026 14:10:09 -0400 Subject: [PATCH] doc: reorganize test documentation --- .../docs/contribution_guidelines/testing.md | 233 +++++++++----- .../docs/contribution_guidelines/testing.md | 301 +++++++++++++----- 2 files changed, 373 insertions(+), 161 deletions(-) diff --git a/site/content/en/docs/contribution_guidelines/testing.md b/site/content/en/docs/contribution_guidelines/testing.md index 98c6fdcd701..45459d62f88 100644 --- a/site/content/en/docs/contribution_guidelines/testing.md +++ b/site/content/en/docs/contribution_guidelines/testing.md @@ -6,12 +6,50 @@ description: > Running and debugging tests --- -## Running presubmission verification tests +## Quick start + +The most common development workflows: + ```shell -make verify +# Unit tests +make test + +# Integration tests +make test-integration + +# E2E: build image and run +# Note: PLATFORM=linux/arm64 is only needed on Apple Silicon (arm64). +# On amd64, run: make kind-image-build test-e2e-baseline +PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline + +# E2E: build, run, and keep the kind cluster alive between runs +E2E_MODE=dev PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline +``` + +### Quick filter reference + +```shell +# Focus on tests matching a name pattern +GINKGO_ARGS="--focus=Scheduler" make test-integration +GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline + +# Filter integration tests by label +INTEGRATION_FILTERS="--label-filter=controller:workload" make test-integration +INTEGRATION_FILTERS="--label-filter=area:jobs" make test-integration + +# Filter e2e tests by label +GINKGO_ARGS="--label-filter=feature:jobset" make test-e2e-extended + +# Run only a specific integration test directory +INTEGRATION_TARGET='test/integration/singlecluster/scheduler' make test-integration ``` +For the full label taxonomy and more examples, see [Running a subset of tests](#running-subset-of-integration-or-e2e-tests). + +--- + ## Running unit tests + To run all unit tests: ```shell make test @@ -53,78 +91,42 @@ make test-integration For running a subset of tests, see [Running subset of tests](#running-subset-of-integration-or-e2e-tests). -## Running e2e tests using custom build +## Running e2e tests + +E2E tests build a Kueue image and load it into a local Kind cluster. The build step must run before the test target: + ```shell make kind-image-build -make test-tas-e2e-baseline -make test-tas-e2e-extended +``` + +On Apple Silicon (arm64), set `PLATFORM`: +```shell +PLATFORM=linux/arm64 make kind-image-build +``` + +Then run the desired test target: +```shell make test-e2e-baseline make test-e2e-extended -make test-e2e-sequential-extended make test-e2e-sequential-baseline +make test-e2e-sequential-extended make test-e2e-certmanager make test-e2e-kueueviz +make test-tas-e2e-baseline +make test-tas-e2e-extended make test-multikueue-e2e-main make test-multikueue-e2e-sequential ``` -You can specify the Kubernetes version used for running the e2e tests by setting the `E2E_K8S_FULL_VERSION` variable: +You can specify the Kubernetes version: ```shell E2E_K8S_FULL_VERSION=1.35.0 make test-e2e-baseline ``` For running a subset of tests, see [Running subset of tests](#running-subset-of-integration-or-e2e-tests). -## Increase logging verbosity -`TEST_LOG_LEVEL` controls test logging uniformly for all targets: - -- `go test`, `make test` (unit tests) -- `make test-integration` (integration tests) -- `make test-*-e2e` (e2e tests) +### DEV mode (keep the cluster) -Use more negative values for more verbose logs and higher (positive) values for quieter logs. For example: -```shell -TEST_LOG_LEVEL=-5 make test-integration # more verbose -TEST_LOG_LEVEL=-1 make test # less verbose than default -``` -Default is `TEST_LOG_LEVEL=-3`. - -## Debug tests in VSCode -It is possible to debug unit and integration tests in VSCode. -You need to have the [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed. -Now you will have `run test | debug test` text buttons above lines like -```go -func TestValidateClusterQueue(t *testing.T) { -``` -You can click on the `debug test` to debug a specific test. - -For integration tests, an additional step is needed. In settings.json, you need to add two variables inside `go.testEnvVars`: -- Run `ENVTEST_K8S_VERSION=1.35 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` and assign the path to the `KUBEBUILDER_ASSETS` variable -- Set `KUEUE_BIN` to the `bin` directory within your cloned Kueue repository -```json -"go.testEnvVars": { - "KUBEBUILDER_ASSETS": "", - "KUEUE_BIN": "/bin", - }, -``` - -For e2e tests, you can also use [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer). You need to add the following variables to settings.json: -```json - "ginkgotestexplorer.testEnvVars": { - "KIND_CLUSTER_NAME": "kind", - "WORKER1_KIND_CLUSTER_NAME": "kind-worker1", - "MANAGER_KIND_CLUSTER_NAME": "kind-manager", - "WORKER2_KIND_CLUSTER_NAME": "kind-worker2", - "KIND": "/bin/kind", - }, -``` -and then you can use GUI of the Ginkgo Test Explorer to run individual tests, provided you started kind clanter (see [here](#attaching-e2e-tests-to-an-existing-kind-cluster) for the instructions). - -## Attaching e2e tests to an existing kind cluster -You can use the following approach to start up a kind cluster and then run e2e tests from commandline or VSCode, -attaching them to the existing cluster. For example, suppose you want to test some of the multikueue-e2e tests. - -### DEV mode (recommended) Use `E2E_MODE=dev` to create-or-reuse a kind cluster, rebuild/redeploy Kueue, run tests, and keep the cluster running for fast reruns and post-test investigation: ```shell @@ -135,8 +137,7 @@ E2E_MODE=dev make kind-image-build test-e2e-baseline E2E_MODE=dev make kind-image-build test-multikueue-e2e-main # Loop a suite (until it fails) while keeping the cluster -E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline - +E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline # Skip reinstallation of kueue (works only in dev mode) E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-e2e-baseline @@ -146,24 +147,6 @@ E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-multikueue-e2e-m E2E_MODE=dev E2E_SKIP_IMAGE_RELOAD=true make kind-image-build test-e2e-baseline ``` -To use a **released** or **staging** Kueue image instead of building from source (no `kind-image-build` needed), pass `IMAGE_TAG` with the desired image: - -```shell -# Released version -E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline -E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-multikueue-e2e-main - -# Staging image (e.g. from a PR or nightly) -E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:main make test-e2e-baseline -``` - -**Using a released version with matching manifests:** The e2e framework deploys CRDs and other resources from the repo's config and overrides only the controller image via `IMAGE_TAG`. To run e2e against a specific release with manifests that match that image: - -1. Check out that version's tag (e.g. `git checkout v0.16.0`). The CRD and deployment config in the repo are committed at each release, so no `make manifests` step is needed. -2. Run the command above with the same image tag, e.g. `E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline`. - -This is useful to reproduce issues on a specific released version (e.g. for on-call debugging). For installing a released version into a real cluster (not e2e), see [Install a released version](/docs/installation/#install-a-released-version). - {{% alert title="Note" color="primary" %}} When reusing a kept cluster in `E2E_MODE=dev`, external operators (MPI, KubeRay, etc.) are installed only once. To force re-installing them on every run, set `E2E_ENFORCE_OPERATOR_UPDATE=true`. @@ -187,24 +170,35 @@ To delete the kept cluster(s) afterwards: kind delete clusters kind kind-manager kind-worker1 kind-worker2 ``` -### Legacy: interactive attach mode -Run `E2E_RUN_ONLY_ENV=true make kind-image-build test-multikueue-e2e-main` and wait for the `Do you want to cleanup? [Y/n] ` to appear (CI-style behavior). +### Using a released or staging image + +To use a **released** or **staging** Kueue image instead of building from source (no `kind-image-build` needed), pass `IMAGE_TAG`: -The cluster is ready, and now you can run tests from another terminal: ```shell -/bin/ginkgo --json-report ./ginkgo.report -focus "MultiKueue when Creating a multikueue admission check Should run a jobSet on worker if admitted" -r +# Released version +E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline +E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-multikueue-e2e-main + +# Staging image (e.g. from a PR or nightly) +E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:main make test-e2e-baseline ``` -or from VSCode. -## Debugging metrics with Prometheus +**Using a released version with matching manifests:** The e2e framework deploys CRDs and other resources from the repo's config and overrides only the controller image via `IMAGE_TAG`. To run e2e against a specific release with manifests that match that image: -To provision a Kind cluster with Prometheus pre-configured for metrics debugging: +1. Check out that version's tag (e.g. `git checkout v0.16.0`). The CRD and deployment config in the repo are committed at each release, so no `make manifests` step is needed. +2. Run the command above with the same image tag, e.g. `E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline`. + +This is useful to reproduce issues on a specific released version (e.g. for on-call debugging). For installing a released version into a real cluster (not e2e), see [Install a released version](/docs/installation/#install-a-released-version). + +### Legacy: interactive attach mode + +Run `E2E_RUN_ONLY_ENV=true make kind-image-build test-multikueue-e2e-main` and wait for the `Do you want to cleanup? [Y/n] ` to appear (CI-style behavior). +The cluster is ready, and now you can run tests from another terminal: ```shell -E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e-baseline +/bin/ginkgo --json-report ./ginkgo.report -focus "MultiKueue when Creating a multikueue admission check Should run a jobSet on worker if admitted" -r ``` - -For more details, see [Setup Dev Monitoring](/docs/tasks/dev/setup_dev_monitoring). +or from VSCode. ## Running subset of integration or e2e tests @@ -252,7 +246,6 @@ SingleCluster tests are labeled by feature and area. You can use `GINKGO_ARGS` w # Run only appwrapper tests GINKGO_ARGS="--label-filter=feature:appwrapper" make test-e2e-extended - # Run only deployment tests with helm GINKGO_ARGS="--label-filter=feature:deployment" make test-e2e-baseline-helm @@ -283,6 +276,7 @@ GINKGO_ARGS="--label-filter=feature:spark" make test-e2e-sequential-extended GINKGO_ARGS="--focus=Scheduler" make test-integration GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline ``` + ### Use ginkgo.FIt If you want to focus on specific tests, you can change `ginkgo.It` to `ginkgo.FIt` for these tests. @@ -333,7 +327,70 @@ For each log message you can from which file and line the message is coming from ```log 2025-02-03T15:51:51.502425029Z stderr F 2025-02-03T15:51:51.502117824Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:341 ClusterQueue update event {"clusterQueue": {"name":"cluster-queue"}} ``` -Here, it's `core/clusterqueue_controller.go:341`. + +--- + +## Advanced + +### Running presubmission verification tests +```shell +make verify +``` + +### Increase logging verbosity +`TEST_LOG_LEVEL` controls test logging uniformly for all targets: + +- `go test`, `make test` (unit tests) +- `make test-integration` (integration tests) +- `make test-*-e2e` (e2e tests) + +Use more negative values for more verbose logs and higher (positive) values for quieter logs. For example: +```shell +TEST_LOG_LEVEL=-5 make test-integration # more verbose +TEST_LOG_LEVEL=-1 make test # less verbose than default +``` +Default is `TEST_LOG_LEVEL=-3`. + +### Debug tests in VSCode +It is possible to debug unit and integration tests in VSCode. +You need to have the [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed. +Now you will have `run test | debug test` text buttons above lines like +```go +func TestValidateClusterQueue(t *testing.T) { +``` +You can click on the `debug test` to debug a specific test. + +For integration tests, an additional step is needed. In settings.json, you need to add two variables inside `go.testEnvVars`: +- Run `ENVTEST_K8S_VERSION=1.35 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` and assign the path to the `KUBEBUILDER_ASSETS` variable +- Set `KUEUE_BIN` to the `bin` directory within your cloned Kueue repository +```json +"go.testEnvVars": { + "KUBEBUILDER_ASSETS": "", + "KUEUE_BIN": "/bin", + }, +``` + +For e2e tests, you can also use [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer). You need to add the following variables to settings.json: +```json + "ginkgotestexplorer.testEnvVars": { + "KIND_CLUSTER_NAME": "kind", + "WORKER1_KIND_CLUSTER_NAME": "kind-worker1", + "MANAGER_KIND_CLUSTER_NAME": "kind-manager", + "WORKER2_KIND_CLUSTER_NAME": "kind-worker2", + "KIND": "/bin/kind", + }, +``` +and then you can use GUI of the Ginkgo Test Explorer to run individual tests, provided you started kind cluster (see [here](#legacy-interactive-attach-mode) for the instructions). + +### Debugging metrics with Prometheus + +To provision a Kind cluster with Prometheus pre-configured for metrics debugging: + +```shell +E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e-baseline +``` + +For more details, see [Setup Dev Monitoring](/docs/tasks/dev/setup_dev_monitoring). ### See also - [Kubernetes testing guide](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/testing.md) diff --git a/site/content/zh-CN/docs/contribution_guidelines/testing.md b/site/content/zh-CN/docs/contribution_guidelines/testing.md index e8d98cd4e8c..62c979ec378 100644 --- a/site/content/zh-CN/docs/contribution_guidelines/testing.md +++ b/site/content/zh-CN/docs/contribution_guidelines/testing.md @@ -6,12 +6,48 @@ description: > 运行和调试测试 --- -## 运行预提交验证测试 {#running-presubmission-verification-tests} +## 快速入门 {#quick-start} + +最常用的开发测试流程： + ```shell -make verify +# 单元测试 +make test + +# 集成测试 +make test-integration + +# E2E 测试：构建镜像并运行（Apple Silicon / arm64——amd64 无需设置 PLATFORM） +PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline + +# E2E 测试：构建、运行，并在两次运行之间保留 kind 集群 +E2E_MODE=dev PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline ``` +### 快速过滤参考 {#quick-filter-reference} + +```shell +# 按名称模式聚焦测试 +GINKGO_ARGS="--focus=Scheduler" make test-integration +GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline + +# 按标签过滤集成测试 +INTEGRATION_FILTERS="--label-filter=controller:workload" make test-integration +INTEGRATION_FILTERS="--label-filter=area:jobs" make test-integration + +# 按标签过滤 e2e 测试 +GINKGO_ARGS="--label-filter=feature:jobset" make test-e2e-extended + +# 只运行特定集成测试目录 +INTEGRATION_TARGET='test/integration/singlecluster/scheduler' make test-integration +``` + +完整的标签分类和更多示例，请参阅[运行测试子集](#running-subset-of-integration-or-e2e-tests)。 + +--- + ## 运行单元测试 {#running-unit-tests} + 运行所有单元测试： ```shell make test @@ -53,78 +89,42 @@ make test-integration 关于运行测试子集，请参阅[运行测试子集](#running-subset-of-integration-or-e2e-tests)。 -## 使用自定义构建运行 e2e 测试 {#running-e2e-tests-using-custom-build} +## 运行 e2e 测试 {#running-e2e-tests} + +e2e 测试需要构建 Kueue 镜像并将其加载到本地 Kind 集群中。首先执行构建步骤： + ```shell make kind-image-build -make test-tas-e2e-baseline -make test-tas-e2e-extended +``` + +在 Apple Silicon（arm64）上，需设置 `PLATFORM`： +```shell +PLATFORM=linux/arm64 make kind-image-build +``` + +然后运行所需的测试目标： +```shell make test-e2e-baseline make test-e2e-extended -make test-e2e-sequential-extended make test-e2e-sequential-baseline +make test-e2e-sequential-extended make test-e2e-certmanager make test-e2e-kueueviz +make test-tas-e2e-baseline +make test-tas-e2e-extended make test-multikueue-e2e-main make test-multikueue-e2e-sequential ``` 你可以通过设置 `E2E_K8S_FULL_VERSION` 变量来指定用于运行 e2e 测试的 Kubernetes 版本： ```shell -E2E_K8S_FULL_VERSION=1.34.1 make test-e2e-baseline +E2E_K8S_FULL_VERSION=1.35.0 make test-e2e-baseline ``` 关于运行测试子集，请参阅 [运行测试子集](#running-subset-of-integration-or-e2e-tests)。 -## 增加日志详细程度 {#increase-logging-verbosity} -`TEST_LOG_LEVEL` 统一控制所有测试目标的日志级别： - -- `go test`、`make test`（单元测试） -- `make test-integration`（集成测试） -- `make test-*-e2e`（端到端测试） - -使用更小的负数获取更详细的日志，使用更大的正数降低日志量。例如： -```shell -TEST_LOG_LEVEL=-5 make test-integration # 更详细 -TEST_LOG_LEVEL=-1 make test # 比默认更少 -``` -默认值为 `TEST_LOG_LEVEL=-3`。 - -## 在 VSCode 中调试测试 {#debug-tests-in-vscode} -可以在 VSCode 中调试单元测试和集成测试。 -你需要安装 [Go 扩展](https://marketplace.visualstudio.com/items?itemName=golang.Go)。 -现在你将在如下行上方看到 `run test | debug test` 文本按钮： -```go -func TestValidateClusterQueue(t *testing.T) { -``` -你可以点击 `debug test` 来调试特定测试。 +### DEV 模式（保留集群）{#dev-mode} -对于集成测试，需要额外的步骤。在 settings.json 中，你需要在 `go.testEnvVars` 内添加两个变量： -- 运行 `ENVTEST_K8S_VERSION=1.34 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` 并将路径分配给 `KUBEBUILDER_ASSETS` 变量 -- 将 `KUEUE_BIN` 设置为你的 Kueue 仓库克隆目录内的 `bin` 目录 -```json -"go.testEnvVars": { - "KUBEBUILDER_ASSETS": "<上面输出的路径>", - "KUEUE_BIN": "<你的-kueue-文件夹路径>/bin", - }, -``` - -对于 e2e 测试，你也可以使用 [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer)。你需要在 settings.json 中添加以下变量： -```json - "ginkgotestexplorer.testEnvVars": { - "KIND_CLUSTER_NAME": "kind", - "WORKER1_KIND_CLUSTER_NAME": "kind-worker1", - "MANAGER_KIND_CLUSTER_NAME": "kind-manager", - "WORKER2_KIND_CLUSTER_NAME": "kind-worker2", - "KIND": "/bin/kind", - }, -``` -然后你可以使用 Ginkgo Test Explorer 的 GUI 来运行单个测试，前提是你已经启动了 kind 集群（有关说明，请参阅[这里](#attaching-e2e-tests-to-an-existing-kind-cluster)）。 - -## 将 e2e 测试附加到现有的 kind 集群 {#attaching-e2e-tests-to-an-existing-kind-cluster} -你可以使用以下方法来启动 kind 集群，然后从命令行或 VSCode 运行 e2e 测试， -将它们附加到现有集群。例如，假设你想测试一些 multikueue-e2e 测试。 - -### DEV 模式（推荐） 使用 `E2E_MODE=dev` 来创建或复用 kind 集群、重新构建/重新部署 Kueue、运行测试，并在测试结束后保留集群以便快速重跑或人工排查： ```shell @@ -135,10 +135,41 @@ E2E_MODE=dev make kind-image-build test-e2e-baseline E2E_MODE=dev make kind-image-build test-multikueue-e2e-main # 循环运行（直到失败），同时保留集群 -E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline +E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline + +# 跳过重新安装 kueue（仅在 dev 模式下有效） +E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-e2e-baseline +E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-multikueue-e2e-main + +# 跳过重新拉取依赖镜像并将其重新导入 kind（仅 dev 模式） +E2E_MODE=dev E2E_SKIP_IMAGE_RELOAD=true make kind-image-build test-e2e-baseline ``` -若希望使用**已发布**或**预发布（staging）**的 Kueue 镜像而非从源码构建（无需执行 `kind-image-build`），可传入 `IMAGE_TAG` 并指定所需镜像： +{{% alert title="Note" color="primary" %}} +当在 `E2E_MODE=dev` 下复用保留的集群时，外部算子（MPI、KubeRay 等）只会安装一次。 +如需在每次运行时强制重新安装它们，请设置 `E2E_ENFORCE_OPERATOR_UPDATE=true`。 + +将 `E2E_SKIP_IMAGE_RELOAD` 设置为真值（例如 `true`），可跳过对本地 Docker 缓存中已有的依赖镜像执行 `docker pull`， +也可跳过将镜像加载到节点 containerd 存储中已有该镜像引用的 kind worker 节点。 +这在多节点集群上可以加快重复运行的速度。 + +Kueue 控制器镜像始终会重新加载到集群中，除非设置了 `E2E_SKIP_REINSTALL=true`， +因为你可能在同一标签下使用 `make kind-image-build` 重新构建了它。 +{{% /alert %}} + +测试结束后如需删除保留的集群： +- 常规 e2e 测试，请运行： + ```shell + kind delete clusters kind + ``` +- MultiKueue 测试，请运行： + ```shell + kind delete clusters kind kind-manager kind-worker1 kind-worker2 + ``` + +### 使用已发布或预发布镜像 {#using-a-released-or-staging-image} + +若希望使用**已发布**或**预发布（staging）**的 Kueue 镜像而非从源码构建（无需执行 `kind-image-build`），可传入 `IMAGE_TAG`： ```shell # 已发布版本 @@ -156,22 +187,8 @@ E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue 适用于在特定已发布版本上复现问题（例如值班排查）。若要在真实集群（非 e2e）中安装已发布版本，请参阅[安装已发布版本](/zh-CN/docs/installation/#install-a-released-version)。 -{{% alert title="Note" color="primary" %}} -当在 `E2E_MODE=dev` 下复用保留的集群时，外部算子（MPI、KubeRay 等）只会安装一次。 -如需在每次运行时强制重新安装它们，请设置 `E2E_ENFORCE_OPERATOR_UPDATE=true`。 -{{% /alert %}} - -测试结束后如需删除保留的集群： -- 常规 e2e 测试，请运行： - ```shell - kind delete clusters kind - ``` -- MultiKueue 测试，请运行： - ```shell - kind delete clusters kind kind-manager kind-worker1 kind-worker2 - ``` +### 旧方式：交互式附加模式 {#legacy-interactive-attach-mode} -### 旧方式：交互式附加模式 运行 `E2E_RUN_ONLY_ENV=true make kind-image-build test-multikueue-e2e-main` 并等待 `Do you want to cleanup? [Y/n] ` 出现（CI 风格行为）。 集群已准备就绪，现在你可以从另一个终端运行测试： @@ -181,11 +198,85 @@ E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue 或从 VSCode 运行。 ## 运行集成或 e2e 测试子集 {#running-subset-of-integration-or-e2e-tests} + +### 使用标签过滤集成测试 {#use-label-filters-for-integration-tests} + +集成测试按控制器、作业类型、特性和区域打标签，可通过 `INTEGRATION_FILTERS` 和 `--label-filter` 运行特定子集： + +**标签分类：** +- 控制器：`controller:workload`、`controller:localqueue`、`controller:clusterqueue`、`controller:admissioncheck`、`controller:resourceflavor`、`controller:provisioning` +- 作业类型：`job:batch`、`job:pod`、`job:jobset`、`job:pytorch`、`job:tensorflow`、`job:mpi`、`job:paddle`、`job:xgboost`、`job:jax`、`job:train`、`job:ray`、`job:appwrapper`、`job:sparkapplication` +- 特性：`feature:tas`、`feature:multikueue`、`feature:provisioning`、`feature:fairsharing`、`feature:admissionfairsharing` +- 区域：`area:core`、`area:jobs`、`area:admissionchecks`、`area:multikueue` + +**示例：** +```shell +# 只运行 LocalQueue 测试 +INTEGRATION_FILTERS="--label-filter=controller:localqueue" make test-integration + +# 运行所有作业测试 +INTEGRATION_FILTERS="--label-filter=area:jobs" make test-integration + +# 运行 PyTorch 作业测试 +INTEGRATION_FILTERS="--label-filter=job:pytorch" make test-integration + +# 运行除 slow 以外的所有测试 +INTEGRATION_FILTERS="--label-filter=!slow" make test-integration + +# 运行核心测试（排除 slow） +INTEGRATION_FILTERS="--label-filter=area:core && !slow" make test-integration + +# 运行 TAS 相关测试 +INTEGRATION_FILTERS="--label-filter=feature:tas" make test-integration + +# 运行 FairSharing 测试 +INTEGRATION_FILTERS="--label-filter=feature:fairsharing" make test-integration +``` + +### 使用标签过滤 e2e 单集群测试 {#use-label-filters-for-e2e-singlecluster-tests} + +单集群测试按特性和区域打标签，可通过 `GINKGO_ARGS` 和 `--label-filter` 运行特定测试： + +**标签分类：** +- 特性：`appwrapper,certs,deployment,job,fairsharing,jaxjob,jobset,kuberay,kueuectl,leaderworkerset,metrics,pod,pytorchjob,statefulset,tas,trainjob,visibility,e2e_v1beta1,ha` + +**示例：** +```shell +# 只运行 appwrapper 测试 +GINKGO_ARGS="--label-filter=feature:appwrapper" make test-e2e-extended + +# 使用 helm 只运行 deployment 测试 +GINKGO_ARGS="--label-filter=feature:deployment" make test-e2e-baseline-helm + +# 使用 helm 只运行 jobset 和 trainjob 测试 +GINKGO_ARGS="--label-filter=feature:jobset,feature:trainjob" make test-e2e-extended +``` + +### 使用标签过滤 e2e 顺序测试 {#use-label-filters-for-e2e-sequential-tests} + +顺序测试（Baseline 和 Extended）按特性打标签，可通过 `GINKGO_ARGS` 和 `--label-filter` 运行特定测试： + +**标签分类（Baseline）：** +- 特性：`admissionfairsharing, certs, failurerecoverypolicy, localqueuemetrics, objectretentionpolicies, podintegrationautoenablement, reconcile, visibility, waitforpodsready` + +**标签分类（Extended）：** +- 特性：`managejobswithoutqueuename, spark` + +**示例：** +```shell +# 只运行 admissionfairsharing 测试（Baseline） +GINKGO_ARGS="--label-filter=feature:admissionfairsharing" make test-e2e-sequential-baseline + +# 只运行 spark 测试（Extended） +GINKGO_ARGS="--label-filter=feature:spark" make test-e2e-sequential-extended +``` + ### 使用 Ginkgo --focus 参数 {#use-ginkgo-focus-arg} ```shell GINKGO_ARGS="--focus=Scheduler" make test-integration GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline ``` + ### 使用 ginkgo.FIt {#use-ginkgo-fit} 如果你想专注于特定测试，可以将这些测试的 `ginkgo.It` 更改为 `ginkgo.FIt`。 @@ -238,6 +329,70 @@ Kueue 作为常规 pod 在 worker 节点上运行，在 e2e 测试中有 2 个 ``` 这里，它是 `core/clusterqueue_controller.go:341`。 +--- + +## 进阶 {#advanced} + +### 运行预提交验证测试 {#running-presubmission-verification-tests} +```shell +make verify +``` + +### 增加日志详细程度 {#increase-logging-verbosity} +`TEST_LOG_LEVEL` 统一控制所有测试目标的日志级别： + +- `go test`、`make test`（单元测试） +- `make test-integration`（集成测试） +- `make test-*-e2e`（端到端测试） + +使用更小的负数获取更详细的日志，使用更大的正数降低日志量。例如： +```shell +TEST_LOG_LEVEL=-5 make test-integration # 更详细 +TEST_LOG_LEVEL=-1 make test # 比默认更少 +``` +默认值为 `TEST_LOG_LEVEL=-3`。 + +### 在 VSCode 中调试测试 {#debug-tests-in-vscode} +可以在 VSCode 中调试单元测试和集成测试。 +你需要安装 [Go 扩展](https://marketplace.visualstudio.com/items?itemName=golang.Go)。 +现在你将在如下行上方看到 `run test | debug test` 文本按钮： +```go +func TestValidateClusterQueue(t *testing.T) { +``` +你可以点击 `debug test` 来调试特定测试。 + +对于集成测试，需要额外的步骤。在 settings.json 中，你需要在 `go.testEnvVars` 内添加两个变量： +- 运行 `ENVTEST_K8S_VERSION=1.35 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` 并将路径分配给 `KUBEBUILDER_ASSETS` 变量 +- 将 `KUEUE_BIN` 设置为你的 Kueue 仓库克隆目录内的 `bin` 目录 +```json +"go.testEnvVars": { + "KUBEBUILDER_ASSETS": "<上面输出的路径>", + "KUEUE_BIN": "<你的-kueue-文件夹路径>/bin", + }, +``` + +对于 e2e 测试，你也可以使用 [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer)。你需要在 settings.json 中添加以下变量： +```json + "ginkgotestexplorer.testEnvVars": { + "KIND_CLUSTER_NAME": "kind", + "WORKER1_KIND_CLUSTER_NAME": "kind-worker1", + "MANAGER_KIND_CLUSTER_NAME": "kind-manager", + "WORKER2_KIND_CLUSTER_NAME": "kind-worker2", + "KIND": "/bin/kind", + }, +``` +然后你可以使用 Ginkgo Test Explorer 的 GUI 来运行单个测试，前提是你已经启动了 kind 集群（有关说明，请参阅[旧方式：交互式附加模式](#legacy-interactive-attach-mode)）。 + +### 使用 Prometheus 调试指标 {#debugging-metrics-with-prometheus} + +使用预配置了 Prometheus 的 Kind 集群进行指标调试： + +```shell +E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e-baseline +``` + +更多详情，请参阅 [Setup Dev Monitoring](/zh-CN/docs/tasks/dev/setup_dev_monitoring)。 + ### 另请参阅 {#see-also} - [Kubernetes 测试指南](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/testing.md) - [Kubernetes 中的集成测试](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/integration-tests.md)