Skip to content

doc: reorganize test documentation #11169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k8s-ci-robot merged 1 commit into from
May 18, 2026
+373 −161
Merged

doc: reorganize test documentation #11169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
233 changes: 145 additions & 88 deletions site/content/en/docs/contribution_guidelines/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,50 @@ description: >
Running and debugging tests
---

## Running presubmission verification tests
## Quick start

The most common development workflows:

```shell
make verify
# Unit tests
make test

# Integration tests
make test-integration

# E2E: build image and run
# Note: PLATFORM=linux/arm64 is only needed on Apple Silicon (arm64).
# On amd64, run: make kind-image-build test-e2e-baseline
PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline

# E2E: build, run, and keep the kind cluster alive between runs
E2E_MODE=dev PLATFORM=linux/arm64 make kind-image-build test-e2e-baseline
```

### Quick filter reference

```shell
# Focus on tests matching a name pattern
GINKGO_ARGS="--focus=Scheduler" make test-integration
GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline

# Filter integration tests by label
INTEGRATION_FILTERS="--label-filter=controller:workload" make test-integration
INTEGRATION_FILTERS="--label-filter=area:jobs" make test-integration

# Filter e2e tests by label
GINKGO_ARGS="--label-filter=feature:jobset" make test-e2e-extended

# Run only a specific integration test directory
INTEGRATION_TARGET='test/integration/singlecluster/scheduler' make test-integration
```

For the full label taxonomy and more examples, see [Running a subset of tests](#running-subset-of-integration-or-e2e-tests).

---

## Running unit tests

To run all unit tests:
```shell
make test
Expand Down Expand Up @@ -53,78 +91,42 @@ make test-integration

For running a subset of tests, see [Running subset of tests](#running-subset-of-integration-or-e2e-tests).

## Running e2e tests using custom build
## Running e2e tests

E2E tests build a Kueue image and load it into a local Kind cluster. The build step must run before the test target:

```shell
make kind-image-build
make test-tas-e2e-baseline
make test-tas-e2e-extended
```

On Apple Silicon (arm64), set `PLATFORM`:
```shell
PLATFORM=linux/arm64 make kind-image-build
```

Then run the desired test target:
```shell
make test-e2e-baseline
make test-e2e-extended
make test-e2e-sequential-extended
make test-e2e-sequential-baseline
make test-e2e-sequential-extended
make test-e2e-certmanager
make test-e2e-kueueviz
make test-tas-e2e-baseline
make test-tas-e2e-extended
make test-multikueue-e2e-main
make test-multikueue-e2e-sequential
```

You can specify the Kubernetes version used for running the e2e tests by setting the `E2E_K8S_FULL_VERSION` variable:
You can specify the Kubernetes version:
```shell
E2E_K8S_FULL_VERSION=1.35.0 make test-e2e-baseline
```

For running a subset of tests, see [Running subset of tests](#running-subset-of-integration-or-e2e-tests).

## Increase logging verbosity
`TEST_LOG_LEVEL` controls test logging uniformly for all targets:

- `go test`, `make test` (unit tests)
- `make test-integration` (integration tests)
- `make test-*-e2e` (e2e tests)
### DEV mode (keep the cluster)

Use more negative values for more verbose logs and higher (positive) values for quieter logs. For example:
```shell
TEST_LOG_LEVEL=-5 make test-integration # more verbose
TEST_LOG_LEVEL=-1 make test # less verbose than default
```
Default is `TEST_LOG_LEVEL=-3`.

## Debug tests in VSCode
It is possible to debug unit and integration tests in VSCode.
You need to have the [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed.
Now you will have `run test | debug test` text buttons above lines like
```go
func TestValidateClusterQueue(t *testing.T) {
```
You can click on the `debug test` to debug a specific test.

For integration tests, an additional step is needed. In settings.json, you need to add two variables inside `go.testEnvVars`:
- Run `ENVTEST_K8S_VERSION=1.35 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` and assign the path to the `KUBEBUILDER_ASSETS` variable
- Set `KUEUE_BIN` to the `bin` directory within your cloned Kueue repository
```json
"go.testEnvVars": {
"KUBEBUILDER_ASSETS": "<path from output above>",
"KUEUE_BIN": "<path-to-your-kueue-folder>/bin",
},
```

For e2e tests, you can also use [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer). You need to add the following variables to settings.json:
```json
"ginkgotestexplorer.testEnvVars": {
"KIND_CLUSTER_NAME": "kind",
"WORKER1_KIND_CLUSTER_NAME": "kind-worker1",
"MANAGER_KIND_CLUSTER_NAME": "kind-manager",
"WORKER2_KIND_CLUSTER_NAME": "kind-worker2",
"KIND": "<your_kueue_path>/bin/kind",
},
```
and then you can use GUI of the Ginkgo Test Explorer to run individual tests, provided you started kind clanter (see [here](#attaching-e2e-tests-to-an-existing-kind-cluster) for the instructions).

## Attaching e2e tests to an existing kind cluster
You can use the following approach to start up a kind cluster and then run e2e tests from commandline or VSCode,
attaching them to the existing cluster. For example, suppose you want to test some of the multikueue-e2e tests.

### DEV mode (recommended)
Use `E2E_MODE=dev` to create-or-reuse a kind cluster, rebuild/redeploy Kueue, run tests, and keep the cluster running for fast reruns and post-test investigation:

```shell
Expand All @@ -135,8 +137,7 @@ E2E_MODE=dev make kind-image-build test-e2e-baseline
E2E_MODE=dev make kind-image-build test-multikueue-e2e-main

# Loop a suite (until it fails) while keeping the cluster
E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline

E2E_MODE=dev GINKGO_ARGS="--until-it-fails" make kind-image-build test-e2e-baseline

# Skip reinstallation of kueue (works only in dev mode)
E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-e2e-baseline
Expand All @@ -146,24 +147,6 @@ E2E_MODE=dev E2E_SKIP_REINSTALL=true make kind-image-build test-multikueue-e2e-m
E2E_MODE=dev E2E_SKIP_IMAGE_RELOAD=true make kind-image-build test-e2e-baseline
```

To use a **released** or **staging** Kueue image instead of building from source (no `kind-image-build` needed), pass `IMAGE_TAG` with the desired image:

```shell
# Released version
E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline
E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-multikueue-e2e-main

# Staging image (e.g. from a PR or nightly)
E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:main make test-e2e-baseline
```

**Using a released version with matching manifests:** The e2e framework deploys CRDs and other resources from the repo's config and overrides only the controller image via `IMAGE_TAG`. To run e2e against a specific release with manifests that match that image:

1. Check out that version's tag (e.g. `git checkout v0.16.0`). The CRD and deployment config in the repo are committed at each release, so no `make manifests` step is needed.
2. Run the command above with the same image tag, e.g. `E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline`.

This is useful to reproduce issues on a specific released version (e.g. for on-call debugging). For installing a released version into a real cluster (not e2e), see [Install a released version](/docs/installation/#install-a-released-version).

{{% alert title="Note" color="primary" %}}
When reusing a kept cluster in `E2E_MODE=dev`, external operators (MPI, KubeRay, etc.) are installed only once.
To force re-installing them on every run, set `E2E_ENFORCE_OPERATOR_UPDATE=true`.
Expand All @@ -187,24 +170,35 @@ To delete the kept cluster(s) afterwards:
kind delete clusters kind kind-manager kind-worker1 kind-worker2
```

### Legacy: interactive attach mode
Run `E2E_RUN_ONLY_ENV=true make kind-image-build test-multikueue-e2e-main` and wait for the `Do you want to cleanup? [Y/n] ` to appear (CI-style behavior).
### Using a released or staging image

To use a **released** or **staging** Kueue image instead of building from source (no `kind-image-build` needed), pass `IMAGE_TAG`:

The cluster is ready, and now you can run tests from another terminal:
```shell
<your_kueue_path>/bin/ginkgo --json-report ./ginkgo.report -focus "MultiKueue when Creating a multikueue admission check Should run a jobSet on worker if admitted" -r
# Released version
E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline
E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-multikueue-e2e-main

# Staging image (e.g. from a PR or nightly)
E2E_MODE=dev IMAGE_TAG=us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:main make test-e2e-baseline
```
or from VSCode.

## Debugging metrics with Prometheus
**Using a released version with matching manifests:** The e2e framework deploys CRDs and other resources from the repo's config and overrides only the controller image via `IMAGE_TAG`. To run e2e against a specific release with manifests that match that image:

To provision a Kind cluster with Prometheus pre-configured for metrics debugging:
1. Check out that version's tag (e.g. `git checkout v0.16.0`). The CRD and deployment config in the repo are committed at each release, so no `make manifests` step is needed.
2. Run the command above with the same image tag, e.g. `E2E_MODE=dev IMAGE_TAG=registry.k8s.io/kueue/kueue:v0.16.0 make test-e2e-baseline`.

This is useful to reproduce issues on a specific released version (e.g. for on-call debugging). For installing a released version into a real cluster (not e2e), see [Install a released version](/docs/installation/#install-a-released-version).

### Legacy: interactive attach mode

Run `E2E_RUN_ONLY_ENV=true make kind-image-build test-multikueue-e2e-main` and wait for the `Do you want to cleanup? [Y/n] ` to appear (CI-style behavior).

The cluster is ready, and now you can run tests from another terminal:
```shell
E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e-baseline
<your_kueue_path>/bin/ginkgo --json-report ./ginkgo.report -focus "MultiKueue when Creating a multikueue admission check Should run a jobSet on worker if admitted" -r
```

For more details, see [Setup Dev Monitoring](/docs/tasks/dev/setup_dev_monitoring).
or from VSCode.

## Running subset of integration or e2e tests

Expand Down Expand Up @@ -252,7 +246,6 @@ SingleCluster tests are labeled by feature and area. You can use `GINKGO_ARGS` w
# Run only appwrapper tests
GINKGO_ARGS="--label-filter=feature:appwrapper" make test-e2e-extended


# Run only deployment tests with helm
GINKGO_ARGS="--label-filter=feature:deployment" make test-e2e-baseline-helm

Expand Down Expand Up @@ -283,6 +276,7 @@ GINKGO_ARGS="--label-filter=feature:spark" make test-e2e-sequential-extended
GINKGO_ARGS="--focus=Scheduler" make test-integration
GINKGO_ARGS="--focus='Creating a Pod requesting TAS'" make test-e2e-baseline
```

### Use ginkgo.FIt
If you want to focus on specific tests, you can change
`ginkgo.It` to `ginkgo.FIt` for these tests.
Expand Down Expand Up @@ -333,7 +327,70 @@ For each log message you can from which file and line the message is coming from
```log
2025-02-03T15:51:51.502425029Z stderr F 2025-02-03T15:51:51.502117824Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:341 ClusterQueue update event {"clusterQueue": {"name":"cluster-queue"}}
```
Here, it's `core/clusterqueue_controller.go:341`.

---

## Advanced

### Running presubmission verification tests
```shell
make verify
```

### Increase logging verbosity
`TEST_LOG_LEVEL` controls test logging uniformly for all targets:

- `go test`, `make test` (unit tests)
- `make test-integration` (integration tests)
- `make test-*-e2e` (e2e tests)

Use more negative values for more verbose logs and higher (positive) values for quieter logs. For example:
```shell
TEST_LOG_LEVEL=-5 make test-integration # more verbose
TEST_LOG_LEVEL=-1 make test # less verbose than default
```
Default is `TEST_LOG_LEVEL=-3`.

### Debug tests in VSCode
It is possible to debug unit and integration tests in VSCode.
You need to have the [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed.
Now you will have `run test | debug test` text buttons above lines like
```go
func TestValidateClusterQueue(t *testing.T) {
```
You can click on the `debug test` to debug a specific test.

For integration tests, an additional step is needed. In settings.json, you need to add two variables inside `go.testEnvVars`:
- Run `ENVTEST_K8S_VERSION=1.35 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path` and assign the path to the `KUBEBUILDER_ASSETS` variable
- Set `KUEUE_BIN` to the `bin` directory within your cloned Kueue repository
```json
"go.testEnvVars": {
"KUBEBUILDER_ASSETS": "<path from output above>",
"KUEUE_BIN": "<path-to-your-kueue-folder>/bin",
},
```

For e2e tests, you can also use [Ginkgo Test Explorer](https://marketplace.visualstudio.com/items?itemName=joselitofilho.ginkgotestexplorer). You need to add the following variables to settings.json:
```json
"ginkgotestexplorer.testEnvVars": {
"KIND_CLUSTER_NAME": "kind",
"WORKER1_KIND_CLUSTER_NAME": "kind-worker1",
"MANAGER_KIND_CLUSTER_NAME": "kind-manager",
"WORKER2_KIND_CLUSTER_NAME": "kind-worker2",
"KIND": "<your_kueue_path>/bin/kind",
},
```
and then you can use GUI of the Ginkgo Test Explorer to run individual tests, provided you started kind cluster (see [here](#legacy-interactive-attach-mode) for the instructions).

### Debugging metrics with Prometheus

To provision a Kind cluster with Prometheus pre-configured for metrics debugging:

```shell
E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e-baseline
```

For more details, see [Setup Dev Monitoring](/docs/tasks/dev/setup_dev_monitoring).

### See also
- [Kubernetes testing guide](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/testing.md)
Expand Down
Loading