Switch to using kind build node-image for e2e tests#11251
Conversation
Instead of pulling pre-built kindest/node images from Docker Hub, build them locally using "kind build node-image". This downloads K8s release tarballs rather than compiling from source, so the cost is comparable to pulling the pre-built image. This gives kueue control over e2e testing timing without relying on kind to publish official node images for each K8s release. In dev mode, the build is skipped when the image already exists locally. Non-standard images (e.g. k8s-main:latest) are also skipped since they have their own build targets.
k8s-ci-robot
added
release-note-none
✅ Deploy Preview for kubernetes-sigs-kueue ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kannon92 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
size/S
| fi | ||
|
|
||
| echo "Building kind node image: $E2E_KIND_VERSION (K8s v$KIND_VERSION)" | ||
| $KIND build node-image "v$KIND_VERSION" --image "$E2E_KIND_VERSION" |
There was a problem hiding this comment.
How much longer does it take compared to just downloading it? I’m wondering if we can first check whether the image already exists, and only build it if it doesn’t.
There was a problem hiding this comment.
Doesn't seem too bad, the longest taking suite took 26min, which seems normal for the suite: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/11251/pull-kueue-test-e2e-sequential-baseline-main/2055376057527177216
There was a problem hiding this comment.
Either way you have to download the base image layer (squashed) and the kubernetes binaries (Either via a node-image or via the release tarball) so it's similar.
|
/test all |
|
cc @BenTheElder ptal |
| export KIND_VERSION="${E2E_KIND_VERSION/"kindest/node:v"/}" | ||
|
|
||
| function build_kind_node_image { | ||
| if [[ "$E2E_KIND_VERSION" != kindest/node:v* ]]; then |
There was a problem hiding this comment.
Same comments as kubernetes-sigs/jobset#1234 (comment)
5 participants
What type of PR is this?
/kind cleanup
/area testing
What this PR does / why we need it:
Switches e2e tests to build kind node images locally from K8s release
artifacts instead of pulling pre-built
kindest/nodeimages fromDocker Hub.
This adds a
build_kind_node_imagefunction ine2e-common.shthatruns
kind build node-image v<version>before cluster creation. Thecommand downloads release tarballs (does not compile K8s from source),
so the cost is comparable to pulling the pre-built image.
Behavior:
k8s-main:latest) are skipped since theyhave their own build targets
E2E_MODE=dev), the build is skipped when the imagealready exists locally
Which issue(s) this PR fixes:
Fixes #11249
Special notes for your reviewer:
The existing
kind-k8s-main-image-buildMakefile target is unchangedsince it builds from CI artifacts, not releases.
Does this PR introduce a user-facing change?