[chore] more speedups for build-and-test-experimental#48426
Open
TylerHelmuth wants to merge 10 commits into
Open
[chore] more speedups for build-and-test-experimental#48426TylerHelmuth wants to merge 10 commits into
TylerHelmuth wants to merge 10 commits into
Conversation
Speedups under test: - setup-go-tools: bump `make -j2 gomoddownload` to `-j8` so the upfront module download (the dominant cost when the GHA cache is cold, which is the common case in contrib) parallelizes more. - cross-compile-affected: skip the upfront `gomoddownload` entirely. The job only builds the affected module set, so `go build` can fetch what it needs lazily. - exp-checks/read-only-checks shard: skip `gomoddownload` and `make genotelcontribcol` -- checkdoc/checkmetadata/checkapi/ multimod-verify don't need either. CRITICAL_FILES temporarily drops `^.github/workflows/` and `^.github/actions/` so this PR runs in scoped mode while iterating on the workflow itself. To be restored before merge. Touches processor/filterprocessor so scoped mode has an affected module to exercise the cross-compile-affected job.
checkdoc/checkmetadata/checkapi depend on cmd/otelcontribcol/components.go (via the checkfile build-tool). Restore `make genotelcontribcol` for that shard while keeping the `gomoddownload: false` skip.
Convert the root `make generate` from a serial shell loop (`for-all CMD="go generate ./..."`) over ~500 modules to the parallelizable `for-all-target` pattern with `-j8`. Adds a small `generate-mod` submodule target so each module's `go generate ./...` runs as a real make target instead of a shell-iterated command. mdatagen output lives under each module's internal/metadata and isn't imported across modules, so per-module generation is independent and parallel-safe.
The codegen parallelization edit modifies Makefile.Common, which was forcing the test PR back into full mode. Temporary, restore before merge along with the workflow/action exclusions.
`go generate ./...` invokes `//go:generate make mdatagen` directives. With the outer `make -j8` providing a jobserver, the make spawned by `go generate` (one process boundary away via go's exec) sees MAKEFLAGS but can't reach the jobserver pipe, producing: make[3]: warning: jobserver unavailable: using -j1. Clear MAKEFLAGS for the duration of `go generate` so the inner make doesn't try to use the jobserver in the first place. mdatagen is a single command anyway, so losing parallelism there changes nothing.
This reverts commit 8d3671f.
This reverts commit d6dd840.
Replace the single `codegen` exp-checks shard with one per module group (codegen-receiver-0, codegen-processor-0, ..., codegen-other). Each shard runs `make generate GROUP=<group>` over only its subset, parallelizing the work across matrix runners instead of within a single make invocation (which was tried in d6dd840 and reverted because the per-module recursive make overhead exceeded the benefit). Adds a GROUP-aware path to the `generate` target: when GROUP is set at SRC_ROOT, dispatch via `$(FOR_GROUP_TARGET) TARGET=generate` so each submodule's existing generate recipe handles its own `go generate ./... + fmt + gci`. When GROUP is unset, behavior is unchanged (for-all + batched gofmt/gogci). run-checks-shard.sh maps `codegen-<group>` → `make generate GROUP=<group>`, then runs the existing git-clean check.
TylerHelmuth
changed the title
Drop the test-only changes used to exercise scoped mode while iterating on the workflow: - Restore ^.github/workflows/, ^.github/actions/, ^Makefile, and ^Makefile.Common entries in CRITICAL_FILES. - Revert the no-op edit to processor/filterprocessor/doc.go.
andrzej-stencel
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Speeds up
build-and-test-experimental:gomoddownloadforcross-compile-affectedand theread-only-checksshard — neither needs every module pre-downloaded; what they touch is fetched lazily.make gomoddownload:-j2→-j8. Network-bound on the 4 vCPU runner; helps every cold-cache run.codegen-<group>shards (one per module group).make generatelearns to honorGROUP.Example run: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/25942633975