chore(benchmark): use bare metal runners

[rovertrack]

Description

Improved codespeed to use bare metal runners

Motivation

Fixes #467

---

Checklist

• PR title follows conventional commits format
• Code formatted: make format
• Linters pass: make lint
• Tests pass: make test
• Tests added for new functionality
• Tests follow testing guidelines
• Documentation updated (if applicable)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: rovertrack / name: Rover track (6b032be)

[rovertrack]

opentelemetry-go-compile-instrumentation/.github/workflows/benchmark.yaml

Line 38 in 8c7600d

runs-on: ubuntu-latest # TODO: oracle-bare-metal-64cpu-1024gb-x86-64-ubuntu-24

Wouldn't this be running for every PR?

[txabman42]

Wouldn't this be running for every PR?

No, codespeed job just runs on master after PR merge.

From otel docs:

Access to bare metal runners for benchmarking is available to repositories on request (open a community issue)

Let's wait until these issues (open-telemetry/community#3427 and open-telemetry/community#3402) are resolved before merging this PR.

I've moved the PR to DRAFT until issues are resolved.

[rovertrack]

Ok, well you can assign me this 3247
As it's still unassigned!

[txabman42]

@rovertrack I dont have privs on that repo for assigning issues. I guess it will be assigned to the responsible for executing that issue.

[rovertrack]

Ok then I'm gonna ask in that issue section to assign me
Thanks.

[rovertrack]

@txabman42 comment

[txabman42]

@rovertrack could you provide a comparison between current and bare-metal runners? We should ensure they work before merging + waiting times (related to open-telemetry/community#3402) + expected Rel. Std. Dev diff.

[rovertrack]

could you provide a comparison between current and bare-metal runners? We should ensure they work before merging + waiting times (related to open-telemetry/community#3402) + expected Rel. Std. Dev diff.

updating you in some time!

[txabman42]

If we go with bare-metals, lets move codspeed to run daily, not per merge in master. There are some discussions ongoing between OTEL mantainers about limiting or refactoring how the bare-metal instance is used.

[rovertrack]

The latest commit is for runner comparison if you approve the workflow on this PR I can give you the comparison diff.

[rovertrack]

Case	ubuntu-latest	Bare-Metal Runner	Improvements
Queue Wait Time	~4 seconds	~23.5 minutes	—
baseline/otelc (R.S.D)	1%	1%	Identical
multi/otelc (R.S.D)	2%	1%	+1% Stability
largeidle/otelc (R.S.D)	5%	1%	+4% Stability
largeidle/plain (R.S.D)	4%	1%	+3% Stability

Ubuntu-latest benchstat artifact
Bare metal runner benchstat artifact

[txabman42]

Nice, those are great results! Aligned with the expected values

The PR should be mergeable after removing the comparison job and setting codspeed as a daily run.

Don't forget to remove the other comment #477 (comment)

[rovertrack]

@txabman42 removed the comparison job and making it ready for review!

[rovertrack]

@txabman42 i have made it to run daily