Add profiling to the otel demo and use pyroscope as the backend

[osullivandonal]

Changes

This work add's profiling to the OTel-Demo.

Closes: #1601

• OTel collector added for profiling with heightened privileges, this is needed to profiling requiring access to the kernel.

• Send the profiling data to a pyroscope backend.

• Makefile updated with start-profiling target which starts the demo along with the profiling collector.

Note

This work leveraged the documentation from Pyroscope, here.
We can also add this to k8s, this would be done in the helm charts.

Running

make start-profiling

You can view profiles via 2 paths:

• http://localhost:4040/
  • This is the pyroscope UI
• http://localhost:8080/grafana
  • This is via Grafana

Profiling in Grafana

Here we can see an overview of the profiling in Grafana

Ad service High CPU

Here we turn on high CPU for the ad service (feature-flag) and we can use profiling to show us the problem code:

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[osullivandonal]

Closing this PR as #3333 superseeds this one.