Suspected memory leak with the otel.MetricsReporter

[Orenico10]

Component

OBI (eBPF instrumentation)

What happened?

OBI (running as a daemonset) in a semi-large k8s cluster (32 nodes) has a specific pod with usage of 180Gi of memory (while the rest use 2-12Gi).
There is an INFO log that is being written multiple time per ms saying the following:

level=INFO msg="failed to upload metrics: context deadline exceeded"

And this WARN log being written every now and then (lower frequency compared to the INFO log):

level=WARN msg="error flushing evicted metrics provider" component=otel.MetricsReporter service="{Name:Kube-rbac-proxy Namespace: Instance:REDACTED}" error="context deadline exceeded"

What did you expect to happen?

Metrics being sent to the upstream collector and no memory leaks ;)

Steps to reproduce

Not sure.

OBI configuration

network:
  enable: true
  protocol: ["TCP"]
ebpf:
  enable: true
  context_propagation: all
  buffer_sizes:
    http: 8192
    mysql: 8192
    kafka: 8192
    postgres: 8192
    tcp: 8192
attributes:
  select:
    span:
      include:
        - "net.transport"
        - "net.sock.peer.addr"
        - "net.sock.peer.port"
        - "net.host.port"
otel_metrics_export:
  endpoint: "http://my-collector.svc.cluster:4318"
  protocol: http/protobuf
otel_traces_export:
  endpoint: "http://my-collector.svc.cluster:4318"
  protocol: http/protobuf
discovery:
  exclude_instrument:
    - exe_path: '{*ebpf-instrument*,*otelcol*}'
    - k8s_namespace: 'obi'
  instrument:
    - k8s_namespace: '*'
    - exe_path: '/bin/kubelet'

Relevant logs

Additional context

Memory usage graph of the last 2 days:

Version

0.9.0

Kernel version

No response

OS / Distribution

OpenShift 4.16

Deployment

Kubernetes