feat: Replace static musl build on alpine with glibc based build on gcr distroless

[JakeDern]

Change Summary

This PR changes our standard df_engine image from being a static build based on musl + alpine to being based on glibc on gcr distroless.

Part of this change requires updates to the mount paths for orchestrator config files as the home directory is now nonroot instead of dataflow.

What issue does this PR close?

• Closes Replace static musl build + alpine image with glibc build + distroless gcp image #2918

How are these changes tested?

To test this, I did local runs of all the nightly/continuous/comparison dashboard suites using the new image + mount path changes.

I did my best to test cross compiling with the new targets for ARM as well:

Are there any user-facing changes?

Yes, the runtime image has been changed for the repo dockerfile and that comes along with some new expectations for mount paths.

[lquerel]

LGTM

[lalitb]

Nice work on this. I authored #2214, and I think this is the better fix overall. #2214 fixed the immediate musl build issue. This PR avoids that problem completely by moving the image to glibc + distroless, which feels cleaner long term.

I left one small comment about the binary size workflow, but overall this direction looks good to me.

[JakeDern]

I ran a little experiment and indeed after this PR we use a huge amount of memory to build compared to before - Now the question is why:

  ┌─────────────────────────┬───────────┬──────────────────┐
  │         Branch          │ Wall time │ Peak memory used │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ main (9a664bf71)        │ 6m 30s    │ 4.85 GiB         │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ glibc-image (1165f5dbf) │ 8m 22s    │ 18.51 GiB        │
  └─────────────────────────┴───────────┴──────────────────┘

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.93%. Comparing base (9a664bf) to head (17dcbaf).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2919      +/-   ##
==========================================
- Coverage   85.93%   85.93%   -0.01%     
==========================================
  Files         725      725              
  Lines      275782   275782              
==========================================
- Hits       236993   236988       -5     
- Misses      38265    38270       +5     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	87.06% <ø> (-0.01%)	⬇️
query_abstraction	80.61% <ø> (ø)
query_engine	89.57% <ø> (ø)
otel-arrow-go	52.45% <ø> (ø)
quiver	92.25% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lalitb]

I ran a little experiment and indeed after this PR we use a huge amount of memory to build compared to before - Now the question is why:

  ┌─────────────────────────┬───────────┬──────────────────┐
  │         Branch          │ Wall time │ Peak memory used │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ main (9a664bf71)        │ 6m 30s    │ 4.85 GiB         │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ glibc-image (1165f5dbf) │ 8m 22s    │ 18.51 GiB        │
  └─────────────────────────┴───────────┴──────────────────┘

My wild guess is that the final link step is more memory hungry for glibc as compared to musl. Probably can validate by adding temporary memory sampling around the cargo build --release step and checking whether the peak happens near the final rustc/linker invocation.

[JakeDern]

I ran a little experiment and indeed after this PR we use a huge amount of memory to build compared to before - Now the question is why:

  ┌─────────────────────────┬───────────┬──────────────────┐
  │         Branch          │ Wall time │ Peak memory used │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ main (9a664bf71)        │ 6m 30s    │ 4.85 GiB         │
  ├─────────────────────────┼───────────┼──────────────────┤
  │ glibc-image (1165f5dbf) │ 8m 22s    │ 18.51 GiB        │
  └─────────────────────────┴───────────┴──────────────────┘

My wild guess is that the final link step is more memory hungry for glibc as compared to musl. Probably can validate by adding temporary memory sampling around the cargo build --release step and checking whether the peak happens near the final rustc/linker invocation.

Removing the config.toml overrides for the linker brought it back down. I kept them originally and changed them to -gnu because I wasn't sure why they were in there, but then took them out as an experiment which was successful. My guess is that specifying those overrides clobbers more settings/flags than it appears and that somehow influences the memory usage during link time