Skip to content

server: Set a pprof label on new stream goroutines #9082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dfinkel wants to merge 15 commits into
base: master
Choose a base branch
from
Open

server: Set a pprof label on new stream goroutines #9082

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
84dda06
server: Set a pprof label on new stream goroutines
dfinkel Mar 24, 2026
f2029fc
server/envconfig: make goroutine labels a bitfield
dfinkel Apr 22, 2026
3b7e587
test: server: verify goroutine labels in ctxs
dfinkel Apr 23, 2026
c3f70d9
envconfig: add test for goroutineLabelsFromEnv
dfinkel Apr 23, 2026
7aaf9c9
server_test: use testutils.SetEnvConfig
dfinkel Apr 28, 2026
5329657
envconfig: revise goroutineLabelsFromEnv
dfinkel Apr 28, 2026
819e7a7
envconfig: test: actually set the subtest name
dfinkel Apr 28, 2026
e1d919e
envconfig: adjust new comment wrapping and grammar
dfinkel Apr 28, 2026
e32f01e
server_test: wrap comments at 80 columns and punctuate
dfinkel Apr 28, 2026
cbcdf44
envconfig: add AllGoroutineLabels & "all" value
dfinkel May 11, 2026
e176a2b
envconfig: flip goroutine label default to disabled
dfinkel May 11, 2026
05172ce
envconfig: goroutine labels: add "none" value
dfinkel May 11, 2026
921d8cb
Merge remote-tracking branch 'origin/master' into server_goroutine_la…
dfinkel May 11, 2026
e238722
server: add the envconfig import back
dfinkel May 11, 2026
3854612
test: server: s/expVal/wantVal/ & reduce err scopes
dfinkel May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions internal/envconfig/envconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ var (
// unconditionally.
XDSEndpointHashKeyBackwardCompat = boolFromEnv("GRPC_XDS_ENDPOINT_HASH_KEY_BACKWARD_COMPAT", false)

// LabelServerStreamGoroutines controls setting [runtime/pprof.Labels] on the
// goroutines spawned to handle incoming requests on the server.
// Set "GRPC_SERVER_METHOD_GOROUTINE_LABELS" to "false" to disable.
LabelServerStreamGoroutines = boolFromEnv("GRPC_SERVER_METHOD_GOROUTINE_LABELS", true)
Copy link
Copy Markdown
Contributor

@arjan-bal arjan-bal Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking ahead to when we might have more labels, maintaining one environment variable per label seems tedious. Could we instead use a single variable whose value is a comma-separated list of key-value pairs? For example: GRPC_GO_PPROF_LABELS_ENABLED="grpc.method=false".

This would be similar to the parsing of the GODEBUG environment variable

Copy link
Copy Markdown
Author

@dfinkel dfinkel Apr 23, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea
... and Done.

Comment thread
arjan-bal marked this conversation as resolved.
Outdated

// RingHashSetRequestHashKey is set if the ring hash balancer can get the
// request hash header by setting the "requestHashHeader" field, according
// to gRFC A76. It can be disabled by setting the environment variable
Expand Down
7 changes: 7 additions & 0 deletions server.go
View file
Open in desktop
Comment thread
arjan-bal marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
"net/http"
"reflect"
"runtime"
"runtime/pprof"
"strings"
"sync"
"sync/atomic"
Expand Down Expand Up @@ -1785,6 +1786,12 @@ func (s *Server) handleMalformedMethodName(stream *transport.ServerStream, ti *t
func (s *Server) handleStream(t transport.ServerTransport, stream *transport.ServerStream) {
ctx := stream.Context()
ctx = contextWithServer(ctx, s)
if envconfig.LabelServerStreamGoroutines {
// This method always runs in its own goroutine, so we can set a
// goroutine label without needing to restore a previous context.
ctx = pprof.WithLabels(ctx, pprof.Labels("grpc.server.method", stream.Method()))
Copy link
Copy Markdown
Contributor

@arjan-bal arjan-bal Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the key "grpc.method" to be consistent with the OpenTelemetry plugins? gRPC is moving away from OpenCensus in favour of OpenTelemetry. See https://grpc.io/docs/guides/opentelemetry-metrics/#server-instruments and https://grpc.io/docs/guides/opentelemetry-metrics/#labelsattributes

Copy link
Copy Markdown
Author

@dfinkel dfinkel Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Done.

pprof.SetGoroutineLabels(ctx)
}
var ti *traceInfo
if EnableTracing {
tr := newTrace("grpc.Recv."+methodFamily(stream.Method()), stream.Method())
Expand Down
Loading