Skip to content

feat(interceptor): Adds direct-pod routing for cold starts to reduce latency when kube-proxy rule propagation is slow. #1585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AtharvaPakade wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(interceptor): Adds direct-pod routing for cold starts to reduce latency when kube-proxy rule propagation is slow. #1585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,13 @@ This changelog keeps track of work items that have been completed and are ready

### New

- **General**: TODO ([#TODO](https://github.com/kedacore/http-add-on/issues/TODO))
- **Interceptor**: Add `KEDA_HTTP_DIRECT_POD_ROUTING` environment variable (`disabled` | `cold-start-only`). When set to `cold-start-only`, the interceptor routes cold-start requests directly to a ready pod IP instead of through the service ClusterIP, reducing latency when kube-proxy rules are slow to propagate. ([#1473](https://github.com/kedacore/http-add-on/issues/1473))

### Improvements

- **General**: TODO ([#TODO](https://github.com/kedacore/http-add-on/issues/TODO))
- **Interceptor**: TLS server name is captured in context by the routing middleware before any URL rewrites, so downstream transports always use the original service hostname for SNI. ([#1473](https://github.com/kedacore/http-add-on/issues/1473))
- **Interceptor**: `ReadyEndpointsCache` now tracks full `(ip, port)` pairs per named port from EndpointSlices, enabling direct-pod routing (replaces the previous bool-only ready state). ([#1473](https://github.com/kedacore/http-add-on/issues/1473))
- **Interceptor**: `TransportPool` now keys on `(responseHeaderTimeout, serverName)` and applies TLS `ServerName` per transport, enabling correct SNI when the upstream URL is rewritten to a pod IP. ([#1473](https://github.com/kedacore/http-add-on/issues/1473))

### Fixes

Expand Down
11 changes: 7 additions & 4 deletions config/crd/bases/http.keda.sh_interceptorroutes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,15 +195,18 @@ spec:
description: Backend service to route traffic to.
properties:
port:
description: Port number on the Service. Mutually exclusive with
portName.
description: |-
Port number on the Service. Mutually exclusive with portName.
Note: direct-pod routing (when enabled on the interceptor) requires portName;
routes using a numeric port will always be forwarded via the Service ClusterIP.
format: int32
maximum: 65535
minimum: 1
type: integer
portName:
description: Named port on the Service. Mutually exclusive with
port.
description: |-
Named port on the Service. Mutually exclusive with port.
Required for direct-pod routing to activate during cold starts.
minLength: 1
type: string
service:
Expand Down
26 changes: 25 additions & 1 deletion interceptor/config/serving.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,24 @@
package config

import (
"fmt"
"time"

"github.com/caarlos0/env/v11"
)

// DirectPodRoutingMode controls whether and when the interceptor routes requests
// directly to a ready pod IP instead of through the ClusterIP service.
type DirectPodRoutingMode string

const (
// DirectPodRoutingDisabled never bypasses the ClusterIP service (default).
DirectPodRoutingDisabled DirectPodRoutingMode = "disabled"
// DirectPodRoutingColdStartOnly bypasses the ClusterIP service only on cold
// starts, reducing latency when kube-proxy rules are slow to propagate.
DirectPodRoutingColdStartOnly DirectPodRoutingMode = "cold-start-only"
)

// Serving is configuration for how the interceptor serves the proxy
// and admin server
type Serving struct {
Expand Down Expand Up @@ -53,10 +66,21 @@ type Serving struct {
EnableColdStartHeader bool `env:"KEDA_HTTP_ENABLE_COLD_START_HEADER" envDefault:"true"`
// LogRequests enables/disables logging of incoming requests
LogRequests bool `env:"KEDA_HTTP_LOG_REQUESTS" envDefault:"false"`
// DirectPodRouting controls when the interceptor routes directly to a pod IP
// instead of the ClusterIP service. Valid values: "disabled", "cold-start-only".
DirectPodRouting DirectPodRoutingMode `env:"KEDA_HTTP_DIRECT_POD_ROUTING" envDefault:"disabled"`
Copy link
Copy Markdown
Member

@linkvt linkvt May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldnt this be a simple boolean? Internally we later only use a boolean as well.
Is there a reason for why we would use direct pod routing only on cold starts according to this config (didnt check the full PR yet)?

Copy link
Copy Markdown
Contributor Author

@AtharvaPakade AtharvaPakade May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Fedosin Suggested in the review that we should give option for different modes comment link section 2. Scope: cold-start-only vs. all requests

Copy link
Copy Markdown
Member

@linkvt linkvt May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks I missed that comment, quoting parts of it as I'm missing the "why":

it has a clear problem statement (kube-proxy lag)

To be honest this doesn't really explain why we should use the ClusterIP for warm requests and thus build a solution that is split between cold and warm requests? A single implementation just using Endpoint IPs seems simpler.
We also solve the issue, that existing pooled connections in the http.Transport are not targeting new pods. If we always connect to ClusterIP the TCP connection is kept open to the existing pods, new pods are only reached during new connections which is currently a problem.
knative also does not differentiate between cold and warm requests AFAIK.

limits blast radius

I'm not sure what is meant here with the blast radius, but two paths (=different cold and warm paths) increase complexity compared to just one.
If this direct pod routing is broken, users would disable it completely anyways, changing it from "always" to "cold-start-only" or such would still give them broken cold start requests, right?

Let me know if I misunderstood @Fedosin

I think we plan to enable it by default in the near future anyways, adding another step between us now and this goal is IMO not worth it which is why I see this as an on and off feature.

}

// MustParseServing parses standard configs and returns the
// newly created config. It panics if parsing fails.
func MustParseServing() Serving {
return env.Must(env.ParseAs[Serving]())
s := env.Must(env.ParseAs[Serving]())
switch s.DirectPodRouting {
case DirectPodRoutingDisabled, DirectPodRoutingColdStartOnly:
// valid
default:
panic(fmt.Sprintf("invalid KEDA_HTTP_DIRECT_POD_ROUTING value %q: must be %q or %q",
s.DirectPodRouting, DirectPodRoutingDisabled, DirectPodRoutingColdStartOnly))
}
Comment on lines +78 to +84
Copy link
Copy Markdown
Member

@linkvt linkvt May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be implemented directly close to the struct by implementing the TextMarshaller interface so that MustParseServing stays clean:

func (m *DirectPodRoutingMode) UnmarshalText(text []byte) error {
    switch DirectPodRoutingMode(text) {
    case DirectPodRoutingDisabled, DirectPodRoutingColdStartOnly:
        *m = DirectPodRoutingMode(text)
        return nil
    default:
        return fmt.Errorf("invalid value %q: must be %q or %q",
            text, DirectPodRoutingDisabled, DirectPodRoutingColdStartOnly)
    }
}

See https://github.com/caarlos0/env/blob/a72d89a8930fc800372a6a338a1acf33e5cc3a56/example_test.go#L212-L218

return s
}
2 changes: 1 addition & 1 deletion interceptor/handler/upstream.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ func (uh *Upstream) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
}

transport := uh.transportPool.Get(responseHeaderTimeout)
transport := uh.transportPool.Get(responseHeaderTimeout, util.UpstreamServerNameFromContext(ctx))

var rt http.RoundTripper = transport
if uh.tracingCfg.Enabled {
Expand Down
6 changes: 6 additions & 0 deletions interceptor/middleware/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
package middleware

const (
schemeHTTP = "http"
schemeHTTPS = "https"
)
29 changes: 24 additions & 5 deletions interceptor/middleware/endpoint_resolver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ const defaultFallbackReadinessTimeout = 30 * time.Second
type EndpointResolverConfig struct {
ReadinessTimeout time.Duration
EnableColdStartHeader bool
DirectPodOnColdStart bool // route to pod IP directly during cold start
}

type EndpointResolver struct {
Expand Down Expand Up @@ -64,7 +65,7 @@ func (er *EndpointResolver) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}

serviceKey := ir.Namespace + "/" + ir.Spec.Target.Service
isColdStart, err := er.readyCache.WaitForReady(waitCtx, serviceKey)
isColdStart, podHost, err := er.readyCache.WaitForReady(waitCtx, serviceKey, ir.Spec.Target.PortName)
if err != nil {
// No fallback, return an error
if !hasFallback {
Expand All @@ -90,12 +91,30 @@ func (er *EndpointResolver) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Fall back to alternate upstream.
fallbackURL := util.FallbackURLFromContext(ctx)
ctx = util.ContextWithUpstreamURL(ctx, fallbackURL)
// Update SNI to the fallback service hostname for TLS upstreams so the
// transport uses the correct server name instead of the primary service's.
// For non-TLS fallbacks the context may still hold the primary service's
// server name, but the transport ignores it for plain HTTP — no update needed.
if fallbackURL.Scheme == schemeHTTPS {
ctx = util.ContextWithUpstreamServerName(ctx, fallbackURL.Hostname())
}
r = r.WithContext(ctx)
}
} else {
// isColdStart is only meaningful when the backend resolved without errors
if er.cfg.EnableColdStartHeader {
w.Header().Set(kedahttp.HeaderColdStart, strconv.FormatBool(isColdStart))
}

// isColdStart is only meaningful when the backend resolved without errors
if err == nil && er.cfg.EnableColdStartHeader {
w.Header().Set(kedahttp.HeaderColdStart, strconv.FormatBool(isColdStart))
// Cold-start direct-to-pod routing: rewrites upstream to a pod IP, reducing latency when kube-proxy rules are slow to propagate.
// TLS SNI uses the original service hostname captured in context. Empty podHost leaves the upstream URL unchanged.
if isColdStart && er.cfg.DirectPodOnColdStart && podHost != "" {
if upstreamURL := util.UpstreamURLFromContext(ctx); upstreamURL != nil {
podURL := *upstreamURL
podURL.Host = podHost
ctx = util.ContextWithUpstreamURL(ctx, &podURL)
r = r.WithContext(ctx)
}
}
}

er.next.ServeHTTP(w, r)
Expand Down
Loading
Loading