security: disable anonymous auth on kube-apiserver (MK8S-187)#4900
security: disable anonymous auth on kube-apiserver (MK8S-187)#4900g-carre wants to merge 7 commits into
Conversation
kube-apiserver was running with the default --anonymous-auth=true, letting unauthenticated callers reach endpoints bound to system:public-info-viewer (e.g. /version, /healthz) and disclose Kubernetes/Go versions usable for further attack planning. Set --anonymous-auth=false so the system:anonymous user cannot authenticate at all, which neutralises the default kubeadm-managed ClusterRoleBindings granting access to system:unauthenticated. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Bake the security advisory's diagnostic into the post-deploy BDD suite so a regression on --anonymous-auth is caught by CI: hit the advisory's exact endpoint (/api/kubernetes/version through the control-plane ingress, which proxies to kube-apiserver) without credentials and expect a 401 Unauthorized. Reuses existing 'perform a request on ... on control-plane Ingress' and 'the server returns ... with message ...' steps; no new step code. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Hello g-carre,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
|
/create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
…K8S-187) Disabling --anonymous-auth in commit f39d88d caused bootstrap to hang on the salt http.wait_for_successful_query probes that hit /healthz on kube-apiserver: the unauthenticated GET now returns 401, the state times out after 5 minutes, and every dependent orchestration step fails with "One or more requisite failed". Pass the apiserver-kubelet client cert and key (CN kube-apiserver-kubelet-client, O system:masters) to all seven affected probes -- the direct one on https://<host>:6443/healthz in kubernetes/apiserver/installed.sls, and the six going through the local apiserver-proxy on https://127.0.0.1:7443/healthz in the bootstrap, deploy_node, upgrade and downgrade orchestrations. The salt-master Pod already mounts /etc/kubernetes/pki, and the cert exists on every master before kube-apiserver starts, so no extra plumbing is needed. Expose apiserver-kubelet-client.key as certificates.client.files['apiserver-kubelet'].key in defaults.yaml and switch existing literal references in apiserver/installed.sls and apiserver/certs/kubelet-client.sls to it, to keep the path in a single place. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
There was a problem hiding this comment.
Pull request overview
This PR hardens the Kubernetes control plane by restricting kube-apiserver anonymous authentication to health endpoints only, while ensuring Salt orchestration health probes and post-install tests continue to function.
Changes:
- Add a kube-apiserver
AuthenticationConfigurationto allow anonymous access only to/livez,/readyz, and/healthz, and wire it into the apiserver static pod manifest. - Update Salt
http.wait_for_successful_queryprobes (bootstrap/deploy/upgrade/downgrade and apiserver install readiness checks) to use the existing apiserver-kubelet client certificate. - Extend post-install authentication BDD tests to assert
/api/kubernetes/versionis rejected anonymously while health endpoints remain accessible (anonymous via Ingress; authenticated directly to apiserver).
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/post/steps/test_authentication.py | Adds new pytest-bdd scenario bindings and a step to perform mTLS-authenticated requests directly against the apiserver. |
| tests/post/features/authentication.feature | Adds scenarios covering anonymous rejection of /version, anonymous allowance of health endpoints via Ingress, and authenticated health endpoint access directly. |
| salt/metalk8s/orchestrate/upgrade/init.sls | Authenticates apiserver availability checks during upgrade using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/downgrade/init.sls | Authenticates apiserver availability checks during downgrade using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/deploy_node.sls | Authenticates apiserver availability checks before/after highstate using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/bootstrap/pre-upgrade.sls | Authenticates the local proxy apiserver availability check prior to master upgrade. |
| salt/metalk8s/orchestrate/bootstrap/pre-downgrade.sls | Authenticates the local proxy apiserver availability check prior to master downgrade. |
| salt/metalk8s/orchestrate/bootstrap/init.sls | Authenticates bootstrap apiserver availability checks using the apiserver-kubelet client cert. |
| salt/metalk8s/kubernetes/apiserver/installed.sls | Mounts and passes the new authentication config into kube-apiserver; switches kubelet client key path to the centralized defaults map; authenticates readiness probe. |
| salt/metalk8s/kubernetes/apiserver/init.sls | Includes the new authnconfig sub-state in the apiserver state bundle. |
| salt/metalk8s/kubernetes/apiserver/certs/kubelet-client.sls | Switches the kubelet-client private key path to the centralized defaults map. |
| salt/metalk8s/kubernetes/apiserver/authnconfig.sls | New state generating /etc/kubernetes/authentication-config.yaml restricting anonymous auth to health endpoints only. |
| salt/metalk8s/defaults.yaml | Adds the apiserver-kubelet client key path into the certificates.client.files map for centralized reuse. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
61d346a to
15c0da2
Compare
History mismatchMerge commit #1b0256add37d05f04c884aa2f6a8b0f5531c24ed on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
15c0da2 to
10e4ed5
Compare
History mismatchMerge commit #68f2a0394cd43770668080f6bc58eaa70f2ac9b6 on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
10e4ed5 to
6132e5c
Compare
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
1cb9435 to
250a7d0
Compare
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
* Centralize the authn config path in `kube_api.authn_config_path` (defaults.yaml) and consume it from both authnconfig.sls and installed.sls instead of duplicating the literal. * Inline the AuthenticationConfiguration dataset in authnconfig.sls as templated YAML rather than building a Jinja dict + tojson, for readability. * Drop the now-unused `cert:` (mTLS client cert) argument from the healthz http.wait_for_successful_query checks across the apiserver and orchestrate states, along with the matching `certificates` imports that became dead.
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
3af8de3 to
18e87cc
Compare
History mismatchMerge commit #3af8de35e85982271810784d7765ba1551c4fd6f on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
28d254b to
0459545
Compare
History mismatchMerge commit #28d254b1b33ea66fed53ec90242935a20b32371b on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Component: salt, tests
Context:
A security assessment flagged that the ARTESCA Kubernetes cluster permits anonymous access to the API server. With kube-apiserver running on its default
--anonymous-auth=true, unauthenticated callers can reach endpoints bound to the kubeadm-managedsystem:public-info-viewerClusterRoleBinding (e.g./version,/healthz) and disclose Kubernetes / Go versions usable for further attack planning.Reproducer (from the advisory):
curl -k https://<INSTANCE_IP>:8443/api/kubernetes/versionreturns a JSON body withgitVersion/goVersioninstead of401.Summary:
AuthenticationConfiguration(/etc/kubernetes/authentication-config.yaml, generated from a new sibling salt state mirroringcryptconfig.sls) that scopes anonymous access to/livez,/readyz,/healthzonly.--anonymous-auth=falsewas tried first but is incompatible with kubelethttpGetprobes (which cannot carry credentials) and would put the apiserver in a startup-probe crash-loop after ~250s; it is also mutually exclusive with theAuthenticationConfiguration.anonymousblock, so it is dropped.--oidc-*CLI flags intoAuthenticationConfiguration.jwt. K8s 1.32 rejects the two mechanisms together (pkg/kubeapiserver/options/authentication.go: "authentication-config file and oidc- flags are mutually exclusive"*). A first apiserver start can succeed before the control-plane Ingress endpoint is known (no--oidc-*flags emitted), but the subsequent Reconfigure control plane Ingress pass populatesoidc_config, re-renders the manifest with both flag sets, and the apiserver crash-loops on startup. The same five fields the legacy code consumed (issuerURL,clientID,CAFile,usernameClaim,groupsClaim) feed the newjwtissuer block. The CA PEM is read from the salt mine entryingress_ca_b64for the default Dex / Ingress path (avoiding any state-graph ordering hazard), with asalt['file.read']fallback for a pillar-supplied OIDC override pointing at a different CA path. When the username claim isemail, aclaimValidationRulesCEL expressionclaims.?email_verified.orValue(true)is added to reproduce the implicitemail_verified == trueguard that--oidc-username-claim=emailused to apply automatically. ARTESCA configures Keycloak via the same OIDC pillar override (installer/artesca_installer/salt/artesca/base/keycloak/configured.sls), so its reconfiguration path is unaffected.AnonymousAuthConfigurableEndpointsandStructuredAuthenticationConfigurationfeature gates, both beta and on-by-default in Kubernetes 1.32 (perpkg/features/versioned_kube_features.goinrelease-1.32). No--feature-gatesflag needed for the version metalk8s pins.http.wait_for_successful_queryprobes that hit/healthzon the apiserver (one direct, six through the local apiserver-proxy:7443raw stream) using the existingapiserver-kubelet-client.{crt,key}(CNkube-apiserver-kubelet-client, groupsystem:masters). Defence-in-depth: those probes still succeed even though/healthzis now anonymous-allowed, and the cert pattern matches what the etcd state already does.apiserver-kubelet-client.keyascertificates.client.files['apiserver-kubelet'].keyinsalt/metalk8s/defaults.yamland switch the existing literal references inapiserver/installed.slsandapiserver/certs/kubelet-client.slsto it, to keep the path in a single place.tests/post/features/authentication.feature:GET /api/kubernetes/version→401 Unauthorized. Still passes —/versionis not in the allowed anonymous list.GET /api/kubernetes/{livez,readyz,healthz}via the control-plane Ingress →200 ok.GET /{livez,readyz,healthz}directly on the API server (admin client cert pulled from the kubeconfig) →200 ok.Acceptance criteria:
kubectl, salt, kubelet → apiserver, and the bootstrap/post-install flow continue to work end-to-end (single-node and multi-node e2e green).curl -k https://<INSTANCE_IP>:8443/api/kubernetes/versionreturns401, bodyUnauthorized.curl -k https://<INSTANCE_IP>:8443/api/kubernetes/livez(and/readyz,/healthz) returns200, bodyok.kube-apiserver …scenarios intests/post/features/authentication.feature.Follow-ups:
apiVersionfromapiserver.config.k8s.io/v1beta1tov1once metalk8s pins Kubernetes ≥ 1.34 (no urgency — v1beta1 still served through 1.36+).See: MK8S-187