improvement: Bump Salt to 3006.25 by TeddyAndrieux · Pull Request #4898 · scality/metalk8s

TeddyAndrieux · 2026-04-30T14:11:34Z

Salt 3006.25 upgrade

Bump Salt to 3006.25 (master image now uses the onedir-based RPM)
Drop the m2crypto package and the m2crypto Salt state — Salt 3006 ships cryptography-backed x509_v2, enabled via features: { x509_v2: true } on master and minion configs
Remove deprecated verbose parameter from x509 module calls
Remove no-longer-needed use_superseded on module.run states
Replace deprecated module.wait + watch pattern with module.run + onchanges
Drop the module.run Salt state where it has no remaining purpose

Salt master/minion compatibility during upgrade

Add minimum_auth_version: 2 on the master to accept v2-protocol minions during the rolling upgrade (to be removed in development/135)
Add reload_modules: False override on the salt-minion package install to skip the post-install module_refresh that fails when the running 3002 minion's files are replaced by the onedir layout
Add a dedicated salt-ssh ssh_pre_flight script (ssh-preflight.sh) that installs python3.12 and switches the python3 alternative on each target — Salt 3006 thin requires Python >= 3.7, RHEL/Rocky 8 ships 3.6
Switch the bootstrap script to install python3.12 and set the python3 alternative
Add dnf to the salt-master image so the yumpkg Salt module loads (its __virtual__ rejects microdnf-only environments and would otherwise disable pkg.* including pkg.version_cmp)

Certificate authority handling (upgrade path from old `x509`/m2crypto-generated certs)

Add a shared preserved_ski Jinja macro that pins subjectKeyIdentifier on x509.certificate_managed to the value of the existing CA cert (or hash on first install) — prevents x509_v2 from regenerating CAs with a new SKI that would invalidate every leaf cert's AKI, since cryptography and m2crypto compute the SKI differently for the same public key. Critical when upgrading clusters whose CAs were originally generated by the old x509 (m2crypto) module.
Apply the macro to all six CAs: kubernetes, etcd, front-proxy, dex, nginx-ingress, backup-server

Salt orchestration fixes

Replace state.orchestrate_single with state.orchestrate + a dedicated patch_kubesystem_namespace.sls — the _single runner has long tripped a ReferenceError: weakly-referenced object no longer exists on the post-state event firing, but in 3006 it now propagates as a non-zero exit code (was previously swallowed silently)
Use a dedicated state to patch the kube-system namespace cluster-version annotation
Create the salt-master kubeconfig after the highstate completes, not during it (kube-apiserver isn't ready yet during the bootstrap highstate)
Fix invalid require_in in deploy_node orchestrate
Add missing dependency on deploy core component objects in the upgrade orchestrate

Python and OS toolchain

Drop Python 3.6 support across the codebase: remove python36-rpm, python36-pyOpenSSL, python36-psutil package dependencies
Drop the Rocky Linux grains workaround that was needed for older Salt versions
Drop the no-longer-needed RedHat 7 logic in Salt states
Bump the buildchain to require Python 3.10+
Bump lib-alert-tree to Python 3.10
Bump devcontainer Python to 3.10.13; drop Python 2.7 and Python 3.6 from the devcontainer
Bump Python dependencies for tests, docs, salt tests
Remove the virtualenv < 20.22.0 pin in tox.ini (was needed for Python 3.6)

Build / CI / lint

Fix doit >= 0.37 compatibility: doit 0.37 dropped cloudpickle and uses stdlib pickle, which rejects local closures — refactor on_failure and title_* helpers in buildchain/buildchain/iso.py and buildchain/buildchain/utils.py to module-level functions (using functools.partial where they capture an argument)
Fix the docs build: change git config --global to git config --system in docs/entrypoint.sh so the tempuser the build sudo's to inherits the safe.directory setting and git describe no longer fails (which had silently set release = None and broken Sphinx)
Move docs / tests / salt-tests / buildchain pip-compile invocations from tox.ini to pre-commit hooks
Bump black to 26.3.1 (and apply formatting)
Bump pylint to v4.0.5 with associated dependencies
Bump mypy to v1.20.2 with associated dependencies; add a mypy config file
Bump yamllint to v1.38.0

Patch

A custom patch has been applied to make salt-ssh work with python3.12 (see [BUG] file.managed failed via salt-ssh, having python >=3.12 + cffi being installed saltstack/salt#68080)

Fixes: #3436 MK8S-251

TODO:

Target development/134.0 once available
Add changelog entry
Bump to latest 3006.X (current one has warning on every salt-call call see -> [Bug]: 3006.24 regression: AttributeError: 'NoneType' object has no attribute 'set_result' in transport/ipc.py saltstack/salt#68993)

bert-e · 2026-04-30T14:11:39Z

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-04-30T14:11:47Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-05-04T05:13:35Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-05-06T11:28:22Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-05-07T08:34:59Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

eg-ayoub · 2026-05-11T12:51:57Z

+      glibc-all-langpacks langpacks-en \
+ && salt-pip install --no-cache-dir --upgrade pip \
+ # Pin protobuff version since newer versions does not work with etcd3
+ && salt-pip install --no-cache-dir "protobuf <= 3.20.1" "etcd3 != 0.11.0" "kubernetes ~= 33.1.0" \


we should probably update some kind of internal doc somewhere to remember to always bump kubernetes here,
or add a check somewhere.
I see us potentially forgetting this quite often.

Agree, ideally it should be handled by renovate but not linked to this PR

eg-ayoub

Is it possible to push this a little further and require 3.12 in tox and the devcontainer as well ?

bert-e · 2026-05-11T16:00:32Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2026-05-12T11:53:57Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-05-12T18:44:50Z

Conflict

There is a conflict between your branch improvement/bump-salt-3006 and the
destination branch development/134.0.

Please resolve the conflict on the feature branch (improvement/bump-salt-3006).

git fetch && \
git checkout origin/improvement/bump-salt-3006 && \
git merge origin/development/134.0

Resolve merge conflicts and commit

git push origin HEAD:improvement/bump-salt-3006

bert-e · 2026-05-13T05:47:11Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

We also fix some new pylint warnings, and regenerate fresh pylintrc files

We also fix some new mypy warnings, and add a mypy configuration file

…commit

…commit We also remove the platform-requirements.txt file, as it is no longer needed.

Also bump dependencies versions

We also fix tests and tox configuration to works with newer dependencies. Plus we force color in CI runs

We also remove the deprecated assertDictContainsSubset call.

This one was needed to build containerd rpm but we no longer build it ourselves.

salt-ssh's `SSH.__init__` calls `_expand_target`, which when given a single non-glob target that resolves to a reachable host calls `_get_roster` -> `salt.roster.get_roster_file`. That helper requires the configured roster file (default `/etc/salt/roster`) to exist on disk and be readable, even when a non-flat backend like `kubernetes` is selected via `roster: kubernetes` - the file is only stat'd, never read. When no roster file is present, `salt-ssh --roster=kubernetes <single-host> ...` fails with `OSError: Roster file "/etc/salt/roster" not found`. Most invocations don't trip this because they use a glob (`salt-ssh '*' ...`) or target an unreachable host - both bypass the expansion path that hits `_get_roster`. Drop a zero-byte `/etc/salt/roster` into the salt config so the check passes regardless of how operators invoke salt-ssh against the cluster.

NOTE: Due to a bug in salt 3006, we need to patch the salt code to fix salt-ssh compatibility with Python 3.12.

Due to the major bump of Salt and Python version, downgrading from 134 to 133 will not be possible.

bert-e · 2026-05-18T14:27:35Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-05-18T16:19:23Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from b8b4c73 to cc0d369 Compare May 5, 2026 09:39

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from 0ae2995 to 0cc092e Compare May 7, 2026 08:34

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch 3 times, most recently from 486d1a9 to bf2a0cc Compare May 11, 2026 11:40

eg-ayoub reviewed May 11, 2026

View reviewed changes

Comment thread salt/_states/metalk8s.py Outdated

eg-ayoub reviewed May 11, 2026

View reviewed changes

Comment thread tests/post/features/solutions.feature

eg-ayoub reviewed May 11, 2026

View reviewed changes

Comment thread tests/post/steps/test_solutions.py Outdated

eg-ayoub reviewed May 11, 2026

View reviewed changes

Comment thread salt/metalk8s/addons/dex/certs/server.sls

eg-ayoub previously approved these changes May 11, 2026

View reviewed changes

TeddyAndrieux dismissed eg-ayoub’s stale review via bf78d76 May 11, 2026 13:32

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from bf2a0cc to bf78d76 Compare May 11, 2026 13:32

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from bf78d76 to bd491ff Compare May 12, 2026 11:53

TeddyAndrieux changed the base branch from development/133.0 to development/134.0 May 12, 2026 11:53

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch 4 times, most recently from 9bc6414 to 26bffb0 Compare May 12, 2026 15:59

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from 4e75dcf to 20f3d42 Compare May 13, 2026 05:47

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from 20f3d42 to 516cabf Compare May 13, 2026 10:19

TeddyAndrieux added 27 commits May 18, 2026 14:26

refactor(lint): bump pylint to v4.0.5 (and associated dependencies)

160e4bc

We also fix some new pylint warnings, and regenerate fresh pylintrc files

refactor(lint): bump mypy to v1.20.2 (add associated dependencies)

0a10d8e

We also fix some new mypy warnings, and add a mypy configuration file

refactor(lint): bump yamllint to v1.38.0

98b4e38

refactor(pip-tools): move docs pip-compile from tox.ini to pre-commit

f8b97bc

refactor(pip-tools): move tests pip-compile from tox.ini to pre-commit

f4c8ccf

refactor(pip-tools): move salt tests pip-compile from tox.ini to pre-…

217df38

…commit

refactor(pip-tools): move buildchain pip-compile from tox.ini to pre-…

6a6a340

…commit We also remove the platform-requirements.txt file, as it is no longer needed.

refactor(tox.ini): remove no longer needed pin of virtualenv

48a4fb1

refactor(devcontainer): remove python2.7 installation

ff36333

refactor(devcontainer): remove python3.6 installation

313ce1b

Also bump dependencies versions

refactor(devcontainer): bump python version to 3.10.13

a2ff1c9

fix(docs): Fix entrypoint.sh git config safe directory

167b539

refactor(lib-alert-tree): Bump Python version to 3.10

ea8bca5

refactor(tests): Bump python dependencies

0690e0d

We also fix tests and tox configuration to works with newer dependencies. Plus we force color in CI runs

refactor(docs): Bump python dependencies

7f7ba46

refactor(salt/tests): Bump python dependencies

be39e84

We also remove the deprecated assertDictContainsSubset call.

refactor(scripts): Use python3.9 by default in bootstrap script

a168a98

refactor(packages): remove python36 from redhat/8 Dockerfile

d97f962

refactor(packages): remove go from redhat/8 Dockerfile

808eac4

This one was needed to build containerd rpm but we no longer build it ourselves.

refactor(.github): switch ovh flavor to c3-16

e0eed1f

fix(salt): add timeout to metalk8s_solutions pillar call

97e42c7

feat: bump python version installed by default to 3.12

8ca4d13

NOTE: Due to a bug in salt 3006, we need to patch the salt code to fix salt-ssh compatibility with Python 3.12.

tests: ensure ingress pods are ready before testing to avoid flakiness

816688a

ci(.github): disable downgrade from 134 to 133

8d2c5df

Due to the major bump of Salt and Python version, downgrading from 134 to 133 will not be possible.

docs(CHANGELOG): add entry for Salt bump and downgrade limitation

9b64a03

fix(tests): filter out FluentBit alerts if fluent-bit is disabled

e5927ba

TeddyAndrieux force-pushed the improvement/bump-salt-3006 branch from 682e63b to e5927ba Compare May 18, 2026 14:27

Conversation

TeddyAndrieux commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Salt 3006.25 upgrade

Salt master/minion compatibility during upgrade

Certificate authority handling (upgrade path from old x509/m2crypto-generated certs)

Salt orchestration fixes

Python and OS toolchain

Build / CI / lint

Patch

Uh oh!

bert-e commented Apr 30, 2026

Hello teddyandrieux,

Uh oh!

bert-e commented Apr 30, 2026

Waiting for approval

Uh oh!

bert-e commented May 4, 2026

Waiting for approval

Uh oh!

bert-e commented May 6, 2026

Waiting for approval

Uh oh!

bert-e commented May 7, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eg-ayoub May 11, 2026

Choose a reason for hiding this comment

Uh oh!

TeddyAndrieux May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eg-ayoub left a comment

Choose a reason for hiding this comment

Uh oh!

bert-e commented May 11, 2026

Request integration branches

Uh oh!

bert-e commented May 12, 2026

Waiting for approval

Uh oh!

bert-e commented May 12, 2026

Conflict

Uh oh!

bert-e commented May 13, 2026

Waiting for approval

Uh oh!

bert-e commented May 18, 2026

Waiting for approval

Uh oh!

bert-e commented May 18, 2026

Waiting for approval

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TeddyAndrieux commented Apr 30, 2026 •

edited

Loading

Certificate authority handling (upgrade path from old `x509`/m2crypto-generated certs)