Fixed upgrade flow for KMM-hub

⏫ [email protected] can now be upgraded to [email protected].

Upgrading from KMM-hub 2.3.0 to 2.4.0 was not working properly, therefore, [email protected] was not published to operatorhub.io.
The 2.4.1 release was created to address this issue.

Community members who wish to upgrade from 2.3 to 2.4 should upgrade to 2.4.1.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM.

Using make

git fetch
git checkout v2.4.1

# For KMM
make deploy IMAGE_TAG=v20250710-v2.4.1

Full Changelog: v2.4.0...v2.4.1

Main new features

🎄 Support for loading a device-plugin for an in-tree driver.

KMM can now only set a device-plugin for an already loaded, in-tree driver, in the Module.

📦 Added an init-container to the device-plugin.

Allows setup validation before running the device plugin.

🏋️ Operator configuration will now persist operator upgrades.

Custom configuration for the operator via a configMap will now persist to the next KMM version in case on an upgrade.

Additional enhancements

• The operator is now pulling and checking images existence using the cluster's container-runtime.
• KMM can now be installed on worker nodes, when control-plane nodes aren't available, without the need to artificially label the worker nodes as "control-plane".
• Reduce significantly the number of events we had to one of the internal controller (NMC)
• Removed a service duplication (generated by OLM) for the webhook deployment.

Bug Fixes

• Fixed a bug when KMM was not re-loading the kmod when a reboot was very fast (and the kube-api was not noticing that the node even went down).
• Fixed a bug that the NMC CR was inheriting tolerations that were not set in the Module
• Fixed a bug in which a failing worker-pod was hanging forever if its Module was deleted.

Installing

⚠️ This version only contain KMM 2.4.0 and doesn't contain KMM-hub 2.4.0 because the upgrade path for KMM-hub 2.3.0-->2.4.0 is broken.
We are working on a fix and will release KMM + KMM-hub 2.4.1 soon to address this issue.

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM.

Using make

git fetch
git checkout v2.4.0

# For KMM
make deploy IMAGE_TAG=v20250626-v2.4.0

Full Changelog: v2.3.0...v2.4.0

New features

🚫 Added support for user defined tolerances to modules.

By adding tolerations to a `Modules, the workers pods can be scheduled on a tainted node. This is required in order to manage a kmod upgrade when the the method used to drain the node is by tainting it.

🥇 Golang was bumped to 1.23.

Bug fixes

Fixed a bug in which nodes were keeping their kmods ready labels after the kmods were removed from the node.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.3.0

# For KMM
make deploy IMAGE_TAG=v20250310-v2.3.0

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20250310-v2.3.0

Full Changelog: v2.2.1...v2.3.0

Bug fixes

⏫ Fixed an issue preventing the upgrade from v2.1.x when an module is present in the cluster during the upgrade.

The ImagePullPolicy had a kubebuilder instruction for getting a default value, which automatically made it a required field.

In previous versions, this field was optional, meaning that when upgrading the KMM operator, the upgrade would fail , since the NMC object is missing the required field.

This fix kept the default value, but made it optional.

🏎️ Fixed a race condition between a node reboot and the deletion of the KMM module.

In case a node was rebooted (for any reason) and the Module was being deleted at the same period of time, we would have a race condition preventing the Module from being finalized.

This issue is now fixed.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.2.1

# For KMM
make deploy IMAGE_TAG=v20241223-v2.2.1

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20241223-v2.2.1

Full Changelog: v2.2.0...v2.2.1

Improvements / Bug fixes

💻 Worker pod now pull images using the cluster's container-runtime.

By using the cluster's infrastructure we ensure that all the cluster-wide configuration applied that are respected by kubelet are also respected by KMM.

🚫 New validation in the validation webhook

The validation webhook will now ensure that all container images explicitly contain a tags/sha.

💾 Setting the firmware path is now configurable.

It is now possible to configure the path on the host in which the Module's firmware files will be written in the controller's ConfigMap.
The default value is /lib/firmware.

🐛 Fixed a bug

Fixed a bug regarding in-cluster-build modules left in invalid state after upgrading the cluster.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.2.0

# For KMM
make deploy IMAGE_TAG=v20241121-v2.2.0

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20241121-v2.2.0

Full Changelog: v2.1.1...v2.2.0

Bug fixes

🎮 Smaller footprint in managed mode.

The following controllers are not started in managed mode (KMM_MANAGED=1) anymore:

• BuildSignReconciler
• PreflightValidation

In managed mode, we assume compute-intensive tasks are run by the Hub, hence those controllers serve no purpose and will not be started anymore.

➕ Normalize kernel version in labels and image tags

The kernel version can contain some special characters like + that cannot be used neither in a Kubernetes label value nor in a container image tag.
KMM will now convert all those characters to an underscore _ so that this value can be properly used internally as a resource labels or externally as container image tag.
The kernel mapping logic does not change; in literal or regexp fields, you should still target the kernel version as reported by the kubelet (in the Node resource's .status.nodeInfo.kernelVersion field).

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.1.1

# For KMM
make deploy IMAGE_TAG=v20240618-v2.1.1

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20240618-v2.1.1

Full Changelog: v2.1.0...v2.1.1

New features & improvements

⏳ Optional delay for the garbage collection of build pods

The new job.gcDelay operator setting allows specifying a duration for which successful build & signing pods should be kept before they are garbage-collected.

🛂 Separate deployment for the webhook server

The webhook server is now running as a separate Deployment.

📜 CRD changes

Module

Added inTreeModulesToRemove to allow specifying a list of in-tree modules to be removed before the main out-of-tree module is loaded.
The old inTreeModuleToRemove field is still present, but deprecated.

PreflightValidation

Added version v1beta2 with a new status subresource that is compliant with OpenAPI guidelines.
Version v1beta1 is still served.

🧹 Miscellaneous

cert-manager is not a required dependency anymore when KMM is instsalled via OLM.
Images are now built with Go 1.22.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.1.0

# For KMM
make deploy IMAGE_TAG=v20240425-v2.1.0

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20240425-v2.1.0

Changelog

New features & improvements

🛂 Webhook for namespace deletion

A validating webhook will now verify that namespaces do not contain any Module resource before they can be deleted.
This avoids entering situations where the namespace is being deleted and KMM cannot create unloading Pods to honor Module deletion.

📜 CRD changes

• Module: make moduleName an optional field
• ManagedClusterModule: make spokeNamespace a required field

🧹 Miscellaneous

Bug fixes.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout v2.0.2

# For KMM
make deploy IMAGE_TAG=v20240306-v2.0.2

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20240306-v2.0.2

Changelog

Full Changelog: v2.0.1...v2.0.2

New features & improvements

🔗 Symlink support

The worker pod now extracts symbolic links from the kmod image. This means that Dockerfiles can now create symlinks from /opt/lib/modules/${kernelVersion}/host to /lib/modules/${kernelVersion} prior to running depmod to have it figure out dependencies on in-tree kmods from the host's filesystem.

🔧 New selector for the controller

The KMM operator will now run on nodes labeled with kmm.node.kubernetes.io/control-plane: ''.
This should be especially helpful in clusters without master nodes, such as HyperShift, GKE, AKS or EKS.

🧹 Miscellaneous

Bug fixes.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout release-2.0

# For KMM
make deploy IMAGE_TAG=v20240131-v2.0.1

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20240131-v2.0.1

Changelog

New Contributors

• @k8s-infra-cherrypick-robot made their first contribution in #685

Full Changelog: v2.0.0...v2.0.1

New features & improvements

👷🏻 Worker Pods

The operator now creates short-lived, standalone worker Pods to load kmods instead of long-running DaemonSets.
This change improves the reliability of kmod unloads and significantly reduces resource utilization on both worker nodes and the control plane.

ℹ️ Events

The KMM operator now emits events:

• on build & signing job creation, completion or failure (attached to the Module);
• on kmod load or unload (attached to the Node).

💾 Binary firmwares

The firmware_class.path kernel parameter, which configures an alternate firmware lookup path, can now be set on all nodes before the kmod is loaded.
This feature is disabled by default and must be enabled in the operator configuration.

Installing

Using OLM (recommended)

Follow the installations instructions at OperatorHub.io for KMM or KMM-Hub.

Using make

git fetch
git checkout release-2.0

# For KMM
make deploy IMAGE_TAG=v20231130-v2.0.0

# For KMM-Hub
make deploy-hub IMAGE_TAG=v20231130-v2.0.0

Changelog