KubeVirtRelieveAndMigrate: add KubevirtMigrationAware plugin support

README.md

@@ -211,10 +211,13 @@ this asymmetry and foster a quicker convergence.
 This profile requires [PSI](https://docs.kernel.org/accounting/psi.html) metrics to be enabled (psi=1 kernel parameter)
 for all the worker nodes.
 
-The profile exposes the following customization:
+The profile exposes the following customizations:
 - `devLowNodeUtilizationThresholds`: Sets experimental thresholds for the LowNodeUtilization strategy.
 - `devActualUtilizationProfile`: Enable load-aware descheduling.
 - `devDeviationThresholds`: Have the thresholds be based on the average utilization.
+- `devMigrationCooldown`: Base cooldown after a VM migration before the same VM is eligible for re-eviction.
+- `devMaxMigrationCooldown`: Upper bound on the exponential backoff cooldown between successive migrations.
+- `devMigrationHistoryWindow`: Sliding window over which past migrations are counted for backoff calculation.
 
 By default, this profile will enable load-aware descheduling based on the `PrometheusCPUMemoryCombinedProfile` Prometheus query. That query is based on a recording rule combining the impact of CPU and memory utilization and PSI pressure.
 By default, the thresholds will be dynamic (based on the distance from the average utilization) and asymmetric (all the nodes below the average will be considered as underutilized to help rebalancing overutilized outliers) tolerating low deviations (10%).
@@ -225,6 +228,13 @@ Those two values can be customized with `evictionLimits.total` and `evictionLimi
 
 This profile configures the `DefaultEvictor` plugin with a `Mandatory noEvictionPolicy`. As a result, the `descheduler.alpha.kubernetes.io/prefer-no-eviction` annotation can be dynamically applied to VM templates and/or VMIs to exclude them from the descheduler's scope.
 
+This profile also enables the `KubevirtMigrationAware` plugin as both a filter and a pre-eviction filter.
+As a filter, it hard-blocks eviction of any pod whose VMI has a live migration already in progress.
+As a pre-eviction filter, it applies an adaptive cooldown after each migration completes, using exponential
+backoff bounded by a configurable maximum, to prevent cascading re-evictions of the same VM.
+The cooldown parameters can be tuned with the `devMigrationCooldown`, `devMaxMigrationCooldown`, and
+`devMigrationHistoryWindow` profile customizations.
+
 ### EvictPodsWithPVC
 By default, the operator prevents pods with PVCs from being evicted. Enabling this
 profile in combination with any of the above profiles allows pods with PVCs to be
@@ -252,6 +262,9 @@ the `profileCustomizations` field:
 | `devHighNodeUtilizationThresholds` | `string` | Sets thresholds for the [HighNodeUtilization](https://github.com/kubernetes-sigs/descheduler#highnodeutilization) strategy of the `CompactAndScale` profile in the following ratios: `Minimal` for 10%, `Modest` for 20%, `Moderate` for 30%. Currently provided as an experimental feature.|
 |`devActualUtilizationProfile`|`string`| Sets a profile that gets translated into a predefined prometheus query |
 | `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization. An AsymmetricDeviationThreshold will force all nodes below the average to be considered as underutilized to help rebalancing overutilized outliers. Supported settings: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30%, `AsymmetricLow`: 0%:10%, `AsymmetricMedium`: 0%:20%, `AsymmetricHigh`: 0%:30% |
+| `devMigrationCooldown` | `duration` | Sets the base cooldown period for the `KubevirtMigrationAware` plugin of the `KubeVirtRelieveAndMigrate` profile. After a VM migration completes, the plugin will not evict that VM again for at least this duration (subject to exponential backoff). Only applicable to the `KubeVirtRelieveAndMigrate` profile. |
+| `devMaxMigrationCooldown` | `duration` | Sets the upper bound on the exponential backoff cooldown for the `KubevirtMigrationAware` plugin of the `KubeVirtRelieveAndMigrate` profile. Regardless of how many migrations have occurred within the history window, the cooldown will not exceed this value. Only applicable to the `KubeVirtRelieveAndMigrate` profile. |
+| `devMigrationHistoryWindow` | `duration` | Sets the sliding time window used by the `KubevirtMigrationAware` plugin to count past migrations when computing the exponential backoff cooldown. Only applicable to the `KubeVirtRelieveAndMigrate` profile. |
 
 ## Prometheus query profiles
 The operator provides the following profiles:

bindata/assets/kube-descheduler/crdschema.yaml

@@ -137,6 +137,24 @@ properties:
               - High
               - ""
             type: string
+          devMaxMigrationCooldown:
+            description: |-
+              devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin.
+              Only applicable for the KubeVirtRelieveAndMigrate profile.
+            format: duration
+            type: string
+          devMigrationCooldown:
+            description: |-
+              devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin.
+              Only applicable for the KubeVirtRelieveAndMigrate profile.
+            format: duration
+            type: string
+          devMigrationHistoryWindow:
+            description: |-
+              devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin.
+              Only applicable for the KubeVirtRelieveAndMigrate profile.
+            format: duration
+            type: string
           namespaces:
             description: |-
               Namespaces overrides included and excluded namespaces while keeping

bindata/assets/kube-descheduler/operandclusterrole.yaml

@@ -63,3 +63,11 @@ rules:
   - "get"
   - "patch"
   - "delete"
+- apiGroups:
+  - "kubevirt.io"
+  resources:
+  - "virtualmachineinstances"
+  verbs:
+  - "get"
+  - "list"
+  - "watch"

deploy/02_clusterrole.yaml

@@ -208,3 +208,11 @@ rules:
       - patch
       - delete
       - deletecollection
+  - apiGroups:
+      - kubevirt.io
+    resources:
+      - virtualmachineinstances
+    verbs:
+      - get
+      - list
+      - watch

manifests/cluster-kube-descheduler-operator.clusterserviceversion.yaml

@@ -356,6 +356,14 @@ spec:
                 - patch
                 - delete
                 - deletecollection
+            - apiGroups:
+                - kubevirt.io
+              resources:
+                - virtualmachineinstances
+              verbs:
+                - get
+                - list
+                - watch
       deployments:
         - name: descheduler-operator
           spec:

manifests/kube-descheduler-operator.crd.yaml

@@ -153,6 +153,24 @@ spec:
                         - High
                         - ""
                       type: string
+                    devMaxMigrationCooldown:
+                      description: |-
+                        devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin.
+                        Only applicable for the KubeVirtRelieveAndMigrate profile.
+                      format: duration
+                      type: string
+                    devMigrationCooldown:
+                      description: |-
+                        devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin.
+                        Only applicable for the KubeVirtRelieveAndMigrate profile.
+                      format: duration
+                      type: string
+                    devMigrationHistoryWindow:
+                      description: |-
+                        devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin.
+                        Only applicable for the KubeVirtRelieveAndMigrate profile.
+                      format: duration
+                      type: string
                     namespaces:
                       description: |-
                         Namespaces overrides included and excluded namespaces while keeping

pkg/apis/descheduler/v1/types_descheduler.go

@@ -127,6 +127,21 @@ type ProfileCustomizations struct {
 	// devDeviationThresholds enables dynamic thresholds based on average resource utilization
 	// +kubebuilder:validation:Enum=Low;Medium;High;AsymmetricLow;AsymmetricMedium;AsymmetricHigh;""
 	DevDeviationThresholds *DeviationThresholdsType `json:"devDeviationThresholds,omitempty"`
+
+	// devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin.
+	// Only applicable for the KubeVirtRelieveAndMigrate profile.
+	// +kubebuilder:validation:Format=duration
+	DevMigrationCooldown *metav1.Duration `json:"devMigrationCooldown,omitempty"`
+
+	// devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin.
+	// Only applicable for the KubeVirtRelieveAndMigrate profile.
+	// +kubebuilder:validation:Format=duration
+	DevMaxMigrationCooldown *metav1.Duration `json:"devMaxMigrationCooldown,omitempty"`
+
+	// devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin.
+	// Only applicable for the KubeVirtRelieveAndMigrate profile.
+	// +kubebuilder:validation:Format=duration
+	DevMigrationHistoryWindow *metav1.Duration `json:"devMigrationHistoryWindow,omitempty"`
 }
 
 type LowNodeUtilizationThresholdsType string

pkg/operator/target_config_reconciler.go

@@ -77,6 +77,7 @@ const psiPath = "/proc/pressure/"
 const EXPERIMENTAL_DISABLE_PSI_CHECK = "EXPERIMENTAL_DISABLE_PSI_CHECK"
 const defaultKVParallelMigrationsPerCluster = 5
 const defaultKVParallelOutboundMigrationsPerNode = 2
+const kubevirtMigrationAwarePluginName = "KubevirtMigrationAware"
 
 // deschedulerCommand provides descheduler command with policyconfigfile mounted as volume and log-level for backwards
 // compatibility with 3.11
@@ -1097,6 +1098,12 @@ func kubeVirtRelieveAndMigrateProfile(profileCustomizations *deschedulerv1.Profi
 			Filter: v1alpha2.PluginSet{
 				Enabled: []string{
 					defaultevictor.PluginName,
+					kubevirtMigrationAwarePluginName,
+				},
+			},
+			PreEvictionFilter: v1alpha2.PluginSet{
+				Enabled: []string{
+					kubevirtMigrationAwarePluginName,
 				},
 			},
 			Balance: v1alpha2.PluginSet{
@@ -1159,6 +1166,25 @@ func kubeVirtRelieveAndMigrateProfile(profileCustomizations *deschedulerv1.Profi
 		}
 	}
 
+	{
+		var parts []string
+		if profileCustomizations != nil {
+			if profileCustomizations.DevMigrationCooldown != nil {
+				parts = append(parts, fmt.Sprintf(`"migrationCooldown":%q`, profileCustomizations.DevMigrationCooldown.Duration.String()))
+			}
+			if profileCustomizations.DevMaxMigrationCooldown != nil {
+				parts = append(parts, fmt.Sprintf(`"maxMigrationCooldown":%q`, profileCustomizations.DevMaxMigrationCooldown.Duration.String()))
+			}
+			if profileCustomizations.DevMigrationHistoryWindow != nil {
+				parts = append(parts, fmt.Sprintf(`"migrationHistoryWindow":%q`, profileCustomizations.DevMigrationHistoryWindow.Duration.String()))
+			}
+		}
+		profile.PluginConfigs = append(profile.PluginConfigs, v1alpha2.PluginConfig{
+			Name: kubevirtMigrationAwarePluginName,
+			Args: runtime.RawExtension{Raw: []byte("{" + strings.Join(parts, ",") + "}")},
+		})
+	}
+
 	lowThreshold, highThreshold, err := getKubeVirtRelieveAndMigrateThresholds(profileCustomizations, args.UseDeviationThresholds)
 	if err != nil {
 		return nil, err

pkg/operator/target_config_reconciler_test.go

@@ -532,6 +532,20 @@ func TestManageConfigMap(t *testing.T) {
 			err:             fmt.Errorf("unable to get openshift-monitoring/prometheus-k8s route: route.route.openshift.io \"prometheus-k8s\" not found"),
 			forceDeployment: true,
 		},
+		{
+			name: "DevKubeVirtRelieveAndMigrateMigrationCooldown",
+			descheduler: buildKubeDeschedulerSpec(func(spec *deschedulerv1.KubeDeschedulerSpec) {
+				spec.Profiles = []deschedulerv1.DeschedulerProfile{deschedulerv1.DevKubeVirtRelieveAndMigrate}
+				spec.ProfileCustomizations = &deschedulerv1.ProfileCustomizations{
+					DevMigrationCooldown:      &metav1.Duration{Duration: 15 * time.Minute},
+					DevMaxMigrationCooldown:   &metav1.Duration{Duration: 6 * time.Hour},
+					DevMigrationHistoryWindow: &metav1.Duration{Duration: 24 * time.Hour},
+				}
+			}),
+			want:   makeConfigMap("assets/relieveAndMigrateMigrationCooldown.yaml"),
+			routes: makePrometheusRoute(),
+			nodes:  makeKubeVirtNodes(),
+		},
 		{
 			name: "AffinityAndTaintsWithNamespaces",
 			descheduler: buildKubeDeschedulerSpec(func(spec *deschedulerv1.KubeDeschedulerSpec) {

pkg/operator/testdata/assets/relieveAndMigrateDefaults.yaml

@@ -33,6 +33,8 @@ profiles:
       noEvictionPolicy: Mandatory
       podProtections: {}
     name: DefaultEvictor
+  - args: {}
+    name: KubevirtMigrationAware
   plugins:
     balance:
       disabled: null
@@ -45,9 +47,11 @@ profiles:
       disabled: null
       enabled:
       - DefaultEvictor
+      - KubevirtMigrationAware
     preevictionfilter:
       disabled: null
-      enabled: null
+      enabled:
+      - KubevirtMigrationAware
     presort:
       disabled: null
       enabled: null

pkg/operator/testdata/assets/relieveAndMigrateDeviationLowWithCombinedMetrics.yaml

@@ -33,6 +33,8 @@ profiles:
       noEvictionPolicy: Mandatory
       podProtections: {}
     name: DefaultEvictor
+  - args: {}
+    name: KubevirtMigrationAware
   plugins:
     balance:
       disabled: null
@@ -45,9 +47,11 @@ profiles:
       disabled: null
       enabled:
       - DefaultEvictor
+      - KubevirtMigrationAware
     preevictionfilter:
       disabled: null
-      enabled: null
+      enabled:
+      - KubevirtMigrationAware
     presort:
       disabled: null
       enabled: null

pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml

@@ -33,6 +33,8 @@ profiles:
       noEvictionPolicy: Mandatory
       podProtections: {}
     name: DefaultEvictor
+  - args: {}
+    name: KubevirtMigrationAware
   plugins:
     balance:
       disabled: null
@@ -45,9 +47,11 @@ profiles:
       disabled: null
       enabled:
       - DefaultEvictor
+      - KubevirtMigrationAware
     preevictionfilter:
       disabled: null
-      enabled: null
+      enabled:
+      - KubevirtMigrationAware
     presort:
       disabled: null
       enabled: null

pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml

@@ -33,6 +33,8 @@ profiles:
       noEvictionPolicy: Mandatory
       podProtections: {}
     name: DefaultEvictor
+  - args: {}
+    name: KubevirtMigrationAware
   plugins:
     balance:
       disabled: null
@@ -45,9 +47,11 @@ profiles:
       disabled: null
       enabled:
       - DefaultEvictor
+      - KubevirtMigrationAware
     preevictionfilter:
       disabled: null
-      enabled: null
+      enabled:
+      - KubevirtMigrationAware
     presort:
       disabled: null
       enabled: null

pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsMedium.yaml

@@ -33,6 +33,8 @@ profiles:
       noEvictionPolicy: Mandatory
       podProtections: {}
     name: DefaultEvictor
+  - args: {}
+    name: KubevirtMigrationAware
   plugins:
     balance:
       disabled: null
@@ -45,9 +47,11 @@ profiles:
       disabled: null
       enabled:
       - DefaultEvictor
+      - KubevirtMigrationAware
     preevictionfilter:
       disabled: null
-      enabled: null
+      enabled:
+      - KubevirtMigrationAware
     presort:
       disabled: null
       enabled: null