diff --git a/README.md b/README.md index c107dbfaa..e25a23971 100644 --- a/README.md +++ b/README.md @@ -211,10 +211,13 @@ this asymmetry and foster a quicker convergence. This profile requires [PSI](https://docs.kernel.org/accounting/psi.html) metrics to be enabled (psi=1 kernel parameter) for all the worker nodes. -The profile exposes the following customization: +The profile exposes the following customizations: - `devLowNodeUtilizationThresholds`: Sets experimental thresholds for the LowNodeUtilization strategy. - `devActualUtilizationProfile`: Enable load-aware descheduling. - `devDeviationThresholds`: Have the thresholds be based on the average utilization. +- `devMigrationCooldown`: Base cooldown after a VM migration before the same VM is eligible for re-eviction. +- `devMaxMigrationCooldown`: Upper bound on the exponential backoff cooldown between successive migrations. +- `devMigrationHistoryWindow`: Sliding window over which past migrations are counted for backoff calculation. By default, this profile will enable load-aware descheduling based on the `PrometheusCPUMemoryCombinedProfile` Prometheus query. That query is based on a recording rule combining the impact of CPU and memory utilization and PSI pressure. By default, the thresholds will be dynamic (based on the distance from the average utilization) and asymmetric (all the nodes below the average will be considered as underutilized to help rebalancing overutilized outliers) tolerating low deviations (10%). @@ -225,6 +228,13 @@ Those two values can be customized with `evictionLimits.total` and `evictionLimi This profile configures the `DefaultEvictor` plugin with a `Mandatory noEvictionPolicy`. As a result, the `descheduler.alpha.kubernetes.io/prefer-no-eviction` annotation can be dynamically applied to VM templates and/or VMIs to exclude them from the descheduler's scope. +This profile also enables the `KubevirtMigrationAware` plugin as both a filter and a pre-eviction filter. +As a filter, it hard-blocks eviction of any pod whose VMI has a live migration already in progress. +As a pre-eviction filter, it applies an adaptive cooldown after each migration completes, using exponential +backoff bounded by a configurable maximum, to prevent cascading re-evictions of the same VM. +The cooldown parameters can be tuned with the `devMigrationCooldown`, `devMaxMigrationCooldown`, and +`devMigrationHistoryWindow` profile customizations. + ### EvictPodsWithPVC By default, the operator prevents pods with PVCs from being evicted. Enabling this profile in combination with any of the above profiles allows pods with PVCs to be @@ -252,6 +262,9 @@ the `profileCustomizations` field: | `devHighNodeUtilizationThresholds` | `string` | Sets thresholds for the [HighNodeUtilization](https://github.com/kubernetes-sigs/descheduler#highnodeutilization) strategy of the `CompactAndScale` profile in the following ratios: `Minimal` for 10%, `Modest` for 20%, `Moderate` for 30%. Currently provided as an experimental feature.| |`devActualUtilizationProfile`|`string`| Sets a profile that gets translated into a predefined prometheus query | | `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization. An AsymmetricDeviationThreshold will force all nodes below the average to be considered as underutilized to help rebalancing overutilized outliers. Supported settings: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30%, `AsymmetricLow`: 0%:10%, `AsymmetricMedium`: 0%:20%, `AsymmetricHigh`: 0%:30% | +| `devMigrationCooldown` | `duration` | Sets the base cooldown period for the `KubevirtMigrationAware` plugin of the `KubeVirtRelieveAndMigrate` profile. After a VM migration completes, the plugin will not evict that VM again for at least this duration (subject to exponential backoff). Only applicable to the `KubeVirtRelieveAndMigrate` profile. | +| `devMaxMigrationCooldown` | `duration` | Sets the upper bound on the exponential backoff cooldown for the `KubevirtMigrationAware` plugin of the `KubeVirtRelieveAndMigrate` profile. Regardless of how many migrations have occurred within the history window, the cooldown will not exceed this value. Only applicable to the `KubeVirtRelieveAndMigrate` profile. | +| `devMigrationHistoryWindow` | `duration` | Sets the sliding time window used by the `KubevirtMigrationAware` plugin to count past migrations when computing the exponential backoff cooldown. Only applicable to the `KubeVirtRelieveAndMigrate` profile. | ## Prometheus query profiles The operator provides the following profiles: diff --git a/bindata/assets/kube-descheduler/crdschema.yaml b/bindata/assets/kube-descheduler/crdschema.yaml index 4473f2d09..3f2dd6ed2 100644 --- a/bindata/assets/kube-descheduler/crdschema.yaml +++ b/bindata/assets/kube-descheduler/crdschema.yaml @@ -137,6 +137,24 @@ properties: - High - "" type: string + devMaxMigrationCooldown: + description: |- + devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationCooldown: + description: |- + devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationHistoryWindow: + description: |- + devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string namespaces: description: |- Namespaces overrides included and excluded namespaces while keeping diff --git a/bindata/assets/kube-descheduler/operandclusterrole.yaml b/bindata/assets/kube-descheduler/operandclusterrole.yaml index eb6532382..f51be56c3 100644 --- a/bindata/assets/kube-descheduler/operandclusterrole.yaml +++ b/bindata/assets/kube-descheduler/operandclusterrole.yaml @@ -63,3 +63,11 @@ rules: - "get" - "patch" - "delete" +- apiGroups: + - "kubevirt.io" + resources: + - "virtualmachineinstances" + verbs: + - "get" + - "list" + - "watch" diff --git a/deploy/02_clusterrole.yaml b/deploy/02_clusterrole.yaml index c3b1b0015..c74279198 100644 --- a/deploy/02_clusterrole.yaml +++ b/deploy/02_clusterrole.yaml @@ -208,3 +208,11 @@ rules: - patch - delete - deletecollection + - apiGroups: + - kubevirt.io + resources: + - virtualmachineinstances + verbs: + - get + - list + - watch diff --git a/manifests/cluster-kube-descheduler-operator.clusterserviceversion.yaml b/manifests/cluster-kube-descheduler-operator.clusterserviceversion.yaml index 434419f09..18092368c 100644 --- a/manifests/cluster-kube-descheduler-operator.clusterserviceversion.yaml +++ b/manifests/cluster-kube-descheduler-operator.clusterserviceversion.yaml @@ -356,6 +356,14 @@ spec: - patch - delete - deletecollection + - apiGroups: + - kubevirt.io + resources: + - virtualmachineinstances + verbs: + - get + - list + - watch deployments: - name: descheduler-operator spec: diff --git a/manifests/kube-descheduler-operator.crd.yaml b/manifests/kube-descheduler-operator.crd.yaml index 302f63f82..5a37df272 100644 --- a/manifests/kube-descheduler-operator.crd.yaml +++ b/manifests/kube-descheduler-operator.crd.yaml @@ -153,6 +153,24 @@ spec: - High - "" type: string + devMaxMigrationCooldown: + description: |- + devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationCooldown: + description: |- + devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationHistoryWindow: + description: |- + devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string namespaces: description: |- Namespaces overrides included and excluded namespaces while keeping diff --git a/pkg/apis/descheduler/v1/types_descheduler.go b/pkg/apis/descheduler/v1/types_descheduler.go index 419ee70c9..745efc543 100644 --- a/pkg/apis/descheduler/v1/types_descheduler.go +++ b/pkg/apis/descheduler/v1/types_descheduler.go @@ -127,6 +127,21 @@ type ProfileCustomizations struct { // devDeviationThresholds enables dynamic thresholds based on average resource utilization // +kubebuilder:validation:Enum=Low;Medium;High;AsymmetricLow;AsymmetricMedium;AsymmetricHigh;"" DevDeviationThresholds *DeviationThresholdsType `json:"devDeviationThresholds,omitempty"` + + // devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin. + // Only applicable for the KubeVirtRelieveAndMigrate profile. + // +kubebuilder:validation:Format=duration + DevMigrationCooldown *metav1.Duration `json:"devMigrationCooldown,omitempty"` + + // devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin. + // Only applicable for the KubeVirtRelieveAndMigrate profile. + // +kubebuilder:validation:Format=duration + DevMaxMigrationCooldown *metav1.Duration `json:"devMaxMigrationCooldown,omitempty"` + + // devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin. + // Only applicable for the KubeVirtRelieveAndMigrate profile. + // +kubebuilder:validation:Format=duration + DevMigrationHistoryWindow *metav1.Duration `json:"devMigrationHistoryWindow,omitempty"` } type LowNodeUtilizationThresholdsType string diff --git a/pkg/apis/descheduler/v1/zz_generated.deepcopy.go b/pkg/apis/descheduler/v1/zz_generated.deepcopy.go index cb65e99ce..fe68c2add 100644 --- a/pkg/apis/descheduler/v1/zz_generated.deepcopy.go +++ b/pkg/apis/descheduler/v1/zz_generated.deepcopy.go @@ -206,6 +206,21 @@ func (in *ProfileCustomizations) DeepCopyInto(out *ProfileCustomizations) { *out = new(DeviationThresholdsType) **out = **in } + if in.DevMigrationCooldown != nil { + in, out := &in.DevMigrationCooldown, &out.DevMigrationCooldown + *out = new(metav1.Duration) + **out = **in + } + if in.DevMaxMigrationCooldown != nil { + in, out := &in.DevMaxMigrationCooldown, &out.DevMaxMigrationCooldown + *out = new(metav1.Duration) + **out = **in + } + if in.DevMigrationHistoryWindow != nil { + in, out := &in.DevMigrationHistoryWindow, &out.DevMigrationHistoryWindow + *out = new(metav1.Duration) + **out = **in + } return } diff --git a/pkg/generated/applyconfiguration/descheduler/v1/profilecustomizations.go b/pkg/generated/applyconfiguration/descheduler/v1/profilecustomizations.go index 3f7c35be8..ab52603fa 100644 --- a/pkg/generated/applyconfiguration/descheduler/v1/profilecustomizations.go +++ b/pkg/generated/applyconfiguration/descheduler/v1/profilecustomizations.go @@ -45,6 +45,12 @@ type ProfileCustomizationsApplyConfiguration struct { DevActualUtilizationProfile *deschedulerv1.ActualUtilizationProfile `json:"devActualUtilizationProfile,omitempty"` // devDeviationThresholds enables dynamic thresholds based on average resource utilization DevDeviationThresholds *deschedulerv1.DeviationThresholdsType `json:"devDeviationThresholds,omitempty"` + // devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin. + DevMigrationCooldown *metav1.Duration `json:"devMigrationCooldown,omitempty"` + // devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin. + DevMaxMigrationCooldown *metav1.Duration `json:"devMaxMigrationCooldown,omitempty"` + // devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin. + DevMigrationHistoryWindow *metav1.Duration `json:"devMigrationHistoryWindow,omitempty"` } // ProfileCustomizationsApplyConfiguration constructs a declarative configuration of the ProfileCustomizations type for use with @@ -132,3 +138,27 @@ func (b *ProfileCustomizationsApplyConfiguration) WithDevDeviationThresholds(val b.DevDeviationThresholds = &value return b } + +// WithDevMigrationCooldown sets the DevMigrationCooldown field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the DevMigrationCooldown field is set to the value of the last call. +func (b *ProfileCustomizationsApplyConfiguration) WithDevMigrationCooldown(value metav1.Duration) *ProfileCustomizationsApplyConfiguration { + b.DevMigrationCooldown = &value + return b +} + +// WithDevMaxMigrationCooldown sets the DevMaxMigrationCooldown field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the DevMaxMigrationCooldown field is set to the value of the last call. +func (b *ProfileCustomizationsApplyConfiguration) WithDevMaxMigrationCooldown(value metav1.Duration) *ProfileCustomizationsApplyConfiguration { + b.DevMaxMigrationCooldown = &value + return b +} + +// WithDevMigrationHistoryWindow sets the DevMigrationHistoryWindow field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the DevMigrationHistoryWindow field is set to the value of the last call. +func (b *ProfileCustomizationsApplyConfiguration) WithDevMigrationHistoryWindow(value metav1.Duration) *ProfileCustomizationsApplyConfiguration { + b.DevMigrationHistoryWindow = &value + return b +} diff --git a/pkg/operator/target_config_reconciler.go b/pkg/operator/target_config_reconciler.go index 711921f3b..8ec3b373c 100644 --- a/pkg/operator/target_config_reconciler.go +++ b/pkg/operator/target_config_reconciler.go @@ -77,6 +77,7 @@ const psiPath = "/proc/pressure/" const EXPERIMENTAL_DISABLE_PSI_CHECK = "EXPERIMENTAL_DISABLE_PSI_CHECK" const defaultKVParallelMigrationsPerCluster = 5 const defaultKVParallelOutboundMigrationsPerNode = 2 +const kubevirtMigrationAwarePluginName = "KubevirtMigrationAware" // deschedulerCommand provides descheduler command with policyconfigfile mounted as volume and log-level for backwards // compatibility with 3.11 @@ -1097,6 +1098,12 @@ func kubeVirtRelieveAndMigrateProfile(profileCustomizations *deschedulerv1.Profi Filter: v1alpha2.PluginSet{ Enabled: []string{ defaultevictor.PluginName, + kubevirtMigrationAwarePluginName, + }, + }, + PreEvictionFilter: v1alpha2.PluginSet{ + Enabled: []string{ + kubevirtMigrationAwarePluginName, }, }, Balance: v1alpha2.PluginSet{ @@ -1159,6 +1166,25 @@ func kubeVirtRelieveAndMigrateProfile(profileCustomizations *deschedulerv1.Profi } } + { + var parts []string + if profileCustomizations != nil { + if profileCustomizations.DevMigrationCooldown != nil { + parts = append(parts, fmt.Sprintf(`"migrationCooldown":%q`, profileCustomizations.DevMigrationCooldown.Duration.String())) + } + if profileCustomizations.DevMaxMigrationCooldown != nil { + parts = append(parts, fmt.Sprintf(`"maxMigrationCooldown":%q`, profileCustomizations.DevMaxMigrationCooldown.Duration.String())) + } + if profileCustomizations.DevMigrationHistoryWindow != nil { + parts = append(parts, fmt.Sprintf(`"migrationHistoryWindow":%q`, profileCustomizations.DevMigrationHistoryWindow.Duration.String())) + } + } + profile.PluginConfigs = append(profile.PluginConfigs, v1alpha2.PluginConfig{ + Name: kubevirtMigrationAwarePluginName, + Args: runtime.RawExtension{Raw: []byte("{" + strings.Join(parts, ",") + "}")}, + }) + } + lowThreshold, highThreshold, err := getKubeVirtRelieveAndMigrateThresholds(profileCustomizations, args.UseDeviationThresholds) if err != nil { return nil, err diff --git a/pkg/operator/target_config_reconciler_test.go b/pkg/operator/target_config_reconciler_test.go index fd7e90786..d92dd219a 100644 --- a/pkg/operator/target_config_reconciler_test.go +++ b/pkg/operator/target_config_reconciler_test.go @@ -532,6 +532,20 @@ func TestManageConfigMap(t *testing.T) { err: fmt.Errorf("unable to get openshift-monitoring/prometheus-k8s route: route.route.openshift.io \"prometheus-k8s\" not found"), forceDeployment: true, }, + { + name: "DevKubeVirtRelieveAndMigrateMigrationCooldown", + descheduler: buildKubeDeschedulerSpec(func(spec *deschedulerv1.KubeDeschedulerSpec) { + spec.Profiles = []deschedulerv1.DeschedulerProfile{deschedulerv1.DevKubeVirtRelieveAndMigrate} + spec.ProfileCustomizations = &deschedulerv1.ProfileCustomizations{ + DevMigrationCooldown: &metav1.Duration{Duration: 15 * time.Minute}, + DevMaxMigrationCooldown: &metav1.Duration{Duration: 6 * time.Hour}, + DevMigrationHistoryWindow: &metav1.Duration{Duration: 24 * time.Hour}, + } + }), + want: makeConfigMap("assets/relieveAndMigrateMigrationCooldown.yaml"), + routes: makePrometheusRoute(), + nodes: makeKubeVirtNodes(), + }, { name: "AffinityAndTaintsWithNamespaces", descheduler: buildKubeDeschedulerSpec(func(spec *deschedulerv1.KubeDeschedulerSpec) { diff --git a/pkg/operator/testdata/assets/relieveAndMigrateDefaults.yaml b/pkg/operator/testdata/assets/relieveAndMigrateDefaults.yaml index bc89485ca..54f07ea76 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateDefaults.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateDefaults.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateDeviationLowWithCombinedMetrics.yaml b/pkg/operator/testdata/assets/relieveAndMigrateDeviationLowWithCombinedMetrics.yaml index 0341aedc6..143cbfda2 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateDeviationLowWithCombinedMetrics.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateDeviationLowWithCombinedMetrics.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml index d59d78c77..54adace0f 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsHigh.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml index ba8774268..1617a1db1 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsLow.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsMedium.yaml b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsMedium.yaml index 44248f8fd..46f534173 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsMedium.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateDynamicThresholdsMedium.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateEvictionLimits.yaml b/pkg/operator/testdata/assets/relieveAndMigrateEvictionLimits.yaml index e41b0ca3d..a71e9797e 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateEvictionLimits.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateEvictionLimits.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateHighConfig.yaml b/pkg/operator/testdata/assets/relieveAndMigrateHighConfig.yaml index dfabbe680..79cfc2e5d 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateHighConfig.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateHighConfig.yaml @@ -32,6 +32,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -44,9 +46,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateIncludedNamespace.yaml b/pkg/operator/testdata/assets/relieveAndMigrateIncludedNamespace.yaml index bc89485ca..54f07ea76 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateIncludedNamespace.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateIncludedNamespace.yaml @@ -33,6 +33,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -45,9 +47,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateLowConfig.yaml b/pkg/operator/testdata/assets/relieveAndMigrateLowConfig.yaml index c2d07c791..f5568bcd4 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateLowConfig.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateLowConfig.yaml @@ -32,6 +32,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -44,9 +46,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateMediumConfig.yaml b/pkg/operator/testdata/assets/relieveAndMigrateMediumConfig.yaml index f63b61c5c..a843f8a10 100644 --- a/pkg/operator/testdata/assets/relieveAndMigrateMediumConfig.yaml +++ b/pkg/operator/testdata/assets/relieveAndMigrateMediumConfig.yaml @@ -32,6 +32,8 @@ profiles: noEvictionPolicy: Mandatory podProtections: {} name: DefaultEvictor + - args: {} + name: KubevirtMigrationAware plugins: balance: disabled: null @@ -44,9 +46,11 @@ profiles: disabled: null enabled: - DefaultEvictor + - KubevirtMigrationAware preevictionfilter: disabled: null - enabled: null + enabled: + - KubevirtMigrationAware presort: disabled: null enabled: null diff --git a/pkg/operator/testdata/assets/relieveAndMigrateMigrationCooldown.yaml b/pkg/operator/testdata/assets/relieveAndMigrateMigrationCooldown.yaml new file mode 100644 index 000000000..a430c1288 --- /dev/null +++ b/pkg/operator/testdata/assets/relieveAndMigrateMigrationCooldown.yaml @@ -0,0 +1,63 @@ +apiVersion: descheduler/v1alpha2 +kind: DeschedulerPolicy +maxNoOfPodsToEvictPerNode: 2 +maxNoOfPodsToEvictTotal: 5 +metricsProviders: +- prometheus: + url: https://prometheus-k8s-openshift-monitoring.apps.example.com + source: Prometheus +nodeSelector: kubevirt.io/schedulable=true +profiles: +- name: KubeVirtRelieveAndMigrate + pluginConfig: + - args: + evictableNamespaces: + exclude: + - kube-system + - hypershift + - openshift + - openshift-kube-descheduler-operator + - openshift-kube-scheduler + metricsUtilization: + prometheus: + query: descheduler:node:linear_amplified_ideal_point_positive_distance:k3:avg1m + source: Prometheus + targetThresholds: + MetricResource: 10 + thresholds: + MetricResource: 0 + useDeviationThresholds: true + name: LowNodeUtilization + - args: + evictLocalStoragePods: true + noEvictionPolicy: Mandatory + podProtections: {} + name: DefaultEvictor + - args: + maxMigrationCooldown: 6h0m0s + migrationCooldown: 15m0s + migrationHistoryWindow: 24h0m0s + name: KubevirtMigrationAware + plugins: + balance: + disabled: null + enabled: + - LowNodeUtilization + deschedule: + disabled: null + enabled: null + filter: + disabled: null + enabled: + - DefaultEvictor + - KubevirtMigrationAware + preevictionfilter: + disabled: null + enabled: + - KubevirtMigrationAware + presort: + disabled: null + enabled: null + sort: + disabled: null + enabled: null diff --git a/test/e2e/bindata/assets/00_kube-descheduler-operator-crd.yaml b/test/e2e/bindata/assets/00_kube-descheduler-operator-crd.yaml index 302f63f82..5a37df272 100644 --- a/test/e2e/bindata/assets/00_kube-descheduler-operator-crd.yaml +++ b/test/e2e/bindata/assets/00_kube-descheduler-operator-crd.yaml @@ -153,6 +153,24 @@ spec: - High - "" type: string + devMaxMigrationCooldown: + description: |- + devMaxMigrationCooldown configures the maximum migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationCooldown: + description: |- + devMigrationCooldown configures the base migration cooldown for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string + devMigrationHistoryWindow: + description: |- + devMigrationHistoryWindow configures the migration history window for the KubevirtMigrationAware plugin. + Only applicable for the KubeVirtRelieveAndMigrate profile. + format: duration + type: string namespaces: description: |- Namespaces overrides included and excluded namespaces while keeping diff --git a/test/e2e/bindata/assets/03_clusterrole.yaml b/test/e2e/bindata/assets/03_clusterrole.yaml index c3b1b0015..c74279198 100644 --- a/test/e2e/bindata/assets/03_clusterrole.yaml +++ b/test/e2e/bindata/assets/03_clusterrole.yaml @@ -208,3 +208,11 @@ rules: - patch - delete - deletecollection + - apiGroups: + - kubevirt.io + resources: + - virtualmachineinstances + verbs: + - get + - list + - watch