Is your feature request related to a problem? Please describe.
We currently use a PodLifetime policy to de-schedule pods that get stuck with ContainerCreating. We use this policy to catch edge cases where our custom IPAM can hit IP exhaustion and cause pods to get stuck.
However, this policy is problematic when image pulls are slow, because slow image pulls (that don't fail, just slow) also get stuck in ContainerCreating. Due to kubernetes/kubernetes#122905, you can't effectively de-schedule pods that are still pulling images, since they get stuck in Terminating and wait for image pulling to finish, causing a potential DoS against the registry rather than make progress and settle.
Describe the solution you'd like
We'd like to evict pending pods that are in ContainerCreating that are not actively pulling. This seems a bit tricky for de-scheduler to support, since AFAICT determining that containers are pulling images requires looking at pod events, e.g.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 42m default-scheduler Successfully assigned foo/bar to foobar-10.net
Normal PodLifeTime 31m sigs.k8s.io.descheduler pod eviction from foobar-10.net node by sigs.k8s.io/descheduler
Normal Pulled 42m kubelet Container image "foobar.com/images/metrics:1.7.1" already present on machine
Normal Created 42m kubelet Created container metrics
Normal Started 42m kubelet Started container metrics
Normal Pulled 42m kubelet Container image "foobar.com/images/logcategory-collector:1.1.4" already present on machine
Normal Created 42m kubelet Created container logcategory-collector
Normal Started 42m kubelet Started container logcategory-collector
Normal Pulling 42m kubelet Pulling image "foobar.com/this-one-goes-really-slow:999"
Describe alternatives you've considered
Open to alternatives. One is to implement a custom controller that is equivalent to de-scheduler but can cross reference pod events.
What version of descheduler are you using?
descheduler version: 0.33.0
Additional context
Is your feature request related to a problem? Please describe.
We currently use a PodLifetime policy to de-schedule pods that get stuck with
ContainerCreating. We use this policy to catch edge cases where our custom IPAM can hit IP exhaustion and cause pods to get stuck.However, this policy is problematic when image pulls are slow, because slow image pulls (that don't fail, just slow) also get stuck in
ContainerCreating. Due to kubernetes/kubernetes#122905, you can't effectively de-schedule pods that are still pulling images, since they get stuck in Terminating and wait for image pulling to finish, causing a potential DoS against the registry rather than make progress and settle.Describe the solution you'd like
We'd like to evict pending pods that are in
ContainerCreatingthat are not actively pulling. This seems a bit tricky for de-scheduler to support, since AFAICT determining that containers are pulling images requires looking at pod events, e.g.Describe alternatives you've considered
Open to alternatives. One is to implement a custom controller that is equivalent to de-scheduler but can cross reference pod events.
What version of descheduler are you using?
descheduler version: 0.33.0
Additional context