Protect Your Most Important Pods from Eviction with PriorityClass!

Kubernetes has established itself as a leading orchestration engine used by many organizations to efficiently manage workloads. Proper pod scheduling is crucial to ensure that application pods run smoothly within the Kubernetes cluster. In this article, we explore how the PriorityClass object is used to protect mission-critical or high-priority pods from eviction, ensuring they remain available and continue to serve traffic.

Resource Management in Kubernetes

The Kubernetes Control Plane consists of several components, with the scheduler (typically the built-in kube-scheduler) being one of the most important. It is responsible for assigning a node to a pod.

When a pod is created, it initially resides in the “pending” state until the scheduler determines the most suitable node for the new pod. Behind the scenes, the scheduler runs in an endless loop, searching for pods that do not have a nodeName set and are ready for scheduling. For each pod that needs to be scheduled, the scheduler attempts to decide which node should run the pod.

If the scheduler cannot find a suitable node, the pod remains in the pending state, which is not ideal.

Hinweis: Various criteria come into play here, such as nodeSelector, taints and tolerations, nodeAffinity, resource availability (e.g., CPU and memory), and many other factors that determine the pod’s location.

The following diagram illustrates the request flow:

A diagram showing the scheduling of three pods directly created by a client.

Typical Use Cases

Here are some real-world scenarios where control over pod scheduling and eviction might be necessary:

Suppose the pod you want to deploy is critical, and you have some resource constraints. An example would be the DaemonSet of an infrastructure component like Grafana Loki. The Loki pods need to run before other pods on each node. In such cases, you could ensure resource availability by manually identifying and deleting unnecessary pods or adding a new node to the cluster. However, both approaches are impractical, as manual deletion would be tedious, and adding a new node could be time-consuming and costly.
Another use case might be a single cluster holding pods for the following environments with corresponding priorities:
- Production (prod): highest priority
- Pre-production (preprod): medium priority
- Development (dev): lowest priority
With high resource utilization in the cluster, there is competition for CPU and memory resources on the nodes. While cluster-level autoscaling may add more nodes, this takes some time. Meanwhile, some pods might remain in the pending state, or the service could be impacted while they compete for resources. If the kubelet evacuates a pod from a node, it does so randomly, as the kubelet lacks specific information about which pods should be evacuated and which should be retained.
A third example could be a microservice supported by a queue application or a database encountering a resource bottleneck, causing the queue or database to be evicted. In such a case, all other services would be useless until the database can handle traffic again.

There are other scenarios where you might want to control the order of pod scheduling or eviction.

With the proper application of PriorityClass, you can ensure that your most important pods are always prioritized. ayedo helps you make the most of Kubernetes and protect your critical applications!

Source: Kubernetes Blog