Only registred users can make comments

Learn Pod Placement with Kubernetes Taints and Tolerations

TL;DR

In Kubernetes, effective management of workloads across a diverse range of nodes is vital. One of the mechanisms Kubernetes employs to facilitate this process is the concept of 'Taints' and 'Tolerations'. These play a crucial role in how workloads are distributed, ensuring that the right pods are scheduled on suitable nodes.

A 'Taint', in Kubernetes refers to a label assigned to a node serving as a restriction on which pods can be scheduled on it. Taints give administrators the ability to prevent pods from being scheduled on a node for reasons like hardware limitations, maintenance status or resource allocation strategies.

On the hand 'Tolerations are applied to pods. Enable them (but don't make it mandatory) to be scheduled on nodes with matching taints. This feature allows for control over where specific pods can run within the cluster.

Throughout this discussion we will explore Kubernetes Taints and Tolerations, in detail emphasizing their role in efficiently and effectively managing a Kubernetes cluster.

Why bother about Taints and Tolerations?

A taint serves as a way, for a node to repel pods to a sign that says "keep out." To add a taint to a node you can use the command "kubectl taint nodes." The syntax for adding a taint is "kubectl taint nodes <node name> key=value;taint effect."

The taint effect determines what happens to pods that cannot tolerate the taint. There are three effects;

  • NoSchedule; Pods that cannot tolerate this taint will not be scheduled on the node
  • PreferNoSchedule; The Kubernetes scheduler will try to avoid scheduling pods that cannot tolerate this taint, on the node. Its not a rule
  • NoExecute; If a pod cannot tolerate this taint it will be evicted from the node if it is already running.

Example - Special hardware use case

Let's suppose we have a node in your cluster that has special hardware. We want to ensure that only certain pods are scheduled on this node. You can achieve this by adding a taint to the node, and then adding a matching toleration to the pods that you want to schedule on the node.

kubectl taint nodes special-node hardware=special:NoSchedule

This command adds a taint to the node named node-name that prevents any pods from being scheduled on it unless they tolerate the taint.

Then, to schedule a pod on that node, you'd need to add a matching toleration to the pod's specification. Here's an example pod manifest with a matching toleration:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
  tolerations:
  - key: "hardware"
    operator: "Equal"
    value: "special"
    effect: "NoSchedule"

With this configuration only pods that have the specified toleration will be scheduled on the designated node. Otherwise they will be rejected.

While tolerations allow pods to be scheduled on tainted nodes it is important to note that having a toleration does not guarantee that a pod will always be assigned to a node. Instead it simply removes the limitation imposed by a taint. Allows the pod to potentially be scheduled on a node from which it would have been otherwise excluded.

In Kubernetes the scheduling of pods, on nodes is determined by factors such as resource availability, node conditions and other scheduling constraints. Taints and tolerations are one aspect of this scheduling process.

For instance if there are two nodes in your cluster. Node A and Node B. And a pod has a toleration for a taint on Node A it is still possible for the scheduler to assign the pod to Node B if it determines that Node B is more suitable based on considerations.

In the diagram provided below we can observe that all pods on node 1 share the toleration. However due to factors like capacity limitations or other reasons there may also be an instance where another pod, with the toleration has been scheduled onto node 2. 
As mentioned earlier it's important to note that having a toleration doesn't necessarily mean that pods will always be scheduled on a node with a taint.

 

What is the difference between taints and affinity & anti-affinity?

Taints and tolerations primarily focus on ensuring that pods are placed appropriately while affinity and anti affinity rules help in attracting or repelling pods based on conditions.

Affinity comprises a set of guidelines that bring pods together for scheduling purposes. It can be either node affinity or inter pod affinity.

Node affinity is similar to pod affinity but operates at the node level. It can attract or repel pods based on the labels assigned to nodes. There are two types of node affinities: requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.

Pod affinity allows you to specify that certain pods should be scheduled on the same node or within the same topological domain as other specified pods.

On the other hand, anti-affinity involves a set of rules that aim to keep pods apart during scheduling. Like affinity, anti-affinity can be categorized as either node anti-affinity or pod anti-affinity.

Node anti-affinity is essentially the opposite of node affinity as it repels pods from nodes based on their labels.

Pod anti-affinity ensures that specific pods are not placed on the same node or within the same topological domain. This helps maintain availability by distributing pods across different nodes or topological domains and also prevents conflicts between colocated pods with conflicting resource requirements.

In summary, taints and tolerations aim to influence the scheduling of pods onto nodes, while anti-affinity focuses on the relative placement of pods to each other during scheduling. These tools offer a variety of options for controlling where your pods are placed.

Learn more about nodeselector, affinity and anti-affinity rules here

Summary

Taints and tolerations provide a powerful way to control the scheduling of pods in a Kubernetes cluster. They allow you to ensure that pods are scheduled on appropriate nodes, and they can be used to enforce important rules about resource usage and separation of concerns.

Remember, taints and tolerations are advanced features. They give you great control, but they also require careful management. So, start experimenting with them and see how they can improve your Kubernetes deployments.

References

Kubernetes Official Documentation: Taints and Tolerations

About the Author

Aleksandro Matejic, a Cloud Architect, began working in the IT industry over 21 years ago as a technical specialist, right after his studies. Since then, he has worked in various companies and industries in various system engineer and IT architect roles. He currently works on designing Cloud solutions, Kubernetes, and other DevOps technologies.

In his spare time, Aleksandro works on different development projects such as developing devoriales.com, a blog and learning platform launching in 2022/2023. In addition, he likes to read and write technical articles about software development and DevOps methods and tools.

You can contact Aleksandro by visiting his LinkedIn Profile

Comments