Chapter 60: Kubernetes Workload Controllers in Depth: ReplicaSet, Deployment, StatefulSet, DaemonSet, Jobs and CronJobs

60.5 DaemonSet: one pod on every node, tolerations and node selectors

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Ola runs 200 worker nodes on EKS. The team wants logs from every node sent to CloudWatch, and node CPU and disk numbers sent to Prometheus. With a Deployment of 200 replicas, the scheduler might put 3 pods on one node and none on another. When the cluster grows to 250 nodes, someone must remember to scale up. A DaemonSet fixes this. It runs exactly one copy of a pod on every node, or on every node that matches a rule. A new node joins, it gets the pod automatically. A node leaves, the pod goes with it. You already use DaemonSets without knowing it: kube-proxy, the CNI plugin such as aws-node or Calico, and the EBS CSI node driver are all DaemonSets. In this lesson we run a node agent on every kind node, add it to the control plane with a toleration, limit it to some nodes with a nodeSelector, read host files safely, and roll out an update. Then we break it with a missing toleration and a wrong label.

What: every term in plain English

• DaemonSet (DS) — An apps/v1 controller that runs one pod per eligible node.
• Eligible node — A node that passes the pod's nodeSelector, node affinity and taint rules.
• Taint — A mark on a node that repels pods, for example node-role.kubernetes.io/control-plane:NoSchedule.
• Toleration — A pod setting that lets it run on a tainted node.
• nodeSelector — Run only on nodes with these labels, like disktype=ssd.
• hostPath — Mounts a folder of the node into the pod, like /var/log. Read-only when possible.
• hostNetwork — The pod uses the node's network and IP. Used by some agents.
• updateStrategy — RollingUpdate (default, maxUnavailable 1) or OnDelete.
• DESIRED / CURRENT / READY — DESIRED is the number of eligible nodes.
• Node agent — A program that must run on each machine: log shipper, metrics exporter, security scanner, storage driver.

How: step by step

1. Choose what runs per node. Log shipper, node exporter, security agent.
2. Write the DaemonSet. Like a Deployment but no replicas field.
3. Mount host paths read-only. Only what the agent needs.
4. Decide which nodes. nodeSelector or affinity to include, tolerations to reach tainted nodes.
5. Set small resources. The agent runs on every node, so its cost multiplies.
6. Apply and count. DESIRED must equal the number of eligible nodes.
7. Update one node at a time. RollingUpdate with maxUnavailable 1.

Example 1: A node log agent on every worker node

Goal. A node log agent on every worker node.

node-logger.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-logger
  namespace: kube-system
  labels:
    app: node-logger
spec:
  selector:
    matchLabels:
      app: node-logger
  template:
    metadata:
      labels:
        app: node-logger
    spec:
      containers:
        - name: agent
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - while true; do
                echo "$(date +%T) node=$NODE_NAME pods=$(ls /var/log/pods | wc -l)";
                sleep 30;
              done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 32Mi}
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
            type: Directory

What each line does

• kind: DaemonSet — No replicas: the count comes from the nodes.
• namespace: kube-system — Cluster agents usually live here.
• fieldRef spec.nodeName — The Downward API tells the pod which node it is on.
• hostPath /var/log, readOnly — Reads node logs; cannot change them.
• type: Directory — Fails if the folder does not exist, instead of creating it silently.
• Small requests — 10m CPU times 200 nodes is still only 2 CPUs.

Green output

$ kubectl apply -f node-logger.yaml
daemonset.apps/node-logger created
$ kubectl get ds node-logger -n kube-system
NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-logger   2         2         2       2            2           <none>          12s
$ kubectl get pods -n kube-system -l app=node-logger -o wide
NAME                READY   STATUS    NODE
node-logger-7tq2c   1/1     Running   ch59-worker
node-logger-xk9pd   1/1     Running   ch59-worker2
$ kubectl logs -n kube-system node-logger-7tq2c
06:31:10 node=ch59-worker pods=6

Expected result. DESIRED is 2, not 3. The control-plane node has a NoSchedule taint, so it was skipped.

Example 2: Reach the control plane with a toleration

Goal. Reach the control plane with a toleration.

node-logger-all.yaml

spec:
  template:
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule

What each line does

• key — The exact taint key on the control-plane node.
• operator: Exists — Tolerate the taint whatever its value.
• effect: NoSchedule — Matches the taint effect.
• Patch file — Only the changed part; kubectl merges it into the DaemonSet.

Green output

$ kubectl describe node ch59-control-plane | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
$ kubectl patch ds node-logger -n kube-system --type=strategic --patch-file node-logger-all.yaml
daemonset.apps/node-logger patched
$ kubectl get ds node-logger -n kube-system
NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
node-logger   3         3         3       3            3
$ kubectl get ds -n kube-system
NAME          DESIRED   CURRENT   READY
kindnet       3         3         3
kube-proxy    3         3         3
node-logger   3         3         3

Expected result. Now it runs on all 3 nodes. kindnet (the CNI) and kube-proxy are DaemonSets with the same kind of toleration.

Example 3: Run only on labelled nodes, then add a node to the group

Goal. Run only on labelled nodes, then add a node to the group.

ssd-exporter.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ssd-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ssd-exporter
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: ssd-exporter
    spec:
      nodeSelector:
        disktype: ssd
      containers:
        - name: exporter
          image: busybox:1.36
          command: ["sh", "-c", "while true; do df -h /host | tail -1; sleep 60; done"]
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 32Mi}
          volumeMounts:
            - name: root
              mountPath: /host
              readOnly: true
      volumes:
        - name: root
          hostPath:
            path: /

What each line does

• nodeSelector disktype: ssd — Only nodes with this label are eligible.
• maxUnavailable: 1 — During an update, one node at a time loses its agent.
• hostPath / readOnly — Reads disk usage of the node; read-only keeps it safe.
• Adding a node — Label a node and the pod appears within seconds. Remove the label and it goes.

Green output

$ kubectl apply -f ssd-exporter.yaml
$ kubectl get ds ssd-exporter -n kube-system
NAME           DESIRED   CURRENT   READY   NODE SELECTOR
ssd-exporter   0         0         0       disktype=ssd
$ kubectl label node ch59-worker disktype=ssd
node/ch59-worker labeled
$ kubectl get pods -n kube-system -l app=ssd-exporter -o wide
NAME                 READY   STATUS    NODE
ssd-exporter-d8m4q   1/1     Running   ch59-worker
$ kubectl label node ch59-worker2 disktype=ssd
$ kubectl get ds ssd-exporter -n kube-system
NAME           DESIRED   CURRENT   READY
ssd-exporter   2         2         2
$ kubectl label node ch59-worker disktype-
$ kubectl get ds ssd-exporter -n kube-system
NAME           DESIRED   CURRENT   READY
ssd-exporter   1         1         1

Expected result. DESIRED follows the node labels. On EKS the same idea runs GPU drivers only on GPU node groups.

Every command, step by step (kind lab with kubectl)

1. List all DaemonSets

   kubectl get ds -A
   

   What it does: See the system agents already running.

   NAMESPACE     NAME         DESIRED   CURRENT   READY
   kube-system   kindnet      3         3         3
   kube-system   kube-proxy   3         3         3
   

2. Read kube-proxy tolerations

   kubectl get ds kube-proxy -n kube-system -o jsonpath='{.spec.template.spec.tolerations}{"\n"}'
   

   What it does: kube-proxy tolerates everything, so it runs on every node.

   [{"operator":"Exists"}]
   

3. Apply the logger

   kubectl apply -f node-logger.yaml
   kubectl get ds node-logger -n kube-system
   

   What it does: 2 nodes until we add the toleration.

   node-logger   2   2   2
   

4. Add the toleration

   kubectl patch ds node-logger -n kube-system --type=strategic --patch-file node-logger-all.yaml
   kubectl rollout status ds/node-logger -n kube-system
   

   What it does: Rolls one node at a time.

   daemon set "node-logger" successfully rolled out
   

5. Delete one agent pod

   kubectl delete pod -n kube-system $(kubectl get pod -n kube-system -l app=node-logger -o name | head -1 | cut -d/ -f2)
   kubectl get pods -n kube-system -l app=node-logger -o wide
   

   What it does: A new pod appears on the same node.

   node-logger-q2w8z   1/1   Running   0   2s   ch59-worker
   

6. Update the image

   kubectl set image ds/node-logger -n kube-system agent=busybox:1.37
   kubectl rollout status ds/node-logger -n kube-system
   

   What it does: Same rollout commands as a Deployment.

   Waiting for daemon set "node-logger" rollout to finish: 1 out of 3 new pods have been updated...
   daemon set "node-logger" successfully rolled out
   

7. History and undo

   kubectl rollout history ds/node-logger -n kube-system
   kubectl rollout undo ds/node-logger -n kube-system
   

   What it does: DaemonSets keep revisions in ControllerRevisions.

   REVISION  CHANGE-CAUSE
   1         <none>
   2         <none>
   3         <none>
   

8. Drain a node

   kubectl drain ch59-worker2 --ignore-daemonsets --delete-emptydir-data
   kubectl uncordon ch59-worker2
   

   What it does: Drain skips DaemonSet pods; they must stay to serve the node.

   Warning: ignoring DaemonSet-managed Pods: kube-system/node-logger-xk9pd
   node/ch59-worker2 drained
   node/ch59-worker2 uncordoned
   

9. Clean up

   kubectl delete ds node-logger ssd-exporter -n kube-system
   kubectl label node ch59-worker2 disktype-
   

   What it does: Labels removed, agents gone.

   daemonset.apps "node-logger" deleted
   

When it is red 1

NAME          DESIRED   CURRENT   READY
node-logger   2         2         2      (cluster has 3 nodes)

Cause. The pod has no toleration for a tainted node.

Fix, step by step

1. kubectl describe node NAME | grep Taints.
2. Add a matching toleration, or operator: Exists for all.

When it is red 2

ssd-exporter   0   0   0   disktype=ssd

Cause. No node has the label, or it is spelled differently (disk-type, SSD).

Fix, step by step

1. kubectl get nodes --show-labels | grep disk.
2. Fix the label on the node or the nodeSelector.

When it is red 3

error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/node-logger-7tq2c

Cause. kubectl drain refuses to evict DaemonSet pods.

Fix, step by step

1. Add --ignore-daemonsets.
2. The DaemonSet pod stays until the node is removed.

When it is red 4

Warning  FailedMount  MountVolume.SetUp failed for volume "varlog" : hostPath type check failed: /var/logs is not a directory

Cause. Typo in the hostPath, and type: Directory caught it.

Fix, step by step

1. Fix the path to /var/log.
2. Keep type: Directory so typos fail loudly.

Practice tasks

kubectl get ds -A

kubectl logs -n kube-system -l app=node-logger --tail=1

kubectl get nodes --no-headers | wc -l

kubectl patch ds node-logger -n kube-system --type=strategic --patch-file node-logger-all.yaml

kubectl label node ch59-worker disktype=ssd

kubectl label node ch59-worker disktype-

kubectl rollout status ds/node-logger -n kube-system

kubectl drain ch59-worker --ignore-daemonsets

kubectl taint nodes ch59-worker2 edge=true:NoSchedule

kubectl exec -n kube-system ds/node-exporter -- wget -qO- localhost:9100/metrics | head

Recap

• A DaemonSet runs one pod on every eligible node and follows nodes as they join or leave.
• Taints block nodes; tolerations let agents onto them.
• nodeSelector and affinity choose a subset of nodes.
• Mount host paths read-only and keep requests small.
• Update with rollout commands; drain needs --ignore-daemonsets.
• kube-proxy, CNI and CSI node drivers are DaemonSets you already run.

FAQ

Q. Does a DaemonSet have replicas?

No. The count is the number of eligible nodes.

Q. Do DaemonSets run on Fargate?

No. EKS Fargate has no real nodes for you; use sidecars there.

Q. Can I use a DaemonSet for a web app?

Rarely. Only when each node must serve itself, like an ingress on every edge node.

Q. Which scheduler places DaemonSet pods?

The default scheduler, using node affinity that the controller adds for each node.

Q. Is hostPath safe?

Only read-only and only for trusted agents. Pod Security standards restrict it.

Q. How do I limit updates?

maxUnavailable, or OnDelete to update node by node yourself.

Interview questions

1. What is a DaemonSet used for?

   Model answer: Running node-level agents such as log collectors (Fluent Bit), monitoring (node-exporter), networking (CNI, kube-proxy) and storage drivers on every or selected nodes.

2. How do you run a DaemonSet only on some nodes?

   Model answer: Use nodeSelector or node affinity on node labels; use tolerations to include tainted nodes.

3. Why does a DaemonSet skip the control plane?

   Model answer: The control-plane node has a NoSchedule taint; add a toleration if the agent must run there.

4. DaemonSet vs Deployment?

   Model answer: A Deployment runs N pods placed anywhere; a DaemonSet runs one pod per eligible node and scales with the cluster.

5. How are DaemonSets updated?

   Model answer: RollingUpdate with maxUnavailable (and optional maxSurge), or OnDelete; kubectl rollout status, history and undo all work.

Samajla ka?