Chapter 60: Kubernetes Workload Controllers in Depth: ReplicaSet, Deployment, StatefulSet, DaemonSet, Jobs and CronJobs

60.8 Choosing the right controller: one app, every controller, side by side

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose you join the platform team at MakeMyTrip. On day one you get a list: a flight search API, a Redis cache with saved data, a log shipper for every node, a nightly fare refresh, and a one-time data migration. Your lead asks: which Kubernetes object for each? Picking the wrong one causes real outages. A database in a Deployment loses data. A batch script in a Deployment restarts forever. A log shipper in a Deployment misses nodes. A web API in a StatefulSet updates slowly, one pod at a time, for no reason. In this lesson we run the same small app as a Deployment, a StatefulSet, a DaemonSet, a Job and a CronJob, all at once, and compare names, placement, storage, scaling, updates and deletion with real commands. Then we finish with a decision table and the mistakes interviewers love to ask about. This closes chapter 60. Every controller here uses the same loop you met in 60.1: observe, compare, act.

What: every term in plain English

• Stateless — Any pod can serve any request; nothing on its disk matters. Use a Deployment.
• Stateful — Each pod has its own data or identity. Use a StatefulSet.
• Per-node — Must run on every node or on some nodes. Use a DaemonSet.
• Run-to-completion — Work that ends. Use a Job.
• Scheduled — Work that ends and repeats on a clock. Use a CronJob.
• ReplicaSet / RC — Created by Deployments; you do not write them yourself.
• Bare pod — No controller. Only for quick debugging.
• Operator — A custom controller for one product, like a Postgres or Kafka operator, built on these controllers.
• Owner chain — CronJob, then Job, then Pod; Deployment, then ReplicaSet, then Pod.
• Idempotent — Running it twice gives the same result. Jobs and CronJobs should be.

How: step by step

1. Ask: does it ever finish? Yes: Job or CronJob. No: keep going.
2. Ask: does it run on a clock? Yes: CronJob.
3. Ask: must it run on every node? Yes: DaemonSet.
4. Ask: does each copy need its own name or disk? Yes: StatefulSet.
5. Otherwise: Deployment. Most web APIs and workers.
6. Never write ReplicaSets or RCs. Let Deployments make them.
7. Check with a lab. Delete pods, scale, update, and watch what each does.

Example 1: Run the same app as Deployment, StatefulSet and DaemonSet

Goal. Run the same app as Deployment, StatefulSet and DaemonSet.

compare-long-running.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: compare
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deploy
  namespace: compare
spec:
  replicas: 3
  selector:
    matchLabels: {app: demo, kind: deploy}
  template:
    metadata:
      labels: {app: demo, kind: deploy}
    spec:
      containers:
        - name: web
          image: nginx:1.27
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 64Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: app-sts
  namespace: compare
spec:
  clusterIP: None
  selector: {app: demo, kind: sts}
  ports:
    - port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: app-sts
  namespace: compare
spec:
  serviceName: app-sts
  replicas: 3
  selector:
    matchLabels: {app: demo, kind: sts}
  template:
    metadata:
      labels: {app: demo, kind: sts}
    spec:
      containers:
        - name: web
          image: nginx:1.27
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 64Mi}
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Mi
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: app-ds
  namespace: compare
spec:
  selector:
    matchLabels: {app: demo, kind: ds}
  template:
    metadata:
      labels: {app: demo, kind: ds}
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: web
          image: nginx:1.27
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 64Mi}

What each line does

• One namespace — Everything in compare, so cleanup is one command.
• Same image — nginx:1.27 everywhere; only the controller changes.
• kind label — deploy, sts or ds, so we can filter pods.
• Headless Service — Required by the StatefulSet.
• volumeClaimTemplates — Only the StatefulSet gets per-pod disks.
• Toleration — Lets the DaemonSet reach the control plane too.

Green output

$ kubectl apply -f compare-long-running.yaml
$ kubectl get pods -n compare -L kind -o wide --sort-by=.metadata.labels.kind
NAME                          READY   STATUS    NODE                 KIND
app-deploy-5d8c9b7f6-2xk9p    1/1     Running   ch59-worker          deploy
app-deploy-5d8c9b7f6-8mq4t    1/1     Running   ch59-worker2         deploy
app-deploy-5d8c9b7f6-tz7lw    1/1     Running   ch59-worker          deploy
app-ds-4gq8n                  1/1     Running   ch59-control-plane   ds
app-ds-m2c7d                  1/1     Running   ch59-worker          ds
app-ds-x9w5r                  1/1     Running   ch59-worker2         ds
app-sts-0                     1/1     Running   ch59-worker2         sts
app-sts-1                     1/1     Running   ch59-worker          sts
app-sts-2                     1/1     Running   ch59-worker2         sts
$ kubectl get pvc -n compare
NAME             STATUS   CAPACITY
data-app-sts-0   Bound    50Mi
data-app-sts-1   Bound    50Mi
data-app-sts-2   Bound    50Mi

Expected result. Deployment: random names, two on one node. StatefulSet: ordered names and a disk each. DaemonSet: exactly one per node, including the control plane.

Example 2: Compare healing, scaling and updates

Goal. Compare healing, scaling and updates.

compare.sh

#!/usr/bin/env bash
set -euo pipefail
NS=compare
echo "== delete one pod of each"
kubectl delete pod -n $NS app-sts-1 --wait=false
kubectl delete pod -n $NS $(kubectl get pod -n $NS -l kind=deploy -o name | head -1 | cut -d/ -f2) --wait=false
kubectl delete pod -n $NS $(kubectl get pod -n $NS -l kind=ds -o name | head -1 | cut -d/ -f2) --wait=false
sleep 10
kubectl get pods -n $NS -L kind --sort-by=.metadata.creationTimestamp | tail -3
echo "== scale"
kubectl scale deploy/app-deploy -n $NS --replicas=5
kubectl scale sts/app-sts -n $NS --replicas=4
kubectl scale ds/app-ds -n $NS --replicas=5 || echo "DaemonSet cannot be scaled"
echo "== update image and time it"
for o in deploy/app-deploy sts/app-sts ds/app-ds; do
  kubectl set image $o -n $NS web=nginx:1.27-alpine >/dev/null
  start=$(date +%s)
  kubectl rollout status $o -n $NS --timeout=180s >/dev/null
  echo "$o updated in $(( $(date +%s) - start ))s"
done

What each line does

• set -euo pipefail — Stop on the first error, except where we allow it with ||.
• delete app-sts-1 — Comes back with the same name and PVC.
• scale ds — Fails on purpose: a DaemonSet has no replicas.
• rollout status for all — The same command works for all three.
• Timing — Deployment updates in parallel (surge); StatefulSet goes one by one; DaemonSet one node at a time.

Green output

$ bash compare.sh
== delete one pod of each
app-sts-1                    1/1   Running   0   9s    sts
app-deploy-5d8c9b7f6-q4h8n   1/1   Running   0   9s    deploy
app-ds-k7t2m                 1/1   Running   0   9s    ds
== scale
deployment.apps/app-deploy scaled
statefulset.apps/app-sts scaled
error: no scaler has been implemented for {apps DaemonSet}
DaemonSet cannot be scaled
== update image and time it
deploy/app-deploy updated in 9s
sts/app-sts updated in 22s
ds/app-ds updated in 14s

Expected result. All three healed. Only the StatefulSet pod kept its old name. The StatefulSet was the slowest to update, which is the price of safety for databases.

Example 3: Batch work: Job and CronJob, and the wrong way with a Deployment

Goal. Batch work: Job and CronJob, and the wrong way with a Deployment.

compare-batch.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: fare-import
  namespace: compare
spec:
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: task
          image: busybox:1.36
          command: ["sh", "-c", "echo importing fares; sleep 3; echo done"]
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: fare-refresh
  namespace: compare
spec:
  schedule: "*/2 * * * *"
  timeZone: "Asia/Kolkata"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: task
              image: busybox:1.36
              command: ["sh", "-c", "echo refreshing fares; sleep 3; echo done"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fare-import-wrong
  namespace: compare
spec:
  replicas: 1
  selector:
    matchLabels: {app: wrong}
  template:
    metadata:
      labels: {app: wrong}
    spec:
      containers:
        - name: task
          image: busybox:1.36
          command: ["sh", "-c", "echo importing fares; sleep 3; echo done"]

What each line does

• Job — Runs once, ends Complete.
• CronJob every 2 minutes — Makes a new Job each time.
• fare-import-wrong — The same script in a Deployment: it exits, gets restarted, and imports again and again.
• Lesson — Work that ends belongs in a Job.

Green output

$ kubectl apply -f compare-batch.yaml
$ sleep 150; kubectl get jobs,pods -n compare -l 'app notin (demo)'
NAME                              STATUS     COMPLETIONS
job.batch/fare-import             Complete   1/1
job.batch/fare-refresh-29358424   Complete   1/1
NAME                                     READY   STATUS             RESTARTS
pod/fare-import-wrong-6c4d9f8b7-p2v9k    0/1     CrashLoopBackOff   4

Expected result. The Job ran once. The CronJob made one Job per 2 minutes. The Deployment imported fares 5 times and is now in CrashLoopBackOff.

Every command, step by step (kind lab with kubectl)

1. Apply the long-running set

   kubectl apply -f compare-long-running.yaml
   kubectl get deploy,sts,ds -n compare
   

   What it does: One line per controller.

   deployment.apps/app-deploy   3/3   3   3
   statefulset.apps/app-sts     3/3
   daemonset.apps/app-ds        3     3     3
   

2. Show owners of every pod

   kubectl get pods -n compare -o custom-columns=POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind
   

   What it does: Deployment pods are owned by a ReplicaSet, not the Deployment.

   POD                          OWNER
   app-deploy-5d8c9b7f6-2xk9p   ReplicaSet
   app-ds-4gq8n                 DaemonSet
   app-sts-0                    StatefulSet
   

3. Compare DNS

   kubectl run dns -n compare --rm -it --image=busybox:1.36 --restart=Never -- nslookup app-sts-0.app-sts
   

   What it does: Only StatefulSet pods get stable names.

   Name:   app-sts-0.app-sts.compare.svc.cluster.local
   

4. Run the compare script

   bash compare.sh
   

   What it does: Healing, scaling and update timing side by side.

   deploy/app-deploy updated in 9s
   sts/app-sts updated in 22s
   ds/app-ds updated in 14s
   

5. Apply the batch set

   kubectl apply -f compare-batch.yaml
   kubectl get job,cronjob -n compare
   

   What it does: Job and CronJob, plus the wrong Deployment.

   job.batch/fare-import   Complete   1/1
   cronjob.batch/fare-refresh   */2 * * * *   Asia/Kolkata
   

6. Count restarts of the wrong one

   kubectl get pods -n compare -l app=wrong
   

   What it does: Proof that batch work in a Deployment loops.

   fare-import-wrong-6c4d9f8b7-p2v9k   0/1   CrashLoopBackOff   4
   

7. Delete and compare leftovers

   kubectl delete deploy,sts,ds,job,cronjob --all -n compare
   kubectl get pvc -n compare
   

   What it does: Only StatefulSet PVCs remain.

   data-app-sts-0   Bound   50Mi
   data-app-sts-1   Bound   50Mi
   data-app-sts-2   Bound   50Mi
   data-app-sts-3   Bound   50Mi
   

8. Clean up

   kubectl delete namespace compare
   

   What it does: Deleting the namespace deletes the PVCs too.

   namespace "compare" deleted
   

9. Delete the cluster when done

   kind delete cluster --name ch59
   

   What it does: End of chapters 59 and 60. Chapter 61 makes a fresh cluster.

   Deleting cluster "ch59" ...
   

When it is red 1

pod/fare-import-wrong   0/1   CrashLoopBackOff

Cause. Run-to-completion work was put in a Deployment.

Fix, step by step

1. Move it to a Job, or a CronJob if it repeats.

When it is red 2

MySQL data lost after pod restart (Deployment with emptyDir)

Cause. A stateful app was run as a Deployment without persistent storage.

Fix, step by step

1. Use a StatefulSet with volumeClaimTemplates, or a managed database like RDS.

When it is red 3

log shipper missing on 4 of 20 nodes (Deployment replicas: 20)

Cause. The scheduler packed some nodes twice and skipped others.

Fix, step by step

1. Use a DaemonSet; add tolerations for tainted nodes.

When it is red 4

web API rollout takes 25 minutes (StatefulSet with 50 replicas)

Cause. A stateless app was put in a StatefulSet, so it updates one pod at a time.

Fix, step by step

1. Move it to a Deployment with maxSurge 25%.
2. Keep StatefulSets for apps that need identity.

Practice tasks

kubectl get deploy,sts,ds -n compare

kubectl get pods -n compare -L kind

kubectl get pods -n compare -o custom-columns=POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind

kubectl delete pod -n compare app-sts-1

kubectl scale ds/app-ds -n compare --replicas=5

bash compare.sh

kubectl get pods -n compare -l app=wrong

kubectl get pvc -n compare

kind: StatefulSet  # redis keeps saved data

kubectl get pods -n compare -o wide

Recap

• Ends? Job. Ends on a clock? CronJob. Every node? DaemonSet. Own name or disk? StatefulSet. Otherwise, Deployment.
• Deployments own ReplicaSets; never write RS or RC by hand.
• StatefulSet PVCs outlive their pods and controllers.
• DaemonSets cannot be scaled; nodes decide the count.
• Batch work in a Deployment loops forever.
• Chapter 61 goes deep into the storage that StatefulSets depend on.

FAQ

Q. Can I use a Deployment with a PVC?

Yes, for one replica with ReadWriteOnce, or many with ReadWriteMany storage like EFS. But pods still have no stable identity.

Q. Redis cache: Deployment or StatefulSet?

Pure cache that can be lost: Deployment. Saved data or cluster mode: StatefulSet or an operator.

Q. Where do operators fit?

They create StatefulSets, Services and Jobs for you, and add backups and failover.

Q. What about Argo Rollouts?

It replaces Deployments for canary and blue-green releases. See 64.6.

Q. Should I use bare pods?

Only for kubectl run debug sessions.

Q. Kafka or Elasticsearch?

StatefulSet, usually through an operator like Strimzi or ECK.

Interview questions

1. How do you choose between Deployment, StatefulSet and DaemonSet?

   Model answer: Stateless and interchangeable: Deployment. Stable identity or per-pod storage: StatefulSet. One per node: DaemonSet. Finite work: Job; scheduled: CronJob.

2. Why not run everything as a Deployment?

   Model answer: Databases lose identity and storage, node agents miss nodes, and batch work restarts endlessly.

3. What owns a Deployment's pods?

   Model answer: A ReplicaSet, which is owned by the Deployment; ownerReferences show the chain.

4. Which controllers keep PVCs after deletion?

   Model answer: StatefulSets, through volumeClaimTemplates and the default Retain retention policy.

5. Name controllers you use in production and why.

   Model answer: Deployments for APIs, StatefulSets via operators for Kafka or Postgres, DaemonSets for Fluent Bit and node-exporter, CronJobs for backups and reports, Jobs for migrations.

Samajla ka?