Chapter 60: Kubernetes Workload Controllers in Depth: ReplicaSet, Deployment, StatefulSet, DaemonSet, Jobs and CronJobs

60.4 StatefulSet: stable names, headless Service, volumeClaimTemplates and a MySQL lab

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Paytm runs MySQL for its wallet service inside Kubernetes with a Deployment. A pod dies and the new one gets a random name like mysql-7f9c-x2k. Its disk is gone because it used the pod's own storage, or two pods try to write to the same disk at once. Replicas cannot find "the primary" because names change on every restart. Databases, Kafka, ZooKeeper, Elasticsearch and Redis clusters need three things a Deployment does not give: a stable name, a stable disk per pod, and a fixed start and stop order. A StatefulSet gives exactly that. Pods are called mysql-0, mysql-1, mysql-2, always. Each one gets its own PersistentVolumeClaim, data-mysql-0 and so on, that stays even if the pod is deleted. A headless Service gives each pod its own DNS name, like mysql-0.mysql.shop.svc.cluster.local. In this lesson we run a StatefulSet on kind, write data, kill the pod, and see the data survive. We scale it, update it with partitions, and break it with a missing headless Service and a stuck PVC. Chapter 61 covers storage in full; here we focus on the controller.

What: every term in plain English

• StatefulSet — An apps/v1 controller for pods that need stable identity and stable storage.
• Ordinal — The number in the pod name: mysql-0, mysql-1. It never changes for that slot.
• Headless Service — A Service with clusterIP: None. DNS returns pod IPs and gives each pod its own name.
• serviceName — The headless Service the StatefulSet uses for pod DNS names.
• volumeClaimTemplates — A PVC blueprint. The StatefulSet makes one PVC per pod, named TEMPLATE-STS-ORDINAL.
• podManagementPolicy — OrderedReady (default: one by one, in order) or Parallel.
• updateStrategy — RollingUpdate (default, highest ordinal first) or OnDelete.
• partition — Only pods with ordinal >= partition get the new version. Used for canary updates.
• persistentVolumeClaimRetentionPolicy — Whether PVCs are deleted when the StatefulSet is deleted or scaled down. Default: Retain.
• StorageClass standard — kind's built-in local-path provisioner that creates volumes automatically.

How: step by step

1. Create the headless Service first. clusterIP: None and the same labels as the pods.
2. Write the StatefulSet. serviceName, replicas, selector, template.
3. Add volumeClaimTemplates. Size, access mode and storageClassName.
4. Put the password in a Secret. Never hard-code it in the YAML.
5. Apply and watch the order. mysql-0 must be Ready before mysql-1 starts.
6. Test identity. Delete a pod; the same name and the same PVC come back.
7. Update safely. Use partition to update the last pod first.

Example 1: Run MySQL as a StatefulSet with its own disk

Goal. Run MySQL as a StatefulSet with its own disk.

mysql-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mysql-auth
  namespace: shop
type: Opaque
stringData:
  MYSQL_ROOT_PASSWORD: change-me-lab-only

mysql-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: shop
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
    - name: mysql
      port: 3306

mysql-sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: shop
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: mysql
          image: mysql:8.4
          envFrom:
            - secretRef:
                name: mysql-auth
          ports:
            - name: mysql
              containerPort: 3306
          readinessProbe:
            exec:
              command: ["sh", "-c", "mysqladmin ping -h 127.0.0.1 -uroot -p$MYSQL_ROOT_PASSWORD"]
            initialDelaySeconds: 15
            periodSeconds: 5
          resources:
            requests: {cpu: 100m, memory: 512Mi}
            limits: {memory: 1Gi}
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: standard
        resources:
          requests:
            storage: 1Gi

What each line does

• stringData — Plain text in the file; Kubernetes stores it base64. Lab password only; use a real secret store in production.
• clusterIP: None — Makes the Service headless, so each pod gets a DNS name.
• serviceName: mysql — Must equal the headless Service name.
• envFrom secretRef — Loads MYSQL_ROOT_PASSWORD from the Secret.
• readinessProbe mysqladmin ping — The pod is Ready only when MySQL answers.
• volumeClaimTemplates name: data — Creates PVC data-mysql-0, mounted at /var/lib/mysql.
• storageClassName: standard — kind's default class makes the volume for you.

Green output

$ kubectl apply -f mysql-secret.yaml -f mysql-headless.yaml -f mysql-sts.yaml
secret/mysql-auth created
service/mysql created
statefulset.apps/mysql created
$ kubectl get pods,pvc -n shop -l app=mysql
NAME          READY   STATUS    RESTARTS   AGE
pod/mysql-0   1/1     Running   0          45s
$ kubectl get pvc -n shop
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
data-mysql-0   Bound    pvc-5d0e2c1a-8f3b-4a7e-9c6d-1b2a3f4e5d6c   1Gi        RWO            standard
$ kubectl exec -n shop mysql-0 -- sh -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "CREATE DATABASE wallet; CREATE TABLE wallet.tx(id INT, amt INT); INSERT INTO wallet.tx VALUES (1,500);"'
$ kubectl delete pod mysql-0 -n shop
pod "mysql-0" deleted
$ kubectl exec -n shop mysql-0 -- sh -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD -N -e "SELECT * FROM wallet.tx"'
1   500

Expected result. Same name mysql-0, same PVC data-mysql-0, same data. A Deployment pod with emptyDir would have lost the row.

Example 2: Ordered start, stable DNS names and scaling

Goal. Ordered start, stable DNS names and scaling.

web-sts.yaml

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  clusterIP: None
  selector:
    app: web-sts
  ports:
    - port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: shop
spec:
  serviceName: web
  replicas: 3
  podManagementPolicy: OrderedReady
  selector:
    matchLabels:
      app: web-sts
  template:
    metadata:
      labels:
        app: web-sts
    spec:
      containers:
        - name: nginx
          image: nginx:1.26
          command: ["sh", "-c", "echo $(hostname) > /usr/share/nginx/html/index.html; exec nginx -g 'daemon off;'"]
          readinessProbe:
            httpGet: {path: /, port: 80}
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Mi

What each line does

• Two objects in one file — The headless Service and the StatefulSet, separated by ---.
• OrderedReady — web-0 must be Ready before web-1 is created.
• echo $(hostname) — Each pod writes its own name, so we can see which pod answers.
• exec nginx — nginx becomes PID 1 and gets SIGTERM directly.
• No storageClassName — Uses the cluster default class, standard on kind.

Green output

$ kubectl apply -f web-sts.yaml
$ kubectl get pods -n shop -l app=web-sts -w
web-0   0/1   ContainerCreating   0   0s
web-0   1/1   Running             0   4s
web-1   0/1   Pending             0   0s
web-1   1/1   Running             0   4s
web-2   0/1   Pending             0   0s
web-2   1/1   Running             0   4s
$ kubectl run dns -n shop --rm -it --image=busybox:1.36 --restart=Never -- sh -c 'nslookup web-1.web.shop.svc.cluster.local; wget -qO- web-1.web'
Name:   web-1.web.shop.svc.cluster.local
Address: 10.244.2.17
web-1
$ kubectl scale sts web -n shop --replicas=1
$ kubectl get pvc -n shop -l app=web-sts
NAME          STATUS   VOLUME   CAPACITY
www-web-0     Bound    pvc-...  100Mi
www-web-1     Bound    pvc-...  100Mi
www-web-2     Bound    pvc-...  100Mi

Expected result. Pods start in order and each has a DNS name. Scaling down removes web-2 then web-1, but their PVCs stay, so scaling up again finds the old data.

Example 3: Canary update with partition, then full update

Goal. Canary update with partition, then full update.

web-partition-patch.yaml

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2
  template:
    spec:
      containers:
        - name: nginx
          image: nginx:1.27

What each line does

• partition: 2 — Only pods with ordinal 2 or more get nginx:1.27.
• web-0 and web-1 — Stay on 1.26 even if deleted, until partition is lowered.
• Order — RollingUpdate goes from the highest ordinal down: web-2, then web-1, then web-0.
• Full rollout — Set partition to 0.

Green output

$ kubectl scale sts web -n shop --replicas=3
$ kubectl patch sts web -n shop --type=strategic --patch-file web-partition-patch.yaml
statefulset.apps/web patched
$ kubectl get pods -n shop -l app=web-sts -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
NAME    IMAGE
web-0   nginx:1.26
web-1   nginx:1.26
web-2   nginx:1.27
$ kubectl patch sts web -n shop -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
$ kubectl rollout status sts/web -n shop
partitioned roll out complete: 3 new pods have been updated...

Expected result. web-2 was the canary. After checking it, partition 0 updated web-1 then web-0. Databases often use this to update replicas before the primary.

Every command, step by step (kind lab with kubectl)

1. Check the default StorageClass

   kubectl get storageclass
   

   What it does: kind ships standard with the local-path provisioner.

   NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
   standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer
   

2. Apply MySQL

   kubectl apply -f mysql-secret.yaml -f mysql-headless.yaml -f mysql-sts.yaml
   kubectl rollout status sts/mysql -n shop
   

   What it does: Waits until mysql-0 is Ready.

   partitioned roll out complete: 1 new pods have been updated...
   

3. See the PVC name rule

   kubectl get pvc -n shop
   

   What it does: Template name, StatefulSet name, ordinal.

   data-mysql-0   Bound   1Gi   RWO   standard
   

4. Find the node and volume

   kubectl get pod mysql-0 -n shop -o wide
   kubectl get pv
   

   What it does: local-path volumes live on one node; the pod always returns there.

   mysql-0   1/1   Running   0   2m   10.244.1.20   ch59-worker
   

5. Connect by DNS

   kubectl run cli -n shop --rm -it --image=mysql:8.4 --restart=Never --env=P=change-me-lab-only -- sh -c 'mysql -h mysql-0.mysql -uroot -p$P -e "SELECT 1"'
   

   What it does: Clients reach a specific pod by name.

   1
   1
   

6. Kill and check identity

   kubectl delete pod mysql-0 -n shop
   kubectl get pod mysql-0 -n shop -o jsonpath='{.spec.volumes[0].persistentVolumeClaim.claimName}{"\n"}'
   

   What it does: Same PVC is mounted again.

   data-mysql-0
   

7. Read the StatefulSet status

   kubectl get sts -n shop
   kubectl describe sts web -n shop | grep -i -A2 'update strategy'
   

   What it does: READY shows ready/desired.

   NAME    READY   AGE
   mysql   1/1     5m
   web     3/3     3m
   

8. Delete and keep data

   kubectl delete sts web -n shop
   kubectl get pvc -n shop -l app=web-sts
   

   What it does: Pods go; PVCs stay by default.

   www-web-0   Bound   100Mi
   

9. Clean up fully

   kubectl delete sts mysql -n shop
   kubectl delete svc mysql web -n shop
   kubectl delete secret mysql-auth -n shop
   kubectl delete pvc -n shop data-mysql-0 www-web-0 www-web-1 www-web-2
   

   What it does: PVCs must be deleted by hand. Their volumes go too, because the reclaim policy is Delete.

   persistentvolumeclaim "data-mysql-0" deleted
   

When it is red 1

nslookup: can't resolve 'mysql-0.mysql'

Cause. The headless Service is missing, has a different name from serviceName, or is not headless.

Fix, step by step

1. kubectl get svc mysql -n shop -o jsonpath='{.spec.clusterIP}' must print None.
2. Make serviceName equal the Service name.
3. Re-create the Service with clusterIP: None (clusterIP cannot be changed in place).

When it is red 2

mysql-0   0/1   Pending
Warning  FailedScheduling  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims

Cause. No StorageClass can create the volume, or the class name is wrong.

Fix, step by step

1. kubectl get sc and check the name.
2. kubectl describe pvc data-mysql-0 -n shop for the provisioner error.
3. Fix storageClassName; delete the stuck PVC and pod so they are made again.

When it is red 3

web-1 never starts; web-0 is 0/1 Running

Cause. OrderedReady waits for web-0 to be Ready. Its readiness probe fails.

Fix, step by step

1. kubectl describe pod web-0 for probe errors.
2. Fix the probe, or use podManagementPolicy: Parallel if order does not matter.

When it is red 4

The StatefulSet "mysql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Cause. You changed volumeClaimTemplates or serviceName on a live StatefulSet.

Fix, step by step

1. Delete the StatefulSet with --cascade=orphan, keeping pods and PVCs.
2. Apply the new spec; it adopts the pods.
3. To grow disks, edit each PVC (61.7).

Practice tasks

kubectl rollout status sts/mysql -n shop

kubectl exec -n shop mysql-0 -- sh -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "SELECT * FROM wallet.tx"'

kubectl delete pod mysql-0 -n shop

kubectl run dns -n shop --rm -it --image=busybox:1.36 --restart=Never -- nslookup web-2.web

kubectl get pods -n shop -l app=web-sts -w

podManagementPolicy: Parallel

kubectl get pvc -n shop

kubectl patch sts web -n shop --type=strategic --patch-file web-partition-patch.yaml

kubectl patch sts web -n shop -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete"}}}'

kubectl patch sts web -n shop -p '{"spec":{"updateStrategy":{"type":"OnDelete","rollingUpdate":null}}}'

Recap

• StatefulSets give each pod a stable name, DNS name and disk.
• A headless Service is required and must match serviceName.
• volumeClaimTemplates make one PVC per pod; PVCs survive pod and StatefulSet deletion.
• Pods start in order and update from the highest ordinal; partition gives canaries.
• Most fields are immutable; orphan-delete and re-apply to change them.
• On AWS, managed RDS is often simpler than MySQL in a StatefulSet; choose deliberately.

FAQ

Q. Does a StatefulSet make MySQL replication work?

No. It gives identity and disks. Replication is configured by the database or an operator.

Q. Should I run production databases in Kubernetes?

It works with good operators and storage, but many teams on AWS prefer RDS for backups and failover.

Q. Why does mysql-0 always go to the same node on kind?

local-path volumes are tied to one node. EBS volumes are tied to one availability zone.

Q. Can I use a normal ClusterIP Service too?

Yes. Many setups have a headless one for pod names and a normal one for read traffic.

Q. What is an operator?

A controller for one app, like the MySQL or Strimzi Kafka operator, that adds backups and failover on top of StatefulSets.

Q. How do I grow the disk?

Edit the PVC size if the StorageClass allows expansion. See 61.7.

Interview questions

1. Deployment vs StatefulSet?

   Model answer: Deployments have random, interchangeable pods and shared or no storage; StatefulSets have stable ordinal names, per-pod PVCs, stable DNS through a headless Service, and ordered start, stop and update.

2. Why is a headless Service needed?

   Model answer: It creates DNS records for each pod (pod.service.ns.svc.cluster.local) so peers and clients can reach a specific member.

3. What happens to PVCs when a StatefulSet scales down?

   Model answer: By default they are retained, so data returns on scale-up; persistentVolumeClaimRetentionPolicy can change this.

4. How do you do a canary update of a StatefulSet?

   Model answer: Set rollingUpdate.partition so only higher ordinals update, verify, then lower the partition to 0.

5. A StatefulSet pod is stuck Pending. What do you check?

   Model answer: PVC status and events, StorageClass name, provisioner, node capacity and zone affinity of the volume.

Samajla ka?