Chapter 61: Kubernetes Storage in Depth: Volumes, PV, PVC, StorageClass, CSI and Snapshots

61.5 StorageClass and dynamic provisioning: kind local-path and AWS EBS CSI gp3

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Flipkart has 300 microservices, and many of them need a disk. If the platform team had to create every PV by hand, as in 61.4, they would spend their whole Big Billion Days week making disks. Dynamic provisioning removes that work: a developer writes a PVC, and Kubernetes creates the real disk and the PV automatically. The object that makes this possible is the StorageClass. It names a provisioner (a CSI driver like ebs.csi.aws.com or kind's rancher.io/local-path), the parameters for new disks (type gp3, iops, encryption), the reclaim policy, whether volumes can be expanded, and when to bind. One class can be marked as the default, so claims with no storageClassName use it. volumeBindingMode is the setting that confuses most people. Immediate creates the disk the moment the PVC appears, possibly in the wrong availability zone. WaitForFirstConsumer waits until a pod using the claim is scheduled, then creates the disk in that pod's zone. On AWS, where an EBS volume lives in one zone, WaitForFirstConsumer is the safe choice. In this lesson we read kind's default class, create our own class with Retain, change the default, and then walk through the real EKS setup with the EBS CSI driver and an encrypted gp3 class.

What: every term in plain English

• StorageClass — A template for creating PVs on demand.
• provisioner — The driver that creates disks, like ebs.csi.aws.com.
• parameters — Driver-specific settings such as type: gp3 or encrypted: "true".
• reclaimPolicy — Delete (default) or Retain for PVs made by this class.
• allowVolumeExpansion — If true, PVCs of this class can be resized.
• volumeBindingMode — Immediate or WaitForFirstConsumer.
• Default StorageClass — The class with annotation storageclass.kubernetes.io/is-default-class: "true".
• Dynamic provisioning — A PVC triggers the creation of a new PV and real disk.
• CSI (Container Storage Interface) — The standard plugin API every modern storage driver uses.
• EBS CSI driver — The AWS add-on that creates, attaches and resizes EBS volumes for EKS.

How: step by step

1. Inspect kind's default StorageClass. Read provisioner, binding mode and policy.
2. Create a PVC with no class and watch it wait. WaitForFirstConsumer in action.
3. Create a Retain class and use it. Data survives PVC deletion.
4. Move the default to the new class. Remove the annotation from one, add it to the other.
5. Read the EKS version of the same setup. EBS CSI add-on, IAM role and gp3 class.
6. Clean up and restore the default. Keep the cluster for 61.6.

Example 1: See dynamic provisioning and WaitForFirstConsumer on kind

Goal. See dynamic provisioning and WaitForFirstConsumer on kind.

cart-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cart-cache
  namespace: storage
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 200Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: cart
  namespace: storage
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: cart-cache
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}

What each line does

• No storageClassName — The default class standard is used.
• Pending first — With WaitForFirstConsumer, nothing is created until a pod needs it.
• Pod scheduled — The provisioner creates a PV on the chosen node.
• pvc-<uid> name — Dynamic PVs are named after the claim's uid.
• Exact size — The PV is 200Mi, not bigger.

Green output

$ kubectl get storageclass standard -o yaml | grep -E 'provisioner|reclaimPolicy|volumeBindingMode|is-default'
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
$ kubectl apply -f cart-pvc.yaml
$ kubectl get pvc,pv -n storage
NAME                               STATUS   VOLUME                                     CAPACITY   STORAGECLASS
persistentvolumeclaim/cart-cache   Bound    pvc-2b8e41f0-6c7a-4d3e-9a51-7f0c8e2d6b19   200Mi      standard
NAME                                                        CAPACITY   RECLAIM POLICY   STATUS
persistentvolume/pvc-2b8e41f0-6c7a-4d3e-9a51-7f0c8e2d6b19   200Mi      Delete           Bound

Expected result. The developer wrote only a PVC and a pod. The local-path provisioner made the PV, with exactly 200Mi and the class's Delete policy.

Example 2: Create a Retain StorageClass and make it the default

Goal. Create a Retain StorageClass and make it the default.

retain-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false

What each line does

• Same provisioner — Still local-path; only the policy changes.
• reclaimPolicy: Retain — PVs from this class keep data after the PVC is deleted.
• Default switch — Exactly one class should carry the default annotation.
• Two defaults — Kubernetes picks the newest one and warns; avoid it.
• Existing PVs — Changing a class never changes PVs already created.

Green output

$ kubectl apply -f retain-class.yaml
storageclass.storage.k8s.io/local-retain created
$ kubectl patch storageclass standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
$ kubectl patch storageclass local-retain -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
$ kubectl get storageclass
NAME                     PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-retain (default)   rancher.io/local-path   Retain          WaitForFirstConsumer
standard                 rancher.io/local-path   Delete          WaitForFirstConsumer
$ kubectl delete pod,pvc -n storage cart cart-cache; kubectl apply -f cart-pvc.yaml
$ kubectl delete pod,pvc -n storage cart cart-cache
$ kubectl get pv
NAME                                       CAPACITY   RECLAIM POLICY   STATUS     STORAGECLASS
pvc-91d3c7aa-0e4f-4b62-8f15-c3a7d2e9b804   200Mi      Retain           Released   local-retain

Expected result. New claims now use local-retain, and deleting the claim left the PV Released with its data, as we saw in 61.4.

Example 3: Set up encrypted gp3 volumes on Amazon EKS with the EBS CSI driver

Goal. Set up encrypted gp3 volumes on Amazon EKS with the EBS CSI driver.

gp3-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

What each line does

• ebs.csi.aws.com — The EBS CSI driver, installed as an EKS add-on.
• type: gp3 — Cheaper than gp2 with 3000 IOPS baseline.
• encrypted: "true" — Uses the default aws/ebs KMS key unless kmsKeyId is set.
• WaitForFirstConsumer — Creates the disk in the same zone as the pod.
• allowVolumeExpansion — Lets you grow disks later (61.7).

Green output

$ eksctl create iamserviceaccount --cluster flipkart-demo --namespace kube-system \
    --name ebs-csi-controller-sa --role-name EKS_EBS_CSI_Role --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --approve
$ aws eks create-addon --cluster-name flipkart-demo --addon-name aws-ebs-csi-driver \
    --service-account-role-arn arn:aws:iam::111122223333:role/EKS_EBS_CSI_Role
$ kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
NAME                                  READY   STATUS
ebs-csi-controller-6d9c8b7f5-k2lmn    6/6     Running
ebs-csi-node-7xq4p                    3/3     Running
$ kubectl apply -f gp3-class.yaml
$ kubectl annotate storageclass gp2 storageclass.kubernetes.io/is-default-class=false --overwrite
$ kubectl apply -f cart-pvc.yaml
$ kubectl get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]
NAME                                       SIZE    ZONE
pvc-5e0a7b33-1d9c-4f28-b6e4-0c2f9a8d7e51   200Mi   ap-south-1a

Expected result. The account id 111122223333 is the AWS documentation placeholder. EBS rounds up to 1 GiB, and the PV's nodeAffinity pins it to the pod's zone, ap-south-1a (Mumbai).

Every command, step by step (kind lab with kubectl)

1. List classes

   kubectl get storageclass
   

   What it does: One default on kind.

   standard (default)   rancher.io/local-path   Delete   WaitForFirstConsumer   false
   

2. Read the default class

   kubectl get sc standard -o yaml
   

   What it does: Note provisioner and binding mode.

   provisioner: rancher.io/local-path
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   

3. Create only the claim first

   csplit -s -f part cart-pvc.yaml '/^---$/'
   kubectl apply -f part00
   kubectl get pvc -n storage cart-cache
   

   What it does: csplit cuts the file at ---; part00 is the claim only. Alone, it stays Pending.

   cart-cache   Pending   standard
   

4. Read why it waits

   kubectl describe pvc -n storage cart-cache | tail -1
   

   What it does: This is expected.

   Normal  WaitForFirstConsumer  persistentvolume-controller  waiting for first consumer to be created before binding
   

5. See the provisioner work

   kubectl apply -f cart-pvc.yaml
   kubectl get events -n storage --field-selector involvedObject.name=cart-cache
   

   What it does: Provisioning then success.

   Normal  Provisioning           rancher.io/local-path_...  External provisioner is provisioning volume for claim "storage/cart-cache"
   Normal  ProvisioningSucceeded  rancher.io/local-path_...  Successfully provisioned volume pvc-2b8e41f0-...
   

6. Find the folder on the node

   kubectl get pv -o jsonpath='{.items[0].spec.hostPath.path}{"\n"}'
   

   What it does: local-path keeps data under /var/local-path-provisioner.

   /var/local-path-provisioner/pvc-2b8e41f0-..._storage_cart-cache
   

7. Create the Retain class

   kubectl apply -f retain-class.yaml
   kubectl get sc
   

   What it does: Two classes now.

   local-retain   rancher.io/local-path   Retain   WaitForFirstConsumer
   standard (default)   rancher.io/local-path   Delete   WaitForFirstConsumer
   

8. Switch the default

   kubectl patch sc standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
   kubectl patch sc local-retain -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
   

   What it does: Exactly one default.

   local-retain (default)
   

9. Prove Retain

   kubectl delete pod,pvc -n storage cart cart-cache
   kubectl apply -f cart-pvc.yaml; kubectl delete pod,pvc -n storage cart cart-cache
   kubectl get pv
   

   What it does: One Released PV from local-retain.

   pvc-91d3c7aa-...   200Mi   Retain   Released   local-retain
   

10. Restore the default

    kubectl patch sc local-retain -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
    kubectl patch sc standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
    

    What it does: Later lessons expect standard.

    standard (default)
    

11. Clean up released PVs

    kubectl get pv --no-headers | awk '$5=="Released"{print $1}' | xargs -r kubectl delete pv
    

    What it does: Retain PVs need manual delete. Keep the cluster for 61.6.

    persistentvolume "pvc-91d3c7aa-..." deleted
    

When it is red 1

waiting for first consumer to be created before binding

Cause. WaitForFirstConsumer and no pod uses the claim yet.

Fix, step by step

1. Create the pod. This is normal.

When it is red 2

failed to provision volume with StorageClass "gp3-encrypted": could not create volume in EC2: UnauthorizedOperation

Cause. The EBS CSI controller has no IAM permission.

Fix, step by step

1. Attach AmazonEBSCSIDriverPolicy to its role (IRSA or Pod Identity).

When it is red 3

0/3 nodes are available: 3 node(s) had volume node affinity conflict.

Cause. An Immediate-mode disk was made in a zone with no usable node.

Fix, step by step

1. Use WaitForFirstConsumer.
2. Add nodes in that zone.

When it is red 4

storageclass.storage.k8s.io "gp2" not found

Cause. The PVC names a class that does not exist.

Fix, step by step

1. kubectl get sc and use a real name, or remove storageClassName to use the default.

Practice tasks

kubectl get sc standard -o yaml

kubectl describe pvc -n storage cart-cache

kubectl get pv -w

docker exec ch61-worker ls /var/local-path-provisioner

kubectl get pv

kubectl get sc

kubectl describe pvc -n storage

kubectl apply --dry-run=client -f gp3-class.yaml

kubectl get deploy,pvc -n storage

kubectl explain storageclass.volumeBindingMode

Recap

• A StorageClass lets a PVC create its own PV and disk.
• The provisioner and parameters decide what kind of disk you get.
• WaitForFirstConsumer creates the disk where the pod runs.
• Keep exactly one default class.
• Next: ConfigMap, Secret and projected volumes.

FAQ

Q. Is gp3 better than gp2?

Usually yes: about 20% cheaper with 3000 IOPS baseline at any size.

Q. Do I still need in-tree kubernetes.io/aws-ebs?

No. In-tree EBS was migrated to CSI; use ebs.csi.aws.com.

Q. What does GKE or AKS use?

pd.csi.storage.gke.io on GKE and disk.csi.azure.com on AKS. Same ideas.

Q. Can a StatefulSet use a class?

Yes, volumeClaimTemplates name a storageClassName, as in 60.4.

Q. How do I encrypt with my own key?

Add kmsKeyId with the KMS key ARN to the class parameters.

Q. What about shared files across pods?

Use EFS with efs.csi.aws.com and ReadWriteMany.

Interview questions

1. What is a StorageClass?

   Model answer: A template that names a provisioner and parameters for creating PVs on demand.

2. Explain volumeBindingMode.

   Model answer: Immediate binds on claim creation; WaitForFirstConsumer waits for pod scheduling so topology matches.

3. How does EKS get EBS volumes?

   Model answer: The EBS CSI driver add-on with an IAM role creates and attaches them.

4. What happens with two default classes?

   Model answer: Kubernetes uses the newest and warns; you should keep one.

5. How would you make all new disks encrypted?

   Model answer: A default StorageClass with encrypted: "true", and an admission policy rejecting other classes.

Samajla ka?