Chapter 61: Kubernetes Storage in Depth: Volumes, PV, PVC, StorageClass, CSI and Snapshots

61.4 PersistentVolumes and PersistentVolumeClaims: access modes, reclaim policy and binding

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Paytm runs a ledger service on Kubernetes. The ledger is a PostgreSQL database, and losing its data even once would be a disaster. The platform team owns the disks; the app team only wants to say "I need 10Gi that one pod can write". Kubernetes splits these two jobs on purpose. A PersistentVolume (PV) is the real storage: an EBS volume, an NFS share, or a folder on a node. It is a cluster-wide object with its own life, created by an admin or by a StorageClass. A PersistentVolumeClaim (PVC) is a request in a namespace: size, access mode and optionally a class. Kubernetes binds each PVC to exactly one PV that fits, and the pod only mentions the claim. Three settings decide how this works. Access modes say how many nodes can mount the volume and how (ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod). The reclaim policy says what happens to the PV and its data when the PVC is deleted (Retain or Delete). And binding rules match size, mode, class, and optional labels. In this lesson we create PVs by hand (static provisioning), bind claims to them, watch a PVC stay Pending when nothing fits, delete a claim with Retain and recover the data, and use ReadWriteOncePod to stop two pods writing one database.

What: every term in plain English

• PersistentVolume (PV) — Cluster-scoped object that represents real storage.
• PersistentVolumeClaim (PVC) — Namespaced request for storage that binds to one PV.
• Static provisioning — An admin creates PVs first; claims bind to them.
• ReadWriteOnce (RWO) — Mounted read-write by pods on one node.
• ReadWriteOncePod (RWOP) — Mounted read-write by exactly one pod in the whole cluster.
• ReadOnlyMany (ROX) / ReadWriteMany (RWX) — Many nodes read, or many nodes read and write (needs NFS, EFS and similar).
• persistentVolumeReclaimPolicy — Retain keeps the PV and data after the claim is deleted; Delete removes both.
• PV phases — Available, Bound, Released, Failed.
• claimRef — The field on a PV that records which PVC it is bound to.
• volumeName / selector — Ways a PVC can ask for one exact PV or PVs with certain labels.

How: step by step

1. Use the ch61 cluster and namespace storage. Static PVs here use a class name manual so the default provisioner does not interfere.
2. Create two PVs by hand. 1Gi with Retain and 5Gi with Delete.
3. Create a PVC for 2Gi. It binds to the 5Gi PV, the smallest that fits.
4. Ask for more than any PV offers. The claim stays Pending.
5. Delete a claim on a Retain PV. The PV becomes Released and the data stays.
6. Recover the data by clearing claimRef. Bind a new claim to the same PV.
7. Use ReadWriteOncePod for a database. A second pod cannot mount it.

Example 1: Create static PVs and watch a claim bind to the best fit

Goal. Create static PVs and watch a claim bind to the best fit.

static-pvs.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-small-retain
  labels:
    tier: ledger
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/pv-small
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["ch61-worker"]
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-big-delete
spec:
  capacity:
    storage: 5Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Delete
  storageClassName: manual
  hostPath:
    path: /mnt/pv-big
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["ch61-worker"]

claim-2gi.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: reports
  namespace: storage
spec:
  storageClassName: manual
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi

What each line does

• storageClassName: manual — Only PVs and PVCs with the same class can bind; manual is just a name.
• hostPath inside a PV — Fine for a kind lab; on AWS this would be awsElasticBlockStore or a CSI volume.
• nodeAffinity — Tells the scheduler the data is on ch61-worker, so pods go there.
• 2Gi request — 1Gi is too small, so the 5Gi PV is chosen.
• Capacity is not a quota — The PVC shows 5Gi because it got the whole PV.

Green output

$ kubectl apply -f static-pvs.yaml
persistentvolume/pv-small-retain created
persistentvolume/pv-big-delete created
$ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS
pv-big-delete     5Gi        RWO            Delete           Available           manual
pv-small-retain   1Gi        RWO            Retain           Available           manual
$ kubectl apply -f claim-2gi.yaml
$ kubectl get pvc -n storage reports
NAME      STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS
reports   Bound    pv-big-delete   5Gi        RWO            manual

Expected result. The 2Gi claim could not fit in 1Gi, so it bound to the 5Gi PV and got all 5Gi. Binding is one-to-one: no other claim can use pv-big-delete now.

Example 2: Delete a claim on a Retain PV and recover the ledger data

Goal. Delete a claim on a Retain PV and recover the ledger data.

ledger.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ledger-data
  namespace: storage
spec:
  storageClassName: manual
  volumeName: pv-small-retain
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: ledger
  namespace: storage
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: ledger-data
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /var/lib/ledger
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}

What each line does

• volumeName — Pins this claim to one exact PV.
• Retain — Deleting the claim keeps the PV and the files.
• Released — The PV remembers the old claim in claimRef and refuses new ones.
• kubectl patch claimRef null — An admin step that makes the PV Available again.
• Same data — The new claim sees the old files.

Green output

$ kubectl apply -f ledger.yaml
$ kubectl exec -n storage ledger -- sh -c 'echo txn-88231 credit 500 > /var/lib/ledger/ledger.log'
$ kubectl delete pod,pvc -n storage ledger ledger-data
$ kubectl get pv pv-small-retain
NAME              CAPACITY   RECLAIM POLICY   STATUS     CLAIM
pv-small-retain   1Gi        Retain           Released   storage/ledger-data
$ kubectl patch pv pv-small-retain --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
$ kubectl get pv pv-small-retain
pv-small-retain   1Gi   Retain   Available
$ kubectl apply -f ledger.yaml
$ kubectl exec -n storage ledger -- cat /var/lib/ledger/ledger.log
txn-88231 credit 500

Expected result. Someone deleted the claim by mistake, and the money record survived because of Retain. With Delete, the PV and its files would be gone.

Example 3: Stop two writers with ReadWriteOncePod

Goal. Stop two writers with ReadWriteOncePod.

rwop.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-rwop
  namespace: storage
spec:
  accessModes: ["ReadWriteOncePod"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: db-1
  namespace: storage
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: db-rwop
  containers:
    - name: db
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}

What each line does

• ReadWriteOncePod — Only one pod in the whole cluster can use the claim.
• No storageClassName — Uses the default class. RWOP needs a driver that supports it; EBS CSI does, and recent local-path versions on kind do too.
• db-2 — Copy of db-1 with a new name, same claim.
• RWO would allow it — With RWO, two pods on the same node could both write.
• Why it matters — Two PostgreSQL processes on one data folder corrupt it.

Green output

$ kubectl apply -f rwop.yaml
$ sed 's/name: db-1/name: db-2/' rwop.yaml | kubectl apply -f -
$ kubectl get pods -n storage db-1 db-2
NAME   READY   STATUS    RESTARTS   AGE
db-1   1/1     Running   0          20s
db-2   0/1     Pending   0          8s
$ kubectl describe pod -n storage db-2 | tail -1
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node has pod using PersistentVolumeClaim with the same name and ReadWriteOncePod access mode.

Expected result. The scheduler refused to place db-2. This is the safe setting for single-writer databases.

Every command, step by step (kind lab with kubectl)

1. Create the static PVs

   kubectl apply -f static-pvs.yaml
   kubectl get pv
   

   What it does: Both Available.

   pv-big-delete     5Gi   RWO   Delete   Available   manual
   pv-small-retain   1Gi   RWO   Retain   Available   manual
   

2. Bind a 2Gi claim

   kubectl apply -f claim-2gi.yaml
   kubectl get pvc -n storage reports
   

   What it does: Bound to the 5Gi PV.

   reports   Bound   pv-big-delete   5Gi   RWO   manual
   

3. See the binding from the PV side

   kubectl get pv pv-big-delete -o jsonpath='{.spec.claimRef.namespace}/{.spec.claimRef.name}'
   

   What it does: claimRef records the PVC.

   storage/reports
   

4. Ask for too much

   printf '%s\n' 'apiVersion: v1' 'kind: PersistentVolumeClaim' 'metadata:' '  name: huge' '  namespace: storage' 'spec:' '  storageClassName: manual' '  accessModes: ["ReadWriteOnce"]' '  resources:' '    requests:' '      storage: 50Gi' > huge.yaml
   kubectl apply -f huge.yaml
   kubectl describe pvc -n storage huge | tail -1
   

   What it does: No PV fits, and manual has no provisioner.

   Normal  FailedBinding  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
   

5. Run the ledger

   kubectl apply -f ledger.yaml
   kubectl exec -n storage ledger -- sh -c 'echo txn-88231 credit 500 > /var/lib/ledger/ledger.log'
   

   What it does: Pod lands on ch61-worker because of nodeAffinity.

6. Delete the claim

   kubectl delete pod,pvc -n storage ledger ledger-data
   kubectl get pv pv-small-retain
   

   What it does: Released, data kept.

   pv-small-retain   1Gi   RWO   Retain   Released   storage/ledger-data
   

7. Check the files on the node

   docker exec ch61-worker cat /mnt/pv-small/ledger.log
   

   What it does: Still on disk.

   txn-88231 credit 500
   

8. Make the PV Available again

   kubectl patch pv pv-small-retain --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
   

   What it does: Only do this when you are sure who should get the data.

   persistentvolume/pv-small-retain patched
   

9. Rebind and read

   kubectl apply -f ledger.yaml
   kubectl exec -n storage ledger -- cat /var/lib/ledger/ledger.log
   

   What it does: Recovered.

   txn-88231 credit 500
   

10. Try ReadWriteOncePod

    kubectl apply -f rwop.yaml
    sed 's/name: db-1/name: db-2/' rwop.yaml | kubectl apply -f -
    kubectl get pods -n storage db-1 db-2
    

    What it does: db-2 stays Pending.

    db-1   1/1   Running
    db-2   0/1   Pending
    

11. Clean up

    kubectl delete pod -n storage --all
    kubectl delete pvc -n storage --all
    kubectl get pv
    

    What it does: pv-small-retain stays Released (Retain). Delete it by hand.

    pv-small-retain   1Gi   RWO   Retain   Released   storage/ledger-data
    

When it is red 1

no persistent volumes available for this claim and no storage class is set

Cause. No PV matches size, mode and class.

Fix, step by step

1. Check storageClassName on both sides.
2. Create a matching PV or use a StorageClass.

When it is red 2

pv-small-retain   Released   storage/ledger-data

Cause. The old claimRef is still set, so new claims cannot bind.

Fix, step by step

1. Back up the data, then remove spec.claimRef.

When it is red 3

Multi-Attach error for volume "pvc-..." Volume is already exclusively attached to one node

Cause. An RWO EBS volume is still attached to the old node during a reschedule.

Fix, step by step

1. Wait for detach (about 6 minutes on a dead node).
2. Use RWX storage like EFS if many nodes must mount it.

When it is red 4

1 node has pod using PersistentVolumeClaim with the same name and ReadWriteOncePod access mode

Cause. Another pod already holds an RWOP claim.

Fix, step by step

1. Delete the old pod first, or use RWO if sharing on one node is fine.

Practice tasks

kubectl get pv

kubectl get pvc -n storage reports

kubectl get pv -w

kubectl describe pvc -n storage huge

kubectl patch pv pv-small-retain --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'

kubectl get pv

kubectl get pvc -n storage

kubectl describe pod -n storage db-2

kubectl get pv,pvc -n storage

kubectl explain pv.spec.persistentVolumeReclaimPolicy

Recap

• A PV is the real storage; a PVC is a request; they bind one-to-one.
• Access modes set how many nodes or pods can mount the volume.
• Retain keeps data after the claim is gone; Delete removes it.
• storageClassName, size and mode must match for binding.
• Next: StorageClasses that create PVs on demand.

FAQ

Q. Does a PVC get exactly the size it asked for?

With static PVs it gets the whole PV, which can be bigger. With dynamic provisioning it usually gets the exact size.

Q. Can I change a PV's reclaim policy later?

Yes, with kubectl patch pv. Many teams switch production PVs to Retain.

Q. Which AWS storage supports RWX?

EFS through the EFS CSI driver. EBS is RWO only.

Q. What does Recycle do?

It was an old policy that wiped data. It is deprecated; do not use it.

Q. Can a pod use a PVC from another namespace?

No. The PVC must be in the pod's namespace.

Q. What protects a PVC in use from deletion?

The kubernetes.io/pvc-protection finalizer keeps it until no pod uses it.

Interview questions

1. Explain PV and PVC.

   Model answer: A PV is cluster storage; a PVC is a namespaced claim that binds to one PV; pods use PVCs.

2. Explain the access modes.

   Model answer: RWO one node, RWOP one pod, ROX many readers, RWX many writers.

3. Retain vs Delete?

   Model answer: Retain keeps the disk after the claim is removed; Delete removes the disk too.

4. Why would a PVC stay Pending?

   Model answer: No matching PV, wrong class, unsupported access mode, or WaitForFirstConsumer with no pod yet.

5. How do you recover data from a Released PV?

   Model answer: Back it up, remove claimRef, and bind a new PVC with volumeName.

Samajla ka?