Chapter 61: Kubernetes Storage in Depth: Volumes, PV, PVC, StorageClass, CSI and Snapshots

61.7 Expanding volumes, VolumeSnapshots and backups with Velero

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose PhonePe's transaction-history database runs in a StatefulSet with a 50Gi EBS volume. On the evening before a big festival sale, the disk reaches 92%. The team cannot take the database down to copy data to a bigger disk. And if a bad migration deletes a table, they need to get yesterday's data back within minutes. Kubernetes has three tools for this. Volume expansion lets you grow a PVC by editing its size, if its StorageClass has allowVolumeExpansion: true; the CSI driver grows the disk and the filesystem, usually while the pod keeps running. VolumeSnapshots take a point-in-time copy of a PVC through the CSI driver (an EBS snapshot on AWS), and you can create a new PVC from a snapshot. Backup tools like Velero save both the Kubernetes objects and the volume data, so a whole namespace can be restored, even into another cluster. Two facts to remember: you can grow a PVC but never shrink it, and a snapshot is not a backup until it is stored safely outside the failure you fear (another region or account). In this lesson we install the CSI hostpath driver on kind so snapshots work locally, grow a volume, snapshot and restore it, and then walk through a Velero backup and restore of a namespace to S3.

What: every term in plain English

• allowVolumeExpansion — StorageClass flag that allows bigger PVC sizes.
• Online expansion — Growing a mounted volume without stopping the pod.
• FileSystemResizePending — PVC condition while the filesystem waits to be grown.
• VolumeSnapshot — Namespaced request for a point-in-time copy of a PVC.
• VolumeSnapshotContent — The real snapshot, like a PV is to a PVC.
• VolumeSnapshotClass — Like a StorageClass, but for snapshots; names the driver and deletionPolicy.
• dataSource — PVC field that creates a new volume from a snapshot or another PVC.
• Snapshot controller — Cluster add-on that watches snapshot objects; not installed by default.
• Velero — Open-source tool by VMware (Broadcom) for backing up cluster objects and volumes to object storage.
• RPO / RTO — How much data you can lose, and how long restore may take.

How: step by step

1. Install the snapshot CRDs, controller and CSI hostpath driver on ch61. kind's local-path does not support snapshots.
2. Create a PVC on the csi-hostpath-sc class and write data. Our demo database.
3. Grow the PVC from 1Gi to 2Gi. Watch the conditions.
4. Take a VolumeSnapshot. Wait for readyToUse.
5. Delete data, then restore into a new PVC. Mount it and read the old data.
6. Walk through Velero to S3. Backup and restore a namespace.
7. Delete the cluster. Chapter 61 is done.

Example 1: Grow a PVC while the pod keeps running

Goal. Grow a PVC while the pod keeps running.

txn-db.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: txn-data
  namespace: storage
spec:
  storageClassName: csi-hostpath-sc
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: txn-db
  namespace: storage
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: txn-data
  containers:
    - name: db
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}

What each line does

• csi-hostpath-sc — The demo CSI driver's class; it supports expansion and snapshots.
• 1Gi to 2Gi — Change spec.resources.requests.storage with kubectl patch.
• Conditions — Resizing, then FileSystemResizePending, then gone.
• No restart — The pod stays Running with the same restart count.
• Shrink — Setting a smaller size is rejected.

Green output

$ kubectl apply -f txn-db.yaml
$ kubectl exec -n storage txn-db -- sh -c 'seq 1 5000 | sed "s/^/txn-/" > /data/history.txt; wc -l /data/history.txt'
5000 /data/history.txt
$ kubectl patch pvc txn-data -n storage -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
persistentvolumeclaim/txn-data patched
$ kubectl get pvc -n storage txn-data -w
NAME       STATUS   VOLUME          CAPACITY
txn-data   Bound    pvc-4d1e...     1Gi
txn-data   Bound    pvc-4d1e...     2Gi
$ kubectl patch pvc txn-data -n storage -p '{"spec":{"resources":{"requests":{"storage":"1Gi"}}}}'
The PersistentVolumeClaim "txn-data" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than status.capacity

Expected result. The capacity changed from 1Gi to 2Gi with the pod still Running. Shrinking is forbidden, so plan growth carefully.

Example 2: Snapshot the volume, lose data, and restore into a new PVC

Goal. Snapshot the volume, lose data, and restore into a new PVC.

snapshot.yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-hostpath-snapclass
driver: hostpath.csi.k8s.io
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: txn-before-sale
  namespace: storage
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: txn-data

restore.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: txn-restored
  namespace: storage
spec:
  storageClassName: csi-hostpath-sc
  dataSource:
    name: txn-before-sale
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi

What each line does

• VolumeSnapshotClass — Names the CSI driver; on EKS this would be ebs.csi.aws.com.
• source.persistentVolumeClaimName — Which PVC to copy.
• readyToUse — Wait for true before relying on it.
• dataSource — The new PVC starts with the snapshot's data.
• Size — Must be at least the snapshot's restore size.

Green output

$ kubectl apply -f snapshot.yaml
$ kubectl get volumesnapshot -n storage
NAME              READYTOUSE   SOURCEPVC   RESTORESIZE   SNAPSHOTCLASS
txn-before-sale   true         txn-data    2Gi           csi-hostpath-snapclass
$ kubectl exec -n storage txn-db -- rm /data/history.txt
$ kubectl apply -f restore.yaml
$ kubectl run reader -n storage --image=busybox:1.36 --overrides='{"spec":{"volumes":[{"name":"r","persistentVolumeClaim":{"claimName":"txn-restored"}}],"containers":[{"name":"reader","image":"busybox:1.36","command":["sleep","3600"],"volumeMounts":[{"name":"r","mountPath":"/r"}]}]}}'
$ kubectl exec -n storage reader -- wc -l /r/history.txt
5000 /r/history.txt

Expected result. The file was deleted from the live volume, but the restored PVC has all 5000 lines from the snapshot. On EKS the same YAML creates and restores EBS snapshots.

Example 3: Back up and restore a whole namespace with Velero to S3

Goal. Back up and restore a whole namespace with Velero to S3.

velero-install.sh

aws s3api create-bucket --bucket phonepe-demo-velero-backups \
  --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket phonepe-demo-velero-backups \
  --backup-location-config region=ap-south-1 \
  --snapshot-location-config region=ap-south-1 \
  --service-account-name velero \
  --no-secret

What each line does

• S3 bucket — Backups of objects and metadata go here. Use a bucket in another account for real safety.
• --no-secret — Velero uses an IAM role (IRSA or Pod Identity), not access keys.
• velero backup create — Saves every object in the namespace and snapshots its volumes.
• velero restore create — Recreates the objects and PVCs from the backup.
• Schedules — velero schedule create makes daily backups with a TTL.

Green output

$ bash velero-install.sh
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.
$ velero backup create storage-ns-1 --include-namespaces storage --wait
Backup completed with status: Completed.
$ kubectl delete namespace storage
$ velero restore create --from-backup storage-ns-1 --wait
Restore completed with status: Completed. Errors: 0, Warnings: 0
$ kubectl get pvc,pod -n storage
NAME                                 STATUS   CAPACITY
persistentvolumeclaim/txn-data       Bound    2Gi
NAME         READY   STATUS
pod/txn-db   1/1     Running
$ velero schedule create daily-storage --schedule "0 1 * * *" --include-namespaces storage --ttl 720h

Expected result. The whole namespace, including its disk, came back after a full delete. The daily schedule keeps 30 days of backups.

Every command, step by step (kind lab with kubectl)

1. Install snapshot CRDs and controller

   kubectl apply -k ./external-snapshotter/client/config/crd
   kubectl apply -k ./external-snapshotter/deploy/kubernetes/snapshot-controller
   

   What it does: Use a local copy of the kubernetes-csi external-snapshotter release (v8). Download the release archive from the Kubernetes CSI project page and unpack it first.

   customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
   deployment.apps/snapshot-controller created
   

2. Install the CSI hostpath driver

   cd csi-driver-host-path && ./deploy/kubernetes-latest/deploy.sh
   kubectl get pods -l app.kubernetes.io/name=csi-hostpathplugin
   

   What it does: A demo CSI driver from the same project, for labs only.

   csi-hostpathplugin-0   8/8   Running
   

3. Create its StorageClass

   printf '%s\n' 'apiVersion: storage.k8s.io/v1' 'kind: StorageClass' 'metadata:' '  name: csi-hostpath-sc' 'provisioner: hostpath.csi.k8s.io' 'allowVolumeExpansion: true' 'volumeBindingMode: WaitForFirstConsumer' > csi-sc.yaml
   kubectl apply -f csi-sc.yaml
   

   What it does: Expansion turned on.

   storageclass.storage.k8s.io/csi-hostpath-sc created
   

4. Create the database

   kubectl create ns storage --dry-run=client -o yaml | kubectl apply -f -
   kubectl apply -f txn-db.yaml
   kubectl get pvc -n storage txn-data
   

   What it does: Bound at 1Gi.

   txn-data   Bound   pvc-4d1e...   1Gi   RWO   csi-hostpath-sc
   

5. Write history

   kubectl exec -n storage txn-db -- sh -c 'seq 1 5000 | sed "s/^/txn-/" > /data/history.txt; wc -l /data/history.txt'
   

   What it does: Test data.

   5000 /data/history.txt
   

6. Grow the disk

   kubectl patch pvc txn-data -n storage -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
   kubectl get pvc -n storage txn-data -o jsonpath='{.status.capacity.storage}'
   

   What it does: Online expansion.

   2Gi
   

7. Check inside the pod

   kubectl exec -n storage txn-db -- df -h /data
   kubectl get pod -n storage txn-db -o jsonpath='{.status.containerStatuses[0].restartCount}'
   

   What it does: Bigger filesystem, zero restarts.

   /dev/...   2.0G   20.0K   1.9G   0% /data
   0
   

8. Take a snapshot

   kubectl apply -f snapshot.yaml
   kubectl wait volumesnapshot/txn-before-sale -n storage --for=jsonpath='{.status.readyToUse}'=true --timeout=60s
   

   What it does: Ready.

   volumesnapshot.snapshot.storage.k8s.io/txn-before-sale condition met
   

9. See the content object

   kubectl get volumesnapshotcontent
   

   What it does: Cluster-scoped, like a PV.

   NAME                READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER
   snapcontent-7a2...  true         2147483648    Delete           hostpath.csi.k8s.io
   

10. Lose data and restore

    kubectl exec -n storage txn-db -- rm /data/history.txt
    kubectl apply -f restore.yaml
    kubectl get pvc -n storage txn-restored
    

    What it does: A new PVC from the snapshot.

    txn-restored   Pending   ...   waiting for first consumer
    (Bound after the reader pod starts)
    

11. Read restored data

    kubectl exec -n storage reader -- tail -1 /r/history.txt
    

    What it does: Back.

    txn-5000
    

12. Delete the cluster

    kind delete cluster --name ch61
    

    What it does: Chapter 61 done. Chapter 62 uses a new cluster.

    Deleting cluster "ch61" ...
    

When it is red 1

Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller

Cause. The StorageClass or driver does not support expansion.

Fix, step by step

1. Set allowVolumeExpansion: true and use a CSI driver that supports it.

When it is red 2

field can not be less than status.capacity

Cause. You tried to shrink.

Fix, step by step

1. Create a smaller new PVC and copy data instead.

When it is red 3

no matches for kind "VolumeSnapshot" in version "snapshot.storage.k8s.io/v1"

Cause. Snapshot CRDs are not installed.

Fix, step by step

1. Install the external-snapshotter CRDs and controller. On EKS, the EBS CSI add-on docs list the snapshot controller add-on.

When it is red 4

You can modify an EBS volume only once every 6 hours

Cause. AWS limit on EBS modifications.

Fix, step by step

1. Grow generously once, and alert at 70-80% usage.

Practice tasks

kubectl get pods -A | grep -E 'snapshot|hostpath'

kubectl get pvc -n storage

kubectl get pvc -n storage txn-data

kubectl patch pvc txn-data -n storage -p '{"spec":{"resources":{"requests":{"storage":"1Gi"}}}}'

kubectl get volumesnapshot -n storage

kubectl exec -n storage reader -- wc -l /r/history.txt

kubectl get pvc -n storage

velero schedule get

kubectl get volumesnapshot -n storage

kubectl explain volumesnapshot.spec

Recap

• Grow PVCs by editing their size when the class allows expansion; never shrink.
• VolumeSnapshots copy a PVC at a point in time through the CSI driver.
• dataSource creates a new PVC from a snapshot or another PVC.
• Velero backs up objects and volumes to S3 and can restore whole namespaces.
• Chapter 62 moves on to Services, networking and load balancing.

FAQ

Q. Does expansion need a pod restart?

Not with modern CSI drivers that support online expansion, like EBS CSI.

Q. Are EBS snapshots incremental?

Yes. Only changed blocks are stored after the first one.

Q. Can I snapshot a running database?

Yes, but it is crash-consistent. Freeze or use the database's own backup tool for app consistency.

Q. Is AWS Backup an alternative?

Yes, AWS Backup supports EKS and EBS volumes with policies.

Q. Does Velero need snapshots?

It can use CSI snapshots or file-level backup with Kopia.

Q. Where should backups live?

In another region or account, so one mistake cannot delete both.

Interview questions

1. How do you increase a PVC's size?

   Model answer: Edit spec.resources.requests.storage when the class has allowVolumeExpansion.

2. Explain VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass.

   Model answer: Request, real snapshot, and driver template; like PVC, PV and StorageClass.

3. How do you restore a snapshot?

   Model answer: Create a PVC with dataSource pointing to the VolumeSnapshot.

4. How do you back up a Kubernetes namespace?

   Model answer: Velero to S3, with CSI snapshots or Kopia file backup, on a schedule.

5. Snapshot vs backup?

   Model answer: A snapshot can die with the account or region; a backup is stored independently with retention.

Samajla ka?