Chapter 61: Kubernetes Storage in Depth: Volumes, PV, PVC, StorageClass, CSI and Snapshots

61.2 emptyDir in depth: scratch space, shared folders, memory-backed volumes and size limits

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Swiggy runs a menu-image service. For every restaurant menu upload, the pod unzips about 400 photos, resizes them, and pushes the results to S3. The unzipped photos are only needed for a few minutes. Writing them to an EBS disk would be slow and costly, and writing them into the container's own layer would be lost on a restart in the middle of a batch. This is the job emptyDir was made for. An emptyDir is an empty folder that Kubernetes creates on the node when the pod starts. Every container in the pod can mount it. It survives container crashes and restarts, and it is deleted when the pod leaves the node. So it is perfect for scratch space, caches and files that two containers in one pod must share. emptyDir has two extra knobs that matter in production. medium: Memory puts the folder in RAM (tmpfs), which is very fast but counts against the container memory limit. sizeLimit caps how much the volume may hold; if a pod writes more, kubelet evicts that pod so one noisy pod cannot fill the node's disk. In this lesson we use all of these on the ch61 kind cluster: a shared scratch folder, a RAM-backed cache, a pod that gets evicted for writing too much, and the ephemeral-storage requests that tell the scheduler how much local disk a pod needs.

What: every term in plain English

• emptyDir — An empty folder created when the pod is scheduled to a node and removed when the pod leaves.
• medium: "" — The default. The folder lives on the node's disk under /var/lib/kubelet/pods.
• medium: Memory — The folder is a tmpfs in RAM. Fast, and its size counts as container memory.
• sizeLimit — The most data the emptyDir may hold. Going over gets the pod evicted.
• ephemeral-storage — A resource like cpu and memory, for local disk used by logs, writable layers and emptyDir.
• Eviction — kubelet stops a pod and marks it Failed with reason Evicted to protect the node.
• tmpfs — A Linux filesystem that keeps files in memory.
• Scratch space — Temporary working files that can be rebuilt if lost.
• Pod-level sharing — All containers in one pod can mount the same emptyDir; other pods cannot.
• Node pressure — When a node runs low on disk, kubelet evicts pods using the most ephemeral storage.

How: step by step

1. Start from the ch61 cluster and namespace storage. Recreate the namespace if 61.1 cleaned it up.
2. Run a two-container pod sharing an emptyDir. One writes, one reads.
3. Switch the volume to medium: Memory. Check with df that it is tmpfs.
4. Fill the RAM-backed volume past the memory limit. The container is OOMKilled.
5. Add sizeLimit to a disk-backed emptyDir and overfill it. The pod is evicted.
6. Add ephemeral-storage requests and limits. The scheduler and kubelet both use them.
7. Clean up the pods. Keep the cluster for 61.3.

Example 1: Share a scratch folder between an unzip container and a resize container

Goal. Share a scratch folder between an unzip container and a resize container.

menu-scratch.yaml

apiVersion: v1
kind: Pod
metadata:
  name: menu-images
  namespace: storage
  labels:
    app: menu-images
spec:
  volumes:
    - name: scratch
      emptyDir: {}
  initContainers:
    - name: unzip
      image: busybox:1.36
      command: ["sh", "-c", "for i in $(seq 1 400); do echo photo-$i > /work/photo-$i.jpg; done; echo unzipped"]
      volumeMounts:
        - name: scratch
          mountPath: /work
  containers:
    - name: resize
      image: busybox:1.36
      command: ["sh", "-c", "ls /work | wc -l; sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /work
      resources:
        requests: {cpu: 20m, memory: 32Mi}
        limits: {memory: 64Mi}

What each line does

• initContainers.unzip — Runs first and writes 400 fake photos into the shared folder.
• emptyDir: {} — Disk-backed scratch space on the node.
• Same mountPath /work — Both containers see the same files; the path could also differ.
• resize — Starts only after unzip succeeds and finds all 400 files.
• Restart-safe — If resize crashes, the 400 files are still there for the next try.

Green output

$ kubectl create namespace storage
namespace/storage created
$ kubectl apply -f menu-scratch.yaml
pod/menu-images created
$ kubectl logs -n storage menu-images -c unzip
unzipped
$ kubectl logs -n storage menu-images -c resize
400
$ kubectl exec -n storage menu-images -c resize -- kill 1
$ kubectl get pod -n storage menu-images
NAME          READY   STATUS    RESTARTS   AGE
menu-images   1/1     Running   1          48s
$ kubectl logs -n storage menu-images -c resize
400

Expected result. The init container's files were still there after the resize container restarted, so a crash mid-batch does not force a fresh unzip.

Example 2: Use a RAM-backed emptyDir for a hot cache and see it count against memory

Goal. Use a RAM-backed emptyDir for a hot cache and see it count against memory.

ram-cache.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ram-cache
  namespace: storage
spec:
  volumes:
    - name: cache
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: cache
          mountPath: /cache
      resources:
        requests: {cpu: 20m, memory: 32Mi}
        limits: {memory: 48Mi}

What each line does

• medium: Memory — The folder is tmpfs; reads and writes go to RAM.
• sizeLimit: 64Mi — Sets the tmpfs size and the eviction limit.
• limits.memory: 48Mi — Files in the tmpfs are charged to this container's memory.
• 40 MB write — Fits under the sizeLimit but, with the shell, goes over 48Mi.
• Lesson — Size memory limits to include the cache, or the container is OOMKilled.

Green output

$ kubectl apply -f ram-cache.yaml
$ kubectl exec -n storage ram-cache -- df -h /cache
Filesystem      Size  Used Available Use% Mounted on
tmpfs          64.0M     0     64.0M   0% /cache
$ kubectl exec -n storage ram-cache -- dd if=/dev/zero of=/cache/blob bs=1M count=20
20+0 records in
20+0 records out
$ kubectl exec -n storage ram-cache -- dd if=/dev/zero of=/cache/blob2 bs=1M count=40
command terminated with exit code 137
$ kubectl get pod -n storage ram-cache
NAME        READY   STATUS      RESTARTS   AGE
ram-cache   0/1     OOMKilled   1          1m

Expected result. df shows tmpfs, so the data is in RAM. 20 MB was fine; 60 MB in total went over the 48Mi memory limit and the container was OOMKilled with exit code 137.

Example 3: Protect the node with sizeLimit and ephemeral-storage limits

Goal. Protect the node with sizeLimit and ephemeral-storage limits.

disk-hog.yaml

apiVersion: v1
kind: Pod
metadata:
  name: disk-hog
  namespace: storage
spec:
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 50Mi
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 5; dd if=/dev/zero of=/tmp/big bs=1M count=120; sleep 3600"]
      volumeMounts:
        - name: tmp
          mountPath: /tmp
      resources:
        requests:
          cpu: 20m
          memory: 32Mi
          ephemeral-storage: 100Mi
        limits:
          memory: 64Mi
          ephemeral-storage: 200Mi

What each line does

• sizeLimit: 50Mi — The emptyDir may hold 50 MiB.
• dd 120 MB — Writes much more than allowed.
• requests.ephemeral-storage — The scheduler only picks nodes with 100Mi free local disk.
• limits.ephemeral-storage — Total local disk for logs, writable layer and emptyDir.
• Eviction check — kubelet checks usage every few seconds, so the eviction is not instant.

Green output

$ kubectl apply -f disk-hog.yaml
$ sleep 40; kubectl get pod -n storage disk-hog
NAME       READY   STATUS    RESTARTS   AGE
disk-hog   0/1     Error     0          45s
$ kubectl describe pod -n storage disk-hog | grep -A2 -i reason
Status:   Failed
Reason:   Evicted
Message:  Usage of EmptyDir volume "tmp" exceeds the limit "50Mi".
$ kubectl get events -n storage --field-selector reason=Evicted
LAST SEEN   TYPE      REASON    OBJECT         MESSAGE
30s         Warning   Evicted   pod/disk-hog   Usage of EmptyDir volume "tmp" exceeds the limit "50Mi".

Expected result. kubelet evicted the pod instead of letting it fill the node disk. Evicted pods are not restarted in place; a Deployment would create a new pod.

Every command, step by step (kind lab with kubectl)

1. Make sure the namespace exists

   kubectl get ns storage || kubectl create namespace storage
   

   What it does: 61.1 deleted it during cleanup.

   namespace/storage created
   

2. Run the shared scratch pod

   kubectl apply -f menu-scratch.yaml
   kubectl get pod -n storage menu-images -w
   

   What it does: Init:0/1, then PodInitializing, then Running.

   menu-images   0/1   Init:0/1          0   2s
   menu-images   0/1   PodInitializing   0   5s
   menu-images   1/1   Running           0   7s
   

3. Count the shared files

   kubectl exec -n storage menu-images -c resize -- sh -c 'ls /work | wc -l'
   

   What it does: The resize container sees what unzip wrote.

   400
   

4. Find the folder on the node

   kubectl get pod -n storage menu-images -o wide
   kubectl get pod -n storage menu-images -o jsonpath='{.metadata.uid}'
   

   What it does: Note the node and the uid.

   menu-images   1/1   Running   0   1m   10.244.1.7   ch61-worker
   3c9e2f4a-8b1d-4e7a-a0c6-5d2f7b9e1a44
   

5. Look inside kubelet's pod folder

   docker exec ch61-worker sh -c 'ls /var/lib/kubelet/pods/3c9e2f4a-8b1d-4e7a-a0c6-5d2f7b9e1a44/volumes/kubernetes.io~empty-dir/scratch | head -3'
   

   What it does: Replace the uid with yours. This is the real folder.

   photo-1.jpg
   photo-10.jpg
   photo-100.jpg
   

6. Delete the pod and check the folder

   kubectl delete pod -n storage menu-images
   docker exec ch61-worker ls /var/lib/kubelet/pods/3c9e2f4a-8b1d-4e7a-a0c6-5d2f7b9e1a44
   

   What it does: The folder is removed with the pod.

   ls: cannot access '/var/lib/kubelet/pods/3c9e2f4a-...': No such file or directory
   

7. Run the RAM cache

   kubectl apply -f ram-cache.yaml
   kubectl exec -n storage ram-cache -- df -h /cache
   

   What it does: tmpfs, sized by sizeLimit.

   tmpfs   64.0M   0   64.0M   0% /cache
   

8. Watch memory go up

   kubectl exec -n storage ram-cache -- dd if=/dev/zero of=/cache/blob bs=1M count=20
   kubectl top pod -n storage ram-cache
   

   What it does: Needs metrics-server from 59.3. Memory jumps by about 20Mi.

   NAME        CPU(cores)   MEMORY(bytes)
   ram-cache   1m           21Mi
   

9. Trigger the OOM kill

   kubectl exec -n storage ram-cache -- dd if=/dev/zero of=/cache/blob2 bs=1M count=40
   kubectl get pod -n storage ram-cache -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
   

   What it does: The container went over limits.memory.

   OOMKilled
   

10. Run the disk hog

    kubectl apply -f disk-hog.yaml
    sleep 40
    kubectl get pod -n storage disk-hog -o jsonpath='{.status.reason}{"\n"}{.status.message}'
    

    What it does: Evicted for the sizeLimit.

    Evicted
    Usage of EmptyDir volume "tmp" exceeds the limit "50Mi".
    

11. See ephemeral-storage on the node

    kubectl describe node ch61-worker | grep -A6 'Allocated resources'
    

    What it does: Requests are counted like cpu and memory.

    Resource           Requests     Limits
    cpu                170m (2%)    0 (0%)
    memory             96Mi (1%)    128Mi (1%)
    ephemeral-storage  100Mi (0%)   200Mi (0%)
    

12. Clean up the pods

    kubectl delete pod -n storage ram-cache disk-hog --ignore-not-found
    

    What it does: Keep the namespace and cluster for 61.3.

    pod "ram-cache" deleted
    pod "disk-hog" deleted
    

When it is red 1

Status: Failed  Reason: Evicted  Message: Usage of EmptyDir volume "tmp" exceeds the limit "50Mi".

Cause. The app wrote more than sizeLimit.

Fix, step by step

1. Raise sizeLimit if the need is real.
2. Clean temp files in the app after each batch.
3. Use a PVC for large data.

When it is red 2

Last State: Terminated  Reason: OOMKilled  Exit Code: 137

Cause. Files in a medium: Memory volume count as container memory.

Fix, step by step

1. Raise limits.memory to cover app plus cache.
2. Lower sizeLimit so the cache cannot grow past the budget.

When it is red 3

0/3 nodes are available: 3 Insufficient ephemeral-storage.

Cause. requests.ephemeral-storage is larger than any node has free.

Fix, step by step

1. Lower the request.
2. Use nodes with bigger root disks.

When it is red 4

The node was low on resource: ephemeral-storage.

Cause. Node disk pressure; kubelet evicted the biggest users.

Fix, step by step

1. Set sizeLimit and ephemeral-storage limits on every pod.
2. Rotate container logs.

Practice tasks

kubectl exec -n storage menu-images -c resize -- sh -c 'ls /work | wc -l'

kubectl exec -n storage menu-images -c resize -- kill 1

kubectl exec -n storage ram-cache -- df -h /cache

docker exec ch61-worker ls /var/lib/kubelet/pods

kubectl get pod -n storage ram-cache

kubectl get events -n storage --field-selector reason=Evicted

kubectl describe pod -n storage disk-hog

kubectl apply -f ram-cache.yaml

kubectl get pod -n storage -o wide

kubectl explain pod.spec.volumes.emptyDir

Recap

• emptyDir is scratch space that lives as long as the pod on its node.
• All containers in a pod can share it; it survives container restarts.
• medium: Memory is fast tmpfs but uses the container's memory limit.
• sizeLimit and ephemeral-storage limits stop one pod from filling the node.
• Next: hostPath, the node's own folders, and why it is risky.

FAQ

Q. Is emptyDir faster than a PVC?

Usually yes, because it is local disk or RAM with no network.

Q. Does a Deployment rollout keep emptyDir data?

No. New pods get new empty folders.

Q. What if the node reboots?

Disk emptyDir may stay while the pod is the same, but tmpfs is lost. Never keep important data there.

Q. Can I set sizeLimit without ephemeral-storage limits?

Yes. sizeLimit applies to that volume; ephemeral-storage limits cover everything local.

Q. Is there a generic ephemeral volume?

Yes. spec.volumes.ephemeral creates a PVC per pod from a StorageClass and deletes it with the pod; useful for large scratch on EBS.

Q. Does Netflix-style caching use emptyDir?

Many services use RAM emptyDir for small hot caches and keep big caches outside the pod.

Interview questions

1. What is emptyDir and when do you use it?

   Model answer: An empty pod-scoped folder for scratch data and sharing between containers.

2. What is the risk of medium: Memory?

   Model answer: It uses RAM charged to the container, so it can cause OOMKilled.

3. How do you stop a pod filling a node's disk?

   Model answer: sizeLimit on emptyDir and ephemeral-storage requests and limits.

4. What does an Evicted pod mean?

   Model answer: kubelet removed it for node pressure or limit breach; a controller must recreate it.

5. emptyDir vs generic ephemeral volume?

   Model answer: emptyDir uses node disk or RAM; a generic ephemeral volume is a per-pod PVC on real storage.

Samajla ka?