Chapter 18: Kubernetes Workloads

18.14 Safety basics

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Copy a random YAML from a blog and you might run a container as root, with privileged: true, no CPU or memory limits, in the default namespace. It works on your laptop. In a shared cluster at a company like Infosys it is a risk. A privileged container can control the whole node. A container with no memory limit can eat the node and get other teams' Pods killed. Everything in default means anyone can delete anything. Kubernetes gives you simple guard rails. securityContext to run as a normal user, drop Linux capabilities and make the root file system read only. resources so every Pod has requests and limits. Namespaces with a ResourceQuota and LimitRange so one team cannot take all the room. Pod Security Admission labels so the cluster refuses dangerous Pods. In this lesson you will try a dangerous Pod, see the cluster reject it, and fix it step by step.

What we will build

You will create a team-a namespace with the restricted Pod Security level, a ResourceQuota and a LimitRange. Then you will apply an unsafe Pod, read the rejection, and fix the YAML until it is accepted.

What: every term in plain English

• securityContext — Pod or container settings for user, group, capabilities and file system.
• runAsNonRoot — Refuse to start if the image runs as root.
• privileged — Gives the container almost full control of the node. Avoid.
• Capabilities — Small pieces of root power, like NET_ADMIN. Drop ALL by default.
• readOnlyRootFilesystem — The container cannot write to its image files.
• Pod Security Admission — Built-in checks with three levels: privileged, baseline, restricted.
• ResourceQuota — Total CPU, memory and object limits for a namespace.
• LimitRange — Default and maximum requests and limits for each container in a namespace.

How: step by step

1. Start a lab cluster

   kind create cluster --name lab
   kubectl get nodes
   

   What it does: Creates a one-node Kubernetes cluster inside Docker. On minikube use: minikube start.

   Output

   Creating cluster "lab" ...
    ✓ Ensuring node image (kindest/node:v1.30.0)
    ✓ Preparing nodes
   Set kubectl context to "kind-lab"
   NAME                STATUS   ROLES           AGE   VERSION
   lab-control-plane   Ready    control-plane   40s   v1.30.0
   

2. Create a team namespace

   kubectl create namespace team-a
   kubectl label namespace team-a pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/warn=restricted
   

   What it does: The cluster now refuses unsafe Pods in team-a.

   Output

   namespace/team-a created
   namespace/team-a labeled
   

3. Add a quota

   kubectl -n team-a create quota team-a --hard=requests.cpu=2,requests.memory=2Gi,limits.memory=4Gi,pods=20
   

   What it does: The team cannot use more than this in total.

   Output

   resourcequota/team-a created
   

4. Add default limits

   kubectl -n team-a apply -f limits.yaml
   

   What it does: The file is in Example 3. Pods without resources get defaults.

   Output

   limitrange/defaults created
   

5. Try a privileged Pod

   kubectl -n team-a run bad --image=nginx:1.27 --overrides='{"spec":{"containers":[{"name":"bad","image":"nginx:1.27","securityContext":{"privileged":true}}]}}'
   

   What it does: The API server rejects it.

   Output

   Error from server (Forbidden): pods "bad" is forbidden: violates PodSecurity "restricted:latest": privileged (container "bad" must not set securityContext.privileged=true), allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
   

6. See the quota use

   kubectl -n team-a describe quota team-a
   

   What it does: Shows used versus hard limits.

   Output

   Resource         Used  Hard
   --------         ----  ----
   pods             0     20
   requests.cpu     0     2
   requests.memory  0     2Gi
   

Example 1: The unsafe Pod and every reason it is rejected

Stack. Pod Security Admission, restricted level

Goal. The unsafe Pod and every reason it is rejected.

unsafe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: unsafe
  namespace: team-a
spec:
  hostNetwork: true
  containers:
  - name: app
    image: nginx:1.27
    securityContext:
      privileged: true
    volumeMounts:
    - {name: host, mountPath: /host}
  volumes:
  - name: host
    hostPath: {path: /}

terminal

kubectl apply -f unsafe.yaml

What each line does

• hostNetwork: true — Shares the node network. Can sniff traffic.
• privileged: true — Full access to node devices.
• hostPath / — Mounts the node root disk. Could read every secret on the node.
• No securityContext user — nginx image runs as root by default.

Green output

Error from server (Forbidden): error when creating "unsafe.yaml": pods "unsafe" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true), privileged (container "app" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "app" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "app" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "app" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "app" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Expected result. The error is a checklist. Fix each item in order.

Example 2: The fixed, restricted-safe Pod

Stack. Pod Security restricted, nginx-unprivileged 1.27

Goal. The fixed, restricted-safe Pod.

safe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: safe
  namespace: team-a
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 101
    runAsGroup: 101
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: app
    image: nginxinc/nginx-unprivileged:1.27
    ports: [{containerPort: 8080}]
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: {drop: ["ALL"]}
    resources:
      requests: {cpu: 50m, memory: 64Mi}
      limits: {memory: 128Mi}
    volumeMounts:
    - {name: tmp, mountPath: /tmp}
  volumes:
  - name: tmp
    emptyDir: {}

terminal

kubectl apply -f safe.yaml
kubectl -n team-a wait --for=condition=Ready pod/safe
kubectl -n team-a exec safe -- id
kubectl -n team-a exec safe -- touch /etc/test

What each line does

• runAsNonRoot and runAsUser 101 — Runs as a normal user.
• seccompProfile RuntimeDefault — Blocks rare, risky system calls.
• nginx-unprivileged — An image built to run as non-root on port 8080.
• allowPrivilegeEscalation: false — No sudo-style tricks.
• readOnlyRootFilesystem — Image files cannot be changed.
• drop ALL — No extra root powers.
• emptyDir /tmp — A writable scratch folder nginx needs.

Green output

pod/safe created
pod/safe condition met
uid=101(nginx) gid=101(nginx) groups=101(nginx)
touch: /etc/test: Read-only file system
command terminated with exit code 1

Expected result. The app runs fine, as a normal user, and even an attacker inside cannot change files.

Example 3: LimitRange defaults and quota in action

Stack. ResourceQuota, LimitRange

Goal. LimitRange defaults and quota in action.

limits.yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
  - type: Container
    defaultRequest: {cpu: 100m, memory: 128Mi}
    default: {memory: 256Mi}
    max: {cpu: "1", memory: 1Gi}

terminal

kubectl -n team-a create deployment hog --image=nginxinc/nginx-unprivileged:1.27 --replicas=30 --dry-run=client -o yaml > hog.yaml
kubectl -n team-a apply -f hog.yaml 2>&1 | tail -1
kubectl -n team-a get deploy hog
kubectl -n team-a get events --field-selector reason=FailedCreate | tail -1

What each line does

• defaultRequest — Requests added to containers that have none.
• default — Limit added to containers that have none.
• max — No container may ask for more.
• replicas=30 — Too many for the quota pods=20 and requests.memory=2Gi.
• FailedCreate — The ReplicaSet cannot make more Pods.

Green output

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
NAME   READY   UP-TO-DATE   AVAILABLE
hog    0/30    0            0
Warning  FailedCreate  replicaset/hog-5c8d7f9b6  Error creating: pods "hog-5c8d7f9b6-xk2m4" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false

Expected result. Two guard rails fired: Pod Security blocks the unsafe template, and even a safe template would stop at the quota with exceeded quota: team-a, requested: requests.memory.

Verify: did it work?

1. Check one Pod got defaults

   kubectl -n team-a get pod safe -o jsonpath='{.spec.containers[0].resources}{"\n"}'
   

   What it does: Shows the requests and limits used.

   Output

   {"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}
   

2. Audit a namespace

   kubectl label --dry-run=server --overwrite ns default pod-security.kubernetes.io/enforce=restricted
   

   What it does: Shows which running Pods would break, without changing anything.

   Output

   Warning: existing pods in namespace "default" violate the new PodSecurity enforce level "restricted:latest"
   namespace/default labeled (server dry run)
   

When it is red 1

container has runAsNonRoot and image will run as root

Cause. The image user is root but the Pod says runAsNonRoot.

Fix, step by step

1. Use a non-root image like nginx-unprivileged.
2. Or set runAsUser to a non-zero id the image supports.

When it is red 2

nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

Cause. readOnlyRootFilesystem is on and the app writes to a folder.

Fix, step by step

1. Mount an emptyDir at that folder.
2. Repeat for each path in the error.

When it is red 3

exceeded quota: team-a, requested: requests.memory=128Mi, used: requests.memory=2Gi, limited: requests.memory=2Gi

Cause. The namespace is full.

Fix, step by step

1. Lower requests, delete unused Deployments, or ask for a bigger quota.

Cleanup

1. Delete the team namespace

   kubectl delete namespace team-a
   

   What it does: Removes the Pods, quota and LimitRange together.

   Output

   namespace "team-a" deleted
   

2. Delete the lab cluster

   kind delete cluster --name lab
   

   What it does: Removes the whole cluster container. On minikube use: minikube delete.

   Output

   Deleting cluster "lab" ...
   Deleted nodes: ["lab-control-plane"]
   

Practice tasks

Homework: try at home

Recap

• Never use privileged, hostPath or hostNetwork for apps.
• Run as non-root, drop ALL, read-only root file system.
• Every container needs requests and a memory limit.
• Use a namespace per team with quota and LimitRange.
• Pod Security labels make the cluster enforce it.

FAQ

Is PodSecurityPolicy still used?

No. It was removed in 1.25. Pod Security Admission or Kyverno and OPA Gatekeeper replace it.

My image needs root. What now?

Find or build a non-root image. Most official images have one.

Should I set CPU limits?

Teams differ. CPU limits can throttle. Always set CPU requests and memory limits.

Is a namespace a security wall?

Only partly. Add RBAC, NetworkPolicies and quotas.

Where do I store passwords?

In Secrets with RBAC and encryption at rest, or an external store like AWS Secrets Manager.

Interview questions

1. How do you secure a Pod spec?

   Non-root user, no privilege escalation, drop ALL capabilities, read-only root file system, seccomp RuntimeDefault, resources set, no host namespaces or hostPath.

2. What replaced PodSecurityPolicy?

   Pod Security Admission with privileged, baseline and restricted levels, often plus Kyverno or Gatekeeper.

3. ResourceQuota versus LimitRange?

   Quota caps a namespace total; LimitRange sets per-container defaults and limits.

4. Why is privileged dangerous?

   It gives the container near-root access to the host, so a break-in becomes a node takeover.

5. How do you roll out Pod Security safely?

   Start with warn and audit labels, fix workloads, then enforce.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards