Chapter 60: Kubernetes Workload Controllers in Depth: ReplicaSet, Deployment, StatefulSet, DaemonSet, Jobs and CronJobs

60.6 Jobs: completions, parallelism, backoffLimit, indexed jobs and TTL

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Nykaa must resize 30,000 product photos after a website redesign, or run a database migration before a release. If you run this work in a Deployment pod, it finishes, exits, and the Deployment restarts it again and again. You get CrashLoopBackOff, or the migration runs twice. Work that has an end needs a different controller: the Job. A Job runs pods until a set number of them succeed. If a pod fails, the Job retries it, up to a limit. When the work is done, the Job is Complete and the pods stop. Jobs can run one pod, many pods one after another, or many pods at the same time. Indexed Jobs give each pod a number, so pod 0 does photos 0 to 999, pod 1 does 1000 to 1999, and so on. In this lesson we run a simple Job, a parallel Job, an indexed Job, and a migration with retries and timeouts. Then we break them with a wrong restartPolicy, endless retries and Jobs that never get cleaned up.

What: every term in plain English

• Job — A batch/v1 controller that runs pods until enough of them finish successfully.
• completions — How many successful pods the Job needs. Default 1.
• parallelism — How many pods may run at the same time. Default 1.
• backoffLimit — How many failed retries before the Job is marked Failed. Default 6.
• activeDeadlineSeconds — Maximum total time for the Job. After it, pods are killed and the Job fails.
• ttlSecondsAfterFinished — Delete the Job and its pods this many seconds after it finishes.
• completionMode: Indexed — Each pod gets JOB_COMPLETION_INDEX 0, 1, 2 ... so it knows its share of work.
• restartPolicy — Must be Never or OnFailure in a Job. Never makes a new pod per retry; OnFailure restarts the container in the same pod.
• podFailurePolicy — Rules like: exit code 42 means do not retry; node eviction does not count as a failure.
• suspend — true pauses a Job: no new pods start.

How: step by step

1. Write the work as a command that exits. Exit 0 for success, non-zero for failure.
2. Pick restartPolicy. Never to keep failed pods for logs; OnFailure to save pods.
3. Set completions and parallelism. 1/1 for a migration; N/M for a batch.
4. Limit retries and time. backoffLimit and activeDeadlineSeconds.
5. Clean up automatically. ttlSecondsAfterFinished.
6. Apply and wait. kubectl wait --for=condition=complete.
7. Read logs and status. kubectl logs job/NAME and kubectl describe job.

Example 1: A single migration Job with retries, a deadline and cleanup

Goal. A single migration Job with retries, a deadline and cleanup.

migrate-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-v42
  namespace: shop
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 600
  template:
    metadata:
      labels:
        app: migrate
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - echo "migration 42 start";
              sleep 5;
              echo "ALTER TABLE products ADD COLUMN brand VARCHAR(64)";
              echo "migration 42 done"
          resources:
            requests: {cpu: 50m, memory: 32Mi}
            limits: {memory: 64Mi}

What each line does

• name: migrate-v42 — Put the version in the name. Job templates are immutable, so each release gets a new Job.
• backoffLimit: 2 — At most 2 retries.
• activeDeadlineSeconds: 300 — Kill it if it takes more than 5 minutes.
• ttlSecondsAfterFinished: 600 — The Job and pods are deleted 10 minutes after finishing.
• restartPolicy: Never — Each retry is a new pod, so failed logs stay readable.
• echo ALTER TABLE — Stands in for a real tool like Flyway or Liquibase.

Green output

$ kubectl apply -f migrate-job.yaml
job.batch/migrate-v42 created
$ kubectl wait -n shop --for=condition=complete job/migrate-v42 --timeout=120s
job.batch/migrate-v42 condition met
$ kubectl get job migrate-v42 -n shop
NAME          STATUS     COMPLETIONS   DURATION   AGE
migrate-v42   Complete   1/1           8s         12s
$ kubectl logs -n shop job/migrate-v42
migration 42 start
ALTER TABLE products ADD COLUMN brand VARCHAR(64)
migration 42 done

Expected result. The Job finished once and stopped. In CI, run kubectl wait before kubectl apply of the new Deployment.

Example 2: A parallel Job: 6 completions, 3 at a time, with a failure that retries

Goal. A parallel Job: 6 completions, 3 at a time, with a failure that retries.

resize-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: resize-photos
  namespace: shop
spec:
  completions: 6
  parallelism: 3
  backoffLimit: 4
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: resize
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - N=$((RANDOM % 4));
              echo "resizing a batch on $(hostname)";
              sleep 4;
              if [ "$N" = "0" ]; then echo "disk busy, failing"; exit 1; fi;
              echo "batch done"

What each line does

• completions: 6 — Six pods must succeed.
• parallelism: 3 — At most three run at once.
• RANDOM % 4 — About one in four pods fails, to see retries.
• exit 1 — A failure. The Job starts a replacement pod and counts it toward backoffLimit.
• backoffLimit: 4 — Up to 4 failures are allowed in total.

Green output

$ kubectl apply -f resize-job.yaml
$ kubectl get pods -n shop -l job-name=resize-photos -w
resize-photos-4kx2m   1/1   Running     0   1s
resize-photos-9fz7q   1/1   Running     0   1s
resize-photos-tq8wd   1/1   Running     0   1s
resize-photos-9fz7q   0/1   Error       0   5s
resize-photos-4kx2m   0/1   Completed   0   5s
resize-photos-tq8wd   0/1   Completed   0   5s
resize-photos-bm3lc   1/1   Running     0   0s
...
$ kubectl get job resize-photos -n shop
NAME            STATUS     COMPLETIONS   DURATION
resize-photos   Complete   6/6           19s
$ kubectl get job resize-photos -n shop -o jsonpath='{.status.succeeded} ok, {.status.failed} failed{"\n"}'
6 ok, 1 failed

Expected result. Three ran at a time, one failed and was retried, and the Job completed when 6 succeeded.

Example 3: An Indexed Job that splits 30,000 photos into 3 shares

Goal. An Indexed Job that splits 30,000 photos into 3 shares.

indexed-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: resize-indexed
  namespace: shop
spec:
  completions: 3
  parallelism: 3
  completionMode: Indexed
  backoffLimitPerIndex: 1
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes:
          containerName: resize
          operator: In
          values: [42]
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: resize
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - START=$((JOB_COMPLETION_INDEX * 10000));
              END=$((START + 9999));
              echo "index $JOB_COMPLETION_INDEX resizes photos $START to $END";
              sleep 3

What each line does

• completionMode: Indexed — Pods get index 0, 1 and 2 in the env var JOB_COMPLETION_INDEX.
• START and END — Each pod computes its own share. No queue needed.
• backoffLimitPerIndex: 1 — Each index may retry once on its own (Kubernetes 1.29+).
• podFailurePolicy FailJob on 42 — Exit code 42 means bad input: stop the whole Job now, do not retry.
• restartPolicy: Never — Required for podFailurePolicy.

Green output

$ kubectl apply -f indexed-job.yaml
$ kubectl wait -n shop --for=condition=complete job/resize-indexed --timeout=60s
job.batch/resize-indexed condition met
$ for i in 0 1 2; do kubectl logs -n shop -l job-name=resize-indexed,batch.kubernetes.io/job-completion-index=$i; done
index 0 resizes photos 0 to 9999
index 1 resizes photos 10000 to 19999
index 2 resizes photos 20000 to 29999

Expected result. Each pod knew its own share from the index. This is how ML and data teams split big batch work.

Every command, step by step (kind lab with kubectl)

1. Explain Job fields

   kubectl explain job.spec | grep -E 'completions|parallelism|backoffLimit|ttl'
   

   What it does: Read the field list from the API.

   backoffLimit    <integer>
   completions <integer>
   parallelism <integer>
   ttlSecondsAfterFinished <integer>
   

2. Quick Job from the CLI

   kubectl create job hello -n shop --image=busybox:1.36 -- echo hello from a job
   kubectl logs -n shop job/hello
   

   What it does: Handy for one-off tasks.

   job.batch/hello created
   hello from a job
   

3. Run the migration

   kubectl apply -f migrate-job.yaml
   kubectl wait -n shop --for=condition=complete job/migrate-v42 --timeout=120s
   

   What it does: Blocks until done or timeout.

   job.batch/migrate-v42 condition met
   

4. Run the parallel Job

   kubectl apply -f resize-job.yaml
   kubectl get job resize-photos -n shop -w
   

   What it does: COMPLETIONS counts up to 6/6.

   resize-photos   Running    2/6   8s
   resize-photos   Complete   6/6   19s
   

5. Read failed pod logs

   kubectl get pods -n shop -l job-name=resize-photos --field-selector=status.phase=Failed
   kubectl logs -n shop POD_NAME_FROM_ABOVE
   

   What it does: With restartPolicy Never, failed pods stay for debugging.

   resizing a batch on resize-photos-9fz7q
   disk busy, failing
   

6. Suspend and resume

   kubectl patch job resize-indexed -n shop -p '{"spec":{"suspend":true}}'
   kubectl patch job resize-indexed -n shop -p '{"spec":{"suspend":false}}'
   

   What it does: Pauses new pods; useful when a database is overloaded.

   job.batch/resize-indexed patched
   

7. Run a Job from a CronJob template

   kubectl create job manual-report -n shop --from=cronjob/daily-report
   

   What it does: Works after 60.7; runs a scheduled job right now.

   job.batch/manual-report created
   

8. Watch TTL cleanup

   kubectl get job migrate-v42 -n shop -o jsonpath='{.spec.ttlSecondsAfterFinished}{"\n"}'
   

   What it does: After 600 seconds the Job disappears by itself.

   600
   

9. Clean up

   kubectl delete job -n shop hello resize-photos resize-indexed --ignore-not-found
   

   What it does: Deleting a Job deletes its pods.

   job.batch "hello" deleted
   

When it is red 1

The Job "migrate-v42" is invalid: spec.template.spec.restartPolicy: Required value: valid values: "OnFailure", "Never"

Cause. restartPolicy Always (the default for pods) is not allowed in a Job.

Fix, step by step

1. Set restartPolicy: Never or OnFailure in the template.

When it is red 2

Warning  BackoffLimitExceeded  Job has reached the specified backoff limit

Cause. Pods kept failing more times than backoffLimit.

Fix, step by step

1. kubectl logs on a failed pod to see why.
2. Fix the code or config.
3. Delete the Job and apply it again; Jobs cannot be restarted.

When it is red 3

The Job "migrate-v42" is invalid: spec.template: Invalid value: ...: field is immutable

Cause. You changed the image or command of an existing Job.

Fix, step by step

1. Delete the old Job, or give the new one a new name like migrate-v43.

When it is red 4

kubectl get jobs shows hundreds of old Complete jobs

Cause. No ttlSecondsAfterFinished and no CronJob history limits.

Fix, step by step

1. Add ttlSecondsAfterFinished to every Job.
2. Delete old ones: kubectl delete jobs -n shop --field-selector status.successful=1.

Practice tasks

kubectl create job hello -n shop --image=busybox:1.36 -- date

kubectl wait -n shop --for=condition=complete job/migrate-v42

kubectl get job resize-photos -n shop -o jsonpath='{.status}'

kubectl get pods -n shop -l job-name=resize-photos -w

kubectl logs -n shop -l batch.kubernetes.io/job-completion-index=1

kubectl describe job NAME -n shop | grep -i deadline

kubectl get pods -n shop -l job-name=NAME

kubectl patch job resize-indexed -n shop -p '{"spec":{"suspend":true}}'

kubectl describe job NAME -n shop | grep -i PodFailurePolicy

kubectl wait -n shop --for=condition=complete job/migrate-v42 --timeout=300s || { kubectl logs -n shop job/migrate-v42; exit 1; }

Recap

• A Job runs pods until enough succeed, then stops.
• completions sets how many; parallelism sets how many at once.
• backoffLimit and activeDeadlineSeconds stop endless retries.
• Indexed Jobs split work by JOB_COMPLETION_INDEX; podFailurePolicy fails fast on bad input.
• Templates are immutable: new release, new Job name.
• ttlSecondsAfterFinished keeps the cluster clean. CronJobs (60.7) create Jobs on a schedule.

FAQ

Q. Job or Deployment for a queue worker?

If it runs forever pulling messages, a Deployment. If it drains a fixed queue and exits, a Job.

Q. How do I rerun a finished Job?

Delete it and apply again, or create a new one with a new name.

Q. Never or OnFailure?

Never keeps failed pods for logs. OnFailure saves pods but loses earlier logs.

Q. Can a Job run on Fargate?

Yes, on EKS with a Fargate profile; each pod gets its own micro VM.

Q. Should migrations run as Jobs or init containers?

A Job runs once per release; an init container would run in every pod, many times.

Q. What is Kueue?

A Kubernetes add-on that queues Jobs when the cluster is full. Used for ML and batch.

Interview questions

1. What is a Kubernetes Job?

   Model answer: A controller that runs one or more pods to successful completion, retrying failures up to backoffLimit.

2. completions vs parallelism?

   Model answer: completions is the total number of successful pods needed; parallelism is how many may run at the same time.

3. How do you prevent finished Jobs piling up?

   Model answer: ttlSecondsAfterFinished on Jobs, and successfulJobsHistoryLimit/failedJobsHistoryLimit on CronJobs.

4. What is an Indexed Job?

   Model answer: A Job with completionMode Indexed, where each pod gets a unique index to process its own partition of the work.

5. How would you run DB migrations in a CD pipeline?

   Model answer: Apply a versioned Job, kubectl wait for complete with a timeout, fail the pipeline on error, then roll out the Deployment.

Samajla ka?