Chapter 27: Jenkins Agents

27.5 Kubernetes pods that go away

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Think of Flipkart before the Big Billion Days sale. Developers push hundreds of changes a day, and the build queue grows to 300 jobs in the morning. At night it drops to almost zero. Permanent agents waste money at night and are too few in the morning. They also collect junk: old workspaces, leftover containers, cached passwords. Kubernetes agents fix both problems. When a build starts, the Jenkins Kubernetes plugin asks the cluster for a new pod. The pod has a jnlp container that connects back to Jenkins, plus tool containers like maven or node. The build runs. When it ends, the pod is deleted. Nothing stays behind. The cluster autoscaler adds nodes when many pods wait and removes them when the queue is empty. In this lesson you will connect Jenkins to a cluster, write a pod template in YAML inside the Jenkinsfile, use several containers in one pod, set CPU and memory limits, and watch pods appear and vanish.

What we will build

You will create a jenkins namespace and a service account in a lab cluster such as kind, minikube or EKS. You will add a Kubernetes cloud in Jenkins. Then you will run three pipelines: a simple maven pod, a pod with node and maven containers plus resource limits, and a pod that uses a shared cache volume. You will watch every pod with kubectl get pods -w.

What: every term in plain English

• Kubernetes plugin — Jenkins plugin that creates agent pods in a cluster for each build.
• Cloud — Jenkins setting that says where the cluster is and which namespace to use.
• Pod template — The pod shape for a build: containers, images, resources, volumes. Can live in the Jenkinsfile as YAML.
• jnlp container — The container in each agent pod that runs agent.jar and talks to Jenkins. Added automatically.
• container('maven') — Pipeline step that runs the next steps inside the named container of the pod.
• Ephemeral — Created for one build and deleted after it. No leftovers.
• Requests and limits — CPU and memory a container is guaranteed (request) and the most it may use (limit).
• Service account — A Kubernetes identity. Jenkins uses it to create and delete pods in its namespace.

How: step by step

1. Create a namespace and service account

   kubectl create namespace jenkins-agents && kubectl -n jenkins-agents create serviceaccount jenkins
   

   What it does: A separate place for agent pods.

   Output

   namespace/jenkins-agents created
   serviceaccount/jenkins created
   

2. Give it pod rights only in that namespace

   kubectl -n jenkins-agents create role jenkins-agents --verb=create,delete,get,list,watch --resource=pods,pods/log,pods/exec && kubectl -n jenkins-agents create rolebinding jenkins-agents --role=jenkins-agents --serviceaccount=jenkins-agents:jenkins
   

   What it does: Least privilege: pods in one namespace, nothing else.

   Output

   role.rbac.authorization.k8s.io/jenkins-agents created
   

3. Make a token for Jenkins

   kubectl -n jenkins-agents create token jenkins --duration=8760h
   

   What it does: Jenkins stores this as a Secret text credential with ID k8s-agents-token.

   Output

   eyJhbGciOiJSUzI1NiIs... (long token)
   

4. Add the cloud

   Manage Jenkins > Clouds > New cloud > Kubernetes > URL https://API_SERVER > Namespace jenkins-agents > Credentials k8s-agents-token > Jenkins URL http://jenkins.jenkins.svc:8080
   

   What it does: Tells Jenkins how to reach the cluster and how pods reach back.

   Output

   Connected to Kubernetes v1.30.2
   

5. Watch pods

   kubectl -n jenkins-agents get pods -w
   

   What it does: Keep this open while you start builds.

   Output

   No resources found in jenkins-agents namespace.
   

6. Run a pod build

   Build Now on the job with agent { kubernetes { yaml ... } }
   

   What it does: A pod is created for the build.

   Output

   shop-api-12-x7k2p-1n9q4   0/2   ContainerCreating
   

7. See it go away

   kubectl -n jenkins-agents get pods
   

   What it does: After the build the pod is gone.

   Output

   No resources found in jenkins-agents namespace.
   

Example 1: A simple Maven pod agent

Stack. Jenkins Kubernetes plugin, kind or EKS, Maven

Goal. A simple Maven pod agent.

Jenkinsfile

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.9-eclipse-temurin-21
    command: ['sleep']
    args: ['infinity']
"""
      defaultContainer 'maven'
    }
  }
  stages {
    stage('Info') { steps { sh 'hostname; mvn -v | head -1' } }
    stage('Build') { steps { sh 'mvn -B -q package -DskipTests && ls target/*.jar' } }
  }
}

What each line does

• kubernetes { yaml ... } — The pod shape for this build, kept in Git with the code.
• name: maven — The tool container. Jenkins adds jnlp next to it.
• command sleep, args infinity — Keeps the tool container alive until Jenkins is done.
• defaultContainer 'maven' — All sh steps run in maven unless you say otherwise.
• hostname — Prints the pod name, which changes every build.

Green output

Created Pod: kubernetes jenkins-agents/shop-api-12-x7k2p-1n9q4
Agent shop-api-12-x7k2p-1n9q4 is provisioned from template shop-api_12-x7k2p
Running on shop-api-12-x7k2p-1n9q4 in /home/jenkins/agent/workspace/shop-api
+ hostname
shop-api-12-x7k2p-1n9q4
Apache Maven 3.9.8
target/shop-api-0.0.1.jar
Finished: SUCCESS

(kubectl -w)
shop-api-12-x7k2p-1n9q4   0/2   ContainerCreating   0   1s
shop-api-12-x7k2p-1n9q4   2/2   Running             0   6s
shop-api-12-x7k2p-1n9q4   2/2   Terminating         0   41s

Expected result. A pod was created, ran the build and was deleted. Run it again and the pod name changes.

Example 2: Two tool containers with CPU and memory limits

Stack. Jenkins Kubernetes plugin, node:20-alpine, Maven

Goal. Two tool containers with CPU and memory limits.

Jenkinsfile

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    team: shop
spec:
  containers:
  - name: node
    image: node:20-alpine
    command: ['sleep']
    args: ['infinity']
    resources:
      requests: { cpu: '500m', memory: '512Mi' }
      limits:   { cpu: '1',    memory: '1Gi' }
  - name: maven
    image: maven:3.9-eclipse-temurin-21
    command: ['sleep']
    args: ['infinity']
    resources:
      requests: { cpu: '1', memory: '1Gi' }
      limits:   { cpu: '2', memory: '2Gi' }
"""
    }
  }
  stages {
    stage('Web') { steps { container('node')  { dir('web') { sh 'npm ci && npm run build' } } } }
    stage('API') { steps { container('maven') { dir('api') { sh 'mvn -B -q package -DskipTests' } } } }
  }
}

What each line does

• labels team: shop — Lets admins see cost and pods per team.
• requests cpu 500m — Kubernetes reserves half a CPU for node.
• limits memory 1Gi — If node uses more, the container is killed with OOMKilled.
• container('node') — Run these steps in the node container.
• container('maven') — Run these steps in the maven container. Same workspace, shared by all containers in the pod.

Green output

[Pipeline] container
+ npm ci
added 318 packages in 8s
+ npm run build
built in 4.0s
[Pipeline] container
+ mvn -B -q package -DskipTests
Finished: SUCCESS

$ kubectl -n jenkins-agents top pod -l team=shop
NAME                        CPU(cores)   MEMORY(bytes)
shop-web-api-7-q2m9d-4zt8k  1140m        1310Mi

Expected result. One pod ran both builds. Limits stop one greedy build from hurting others on the node.

Example 3: Keep a Maven cache with a persistent volume

Stack. Jenkins Kubernetes plugin, EKS, EFS CSI driver

Goal. Keep a Maven cache with a persistent volume.

cache-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: maven-cache
  namespace: jenkins-agents
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: efs-sc
  resources:
    requests:
      storage: 10Gi

Jenkinsfile

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  containers:
  - name: maven
    image: maven:3.9-eclipse-temurin-21
    command: ['sleep']
    args: ['infinity']
    env:
    - name: MAVEN_OPTS
      value: '-Dmaven.repo.local=/cache/m2'
    volumeMounts:
    - name: m2
      mountPath: /cache
  volumes:
  - name: m2
    persistentVolumeClaim:
      claimName: maven-cache
"""
      defaultContainer 'maven'
    }
  }
  stages { stage('Build') { steps { sh 'time mvn -B -q package -DskipTests' } } }
}

What each line does

• PersistentVolumeClaim maven-cache — Disk that lives longer than any pod.
• ReadWriteMany with efs-sc — Many pods can use it at once. On AWS this is EFS.
• runAsUser 1000, fsGroup 1000 — Run as a normal user and let it write the volume. No root.
• MAVEN_OPTS -Dmaven.repo.local=/cache/m2 — Maven uses the shared cache.
• time mvn ... — Shows how long the build took, so you can compare.

Green output

$ kubectl apply -f cache-pvc.yaml
persistentvolumeclaim/maven-cache created
(first build)
real    2m41s
(second build, new pod)
real    0m38s
Finished: SUCCESS

Expected result. The pod is new each time, but the cache stays. Builds stay clean and fast.

Verify: did it work?

1. Pods per build

   kubectl -n jenkins-agents get pods -l jenkins/label --show-labels
   

   What it does: Shows agent pods while builds run.

   Output

   shop-api-13-...   2/2   Running
   

2. Events for a stuck pod

   kubectl -n jenkins-agents describe pod shop-api-13-abcde | tail -8
   

   What it does: The Events part says why a pod cannot start.

   Output

   Warning  FailedScheduling  0/3 nodes are available: 3 Insufficient cpu.
   

When it is red 1

0/3 nodes are available: 3 Insufficient cpu.

Cause. The requests are bigger than free space on the nodes.

Fix, step by step

1. Lower requests in the pod YAML.
2. Add nodes or turn on the cluster autoscaler.
3. Check other pods with kubectl top nodes.

When it is red 2

ERROR: Failed to launch shop-api-14-...
io.fabric8.kubernetes.client.KubernetesClientException: Forbidden: pods is forbidden: User "system:serviceaccount:jenkins-agents:jenkins" cannot create resource "pods"

Cause. The role or rolebinding is missing or in the wrong namespace.

Fix, step by step

1. kubectl -n jenkins-agents get role,rolebinding.
2. kubectl auth can-i create pods -n jenkins-agents --as=system:serviceaccount:jenkins-agents:jenkins.
3. Fix the binding and retry.

When it is red 3

Agent went offline during build
terminated with OOMKilled

Cause. A container used more memory than its limit.

Fix, step by step

1. Raise the memory limit.
2. Lower JVM heap with MAVEN_OPTS -Xmx.
3. Check kubectl describe pod for Reason OOMKilled.

Cleanup

1. Delete the lab namespace

   kubectl delete namespace jenkins-agents
   

   What it does: Removes all agent pods, roles and the cache claim.

   Output

   namespace "jenkins-agents" deleted
   

Practice tasks

Homework: try at home

Recap

• Kubernetes agents are pods created per build and deleted after.
• Pod YAML in the Jenkinsfile keeps tools in Git.
• Use container() to pick the tool container.
• Set requests and limits so builds share nodes fairly.
• Give Jenkins rights only in its agent namespace.

FAQ

Does Jenkins itself have to run in Kubernetes?

No. An outside Jenkins can use a cluster if pods can reach it back.

How fast does a pod start?

A few seconds when images are cached on the node.

Can I keep files between builds?

Only through volumes like a cache PVC, or by archiving artifacts.

Can Windows builds use pods?

Yes, with Windows nodes in the cluster and a nodeSelector.

What about AWS keys in pods?

Use IRSA or EKS Pod Identity on the service account, not keys.

Interview questions

1. How do Kubernetes agents work in Jenkins?

   The Kubernetes plugin creates a pod from a template for each build. The jnlp container connects to Jenkins, steps run in tool containers, and the pod is deleted after.

2. Why use ephemeral agents?

   Clean state every build, no leftover secrets, and automatic scaling with the cluster.

3. A pod agent stays Pending. How do you debug it?

   kubectl describe pod and read Events: scheduling, image pull, quota or resource problems.

4. How do you keep builds fast on pod agents?

   Pre-pulled or small images, cache volumes, and right-sized requests.

5. What RBAC does Jenkins need in the cluster?

   Create, delete, get, list and watch pods, pods/log and pods/exec in the agent namespace only.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards