Chapter 32: Jenkins Governance, Troubleshooting and Expert Projects

32.6 Queue stuck

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we are at Flipkart before a big sale. 60 builds sit in the Jenkins queue, agents look idle, and developers keep pressing Build Now, which makes it worse. A stuck queue has a few usual causes: no agent has the label the job asks for, a job waits for a lock or input, the executors are all busy with hung builds, or the Kubernetes cloud cannot create Pods. Jenkins tells you the reason if you hover on the queue item. This lesson teaches you to read that reason and clear the queue safely.

What we will build

You will read the queue reason for each item, fix a label mismatch, kill a hung build, release a lock, and add timeouts so builds can never hang forever again.

What: every term in plain English

• Build queue — Builds waiting for a free executor.
• Queue reason — The text Jenkins shows for why an item waits.
• Label expression — What agents a job may use, like linux && docker.
• Hung build — A build that runs forever, holding an executor.
• timeout step — Stops a stage or build after a set time.
• lock step — From the Lockable Resources plugin. Only one build may hold a resource at a time.
• input step — Pauses a build until a person clicks Proceed.

How: step by step

1. Look at the queue reasons

   Hover each item in Build Queue on the dashboard
   

   What it does: The reason tells you the cause.

   Output

   Waiting for next available executor on 'linux && gpu'
   There are no nodes with the label 'linux && gpu'
   

2. List queue items by API

   curl -s -u ravi:$JENKINS_TOKEN "http://localhost:8080/queue/api/json?tree=items[id,why,task[name]]" | jq -r ".items[] | \"\(.id) \(.task.name): \(.why)\""
   

   What it does: Good when the queue is long.

   Output

   812 ml-train: There are no nodes with the label linux && gpu
   813 orders-api: Waiting for next available executor
   

3. Find hung builds

   Manage Jenkins > Nodes: look at builds running for hours
   

   What it does: A 9-hour build of a 5-minute job is hung.

   Output

   orders-api #4410  running 9 hr
   

4. Read the hung build console

   Open the build > Console Output, scroll to the end
   

   What it does: Usually waits for a network or a prompt.

   Output

   + npm ci
   (no new lines for 9 hours)
   

5. Abort it

   Build page > the red X, or POST /job/orders-api/4410/stop
   

   What it does: Frees the executor.

   Output

   Aborted by ravi
   Finished: ABORTED
   

6. Fix the label or add an agent

   Job config: change label, or add an agent with that label
   

   What it does: Removes the "no nodes" reason.

   Output

   Label linux && docker matches 4 nodes
   

7. Watch the queue drain

   Refresh the dashboard
   

   What it does: Count goes down.

   Output

   Build Queue (3)
   

Example 1: Add timeouts and safe retries so builds cannot hang forever

Stack. Node.js, Declarative Pipeline

Goal. Add timeouts and safe retries so builds cannot hang forever.

Jenkinsfile

pipeline {
  agent { label 'linux && docker' }
  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds(abortPrevious: true)
    buildDiscarder(logRotator(numToKeepStr: '30'))
  }
  stages {
    stage('Install') {
      steps {
        timeout(time: 5, unit: 'MINUTES') {
          retry(2) { sh 'npm ci --no-audit --fetch-timeout=60000' }
        }
      }
    }
    stage('Test') { steps { sh 'npm test -- --ci' } }
  }
}

What each line does

• label 'linux && docker' — Only agents with both labels. Check real agents have both.
• timeout(time: 30, ...) — The whole build stops after 30 minutes.
• disableConcurrentBuilds(abortPrevious: true) — A new push on the same branch aborts the old build, so the queue does not pile up.
• timeout 5 minutes around npm ci — Network hangs end fast.
• retry(2) — Tries once more if the network blips.

Green output

[Pipeline] timeout
Timeout set to expire in 30 min
[Pipeline] retry
+ npm ci --no-audit --fetch-timeout=60000
added 812 packages in 41s
Finished: SUCCESS

Expected result. No build can hold an executor for hours again.

Example 2: Lock a shared staging environment, so builds queue in order instead of breaking each other

Stack. Lockable Resources plugin, shell deploy

Goal. Lock a shared staging environment, so builds queue in order instead of breaking each other.

Jenkinsfile

pipeline {
  agent { label 'linux && docker' }
  stages {
    stage('Deploy to staging') {
      options { lock(resource: 'staging-env', inversePrecedence: true) }
      steps {
        sh './deploy.sh staging'
        sh './smoke.sh https://staging.example.com'
      }
    }
  }
}

Script Console (stuck lock)

def lrm = org.jenkins.plugins.lockableresources.LockableResourcesManager.get()
def r = lrm.fromName('staging-env')
println "locked by: ${r.buildName}"
// only after you confirm that build is dead:
r.reset()
lrm.save()

What each line does

• lock(resource: 'staging-env') — One build at a time deploys to staging.
• inversePrecedence: true — The newest waiting build goes next, older ones are less useful.
• r.buildName — Who holds the lock now.
• r.reset() — Releases a lock left by a dead build. Use it only after checking.

Green output

[Pipeline] lock
Trying to acquire lock on [Resource: staging-env]
Resource [staging-env] is locked by build payments-api #233 and 2 builds are waiting
...
Lock acquired on [Resource: staging-env]

Expected result. Staging deploys run one by one, and you can free a lock left by a crashed build.

Example 3: Kubernetes cloud cannot create Pods: read the cloud log and set limits

Stack. Kubernetes plugin, EKS or AKS

Goal. Kubernetes cloud cannot create Pods: read the cloud log and set limits.

terminal

kubectl -n jenkins get pods --field-selector=status.phase=Pending
kubectl -n jenkins get resourcequota
kubectl -n jenkins describe pod build-orders-12-abcde | tail -3

Jenkins cloud settings

Manage Jenkins > Clouds > kubernetes
  Concurrency Limit: 20
  Pod Retention: Never
  Seconds to wait for pod to be running: 300
Manage Jenkins > System Log > add logger org.csanchez.jenkins.plugins.kubernetes, level FINE

What each line does

• status.phase=Pending — Pods that cannot start.
• resourcequota — The namespace may have a limit on pods or CPU.
• Concurrency Limit — Max agent Pods at once. Match it to the quota.
• Seconds to wait — Jenkins gives up on a Pod after this and tries again.
• logger FINE — Shows why Jenkins cannot create Pods.

Green output

NAME                     READY  STATUS
build-orders-12-abcde    0/2    Pending
NAME          AGE  REQUEST                         LIMIT
jenkins-quota 90d  pods: 20/20, requests.cpu: 18/20
Warning  FailedCreate  exceeded quota: jenkins-quota, requested: pods=1, used: pods=20, limited: pods=20

Expected result. You see the quota is full. Raise the quota or lower Concurrency Limit to 20 so Jenkins stops trying, and the queue drains as Pods finish.

Verify: did it work?

1. Queue length

   curl -s -u ravi:$JENKINS_TOKEN "http://localhost:8080/queue/api/json" | jq ".items | length"
   

   What it does: Should drop toward 0.

   Output

   2
   

2. Executors busy

   Dashboard > Build Executor Status
   

   What it does: Builds running normal times.

   Output

   All builds under 20 min
   

3. Locks free

   Manage Jenkins > Lockable Resources
   

   What it does: Nothing stuck.

   Output

   staging-env  FREE
   

When it is red 1

There are no nodes with the label 'linux && gpu'

Cause. No agent has that label, or the label was typed wrong.

Fix, step by step

1. Check agent labels on the node pages.
2. Fix the job label, or add the label to an agent.

When it is red 2

All nodes of label linux are offline

Cause. Every agent with the label is down.

Fix, step by step

1. Follow the agent offline lesson.
2. Bring at least one back.

When it is red 3

Waiting for next available executor on 'linux'

Cause. All executors busy, maybe with hung builds.

Fix, step by step

1. Abort hung builds.
2. Add timeouts.
3. Add agents or Pod capacity.

When it is red 4

Build is waiting for input: Deploy to prod? Proceed or Abort

Cause. An input step is holding the build. With agent any at top, it also holds an executor.

Fix, step by step

1. Click Proceed or Abort.
2. Move input outside the agent block, with agent none at the top, so it does not hold an executor.

Cleanup

1. Clear the whole queue (careful)

   Script Console: Jenkins.instance.queue.clear()
   

   What it does: Only when everything is junk, like after an accident.

   Output

   (queue empty)
   

2. Remove a test lock

   Manage Jenkins > Lockable Resources > staging-test > Delete
   

   What it does: Lab only.

   Output

   Resource deleted
   

Practice tasks

Homework: try at home

Recap

• Hover a queue item to see the reason.
• No nodes with label means a label mismatch.
• Busy executors often mean hung builds: add timeouts.
• Use lock for shared environments and know how to free it.
• Kubernetes queues often mean a quota or capacity limit.

FAQ

Is Queue.clear() safe?

It removes all waiting builds. Use it only after an accident, and tell the team.

Can I give one job priority?

Yes, with the Priority Sorter plugin.

How do I stop the same branch from piling up?

disableConcurrentBuilds(abortPrevious: true).

Why do builds wait even though agents look idle?

Usually labels do not match, or the agent is temporarily offline.

Does input use an executor?

Only when it is inside an agent. Put it at agent none level.

How many builds should queue at most?

There is no rule. Watch trends: if it grows every day, add capacity.

Interview questions

1. Jenkins queue is stuck. How do you troubleshoot?

   Read the queue reason, check labels and offline agents, find hung builds and abort them, check locks and cloud capacity, then add timeouts.

2. How do you prevent hung builds?

   Pipeline and stage timeouts, retries with limits, abortPrevious on branches.

3. What does the Lockable Resources plugin do?

   Makes builds wait for a named shared resource so they do not collide.

4. Why should input not run inside an agent?

   It holds the executor while waiting for a person.

5. How do you scale build capacity?

   More agents or Kubernetes Pod agents with autoscaling, matched to quotas.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards