Chapter 25: Shared Libraries, Credentials and Environments

25.6 DEV, TEST, STAGING, PROD

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

At Infosys, a client project has four environments. DEV is for developers. TEST is for testers. STAGING is a copy of production for the final check. PROD is what customers use. A change must move through them in order. If a developer can push straight to PROD, one typo can stop a bank's app. So the pipeline must build the image once, deploy the same image to each environment, and stop before PROD until a named person approves. Jenkins does this with stages, when conditions, the input step, a timeout and the milestone and lock steps. In this lesson you will build the full promotion pipeline and test both the approve path and the reject path.

What we will build

You will build one Jenkinsfile for badge-api. It builds and pushes one image tag. It deploys to DEV automatically on main. It deploys to TEST after DEV smoke passes. It asks for approval for STAGING and again, from a release group only, for PROD. It times out if nobody answers, and it uses a lock so two PROD deploys never overlap.

What: every term in plain English

• Environment — A separate copy of the app, like DEV, TEST, STAGING or PROD.
• Promotion — Moving the same image from one environment to the next.
• Build once — Make the image one time and reuse its tag everywhere. No rebuilds.
• input step — Pauses the pipeline and shows Proceed and Abort buttons.
• submitter — Who may click Proceed, like a user or group name.
• timeout — Stops waiting after a set time.
• lock — From the Lockable Resources plugin. Only one build can hold it at a time.
• milestone — Cancels older builds that reach a point after a newer one.

How: step by step

1. Install plugins

   Manage Jenkins > Plugins > Lockable Resources, Pipeline: Milestone Step
   

   What it does: Adds lock and milestone.

   Output

   Installed
   

2. Make the release group

   Manage Jenkins > Security > add group release-managers (or use your SSO group)
   

   What it does: Only these users can approve PROD.

   Output

   Group ready
   

3. Add kubeconfigs

   Credentials: kubeconfig-dev, kubeconfig-test, kubeconfig-staging, kubeconfig-prod
   

   What it does: One secret file per environment.

   Output

   4 credentials
   

4. Write the Jenkinsfile

   vim Jenkinsfile
   

   What it does: The promotion pipeline below.

   Output

   (saved)
   

5. Push to main

   git push origin main
   

   What it does: Triggers the multibranch job.

   Output

   Started by branch indexing
   

6. Approve staging

   Build page > Paused for Input > Proceed
   

   What it does: Pipeline continues to STAGING.

   Output

   Approved by ravi
   

7. Approve prod as release manager

   Log in as a release-managers user > Proceed
   

   What it does: Only that group can continue to PROD.

   Output

   Approved by priya
   

Example 1: Deploy one image through four environments

Stack. Jenkins, Kubernetes, shared library

Goal. Deploy one image through four environments.

vars/deployTo.groovy (in badge-lib)

def call(String envName, String image) {
  withCredentials([file(credentialsId: "kubeconfig-${envName}", variable: 'KUBECONFIG')]) {
    sh "kubectl -n badge set image deploy/badge-api app=${image}"
    sh "kubectl -n badge rollout status deploy/badge-api --timeout=180s"
  }
}

Jenkinsfile

@Library('badge-lib@v1.4.0') _
pipeline {
  agent { label 'docker' }
  environment { IMAGE = "registry.example.com/badge-api:${env.GIT_COMMIT.take(7)}" }
  stages {
    stage('Build once') { steps { sh 'docker build -t $IMAGE . && docker push $IMAGE' } }
    stage('DEV')  { when { branch 'main' } steps { deployTo('dev', env.IMAGE); smokeTest('https://dev.badge.example.com') } }
    stage('TEST') { when { branch 'main' } steps { deployTo('test', env.IMAGE); smokeTest('https://test.badge.example.com') } }
  }
}

What each line does

• IMAGE uses GIT_COMMIT.take(7) — Tag is the short commit id. Same tag for every environment.
• Build once — Only one docker build per commit.
• kubeconfig-${envName} — Picks the right cluster secret by name.
• when { branch 'main' } — Feature branches build and test but never deploy.
• smokeTest — From lesson 25.2. Fails the stage if health is not 200.

Green output

[Pipeline] stage (Build once)
+ docker push registry.example.com/badge-api:8f3a2c1
[Pipeline] stage (DEV)
deployment "badge-api" successfully rolled out
SMOKE OK
[Pipeline] stage (TEST)
deployment "badge-api" successfully rolled out
SMOKE OK
Finished: SUCCESS

Expected result. DEV and TEST run the same image 8f3a2c1. What you test is what you ship.

Example 2: Add approval gates for STAGING and PROD

Stack. Jenkins input step

Goal. Add approval gates for STAGING and PROD.

Jenkinsfile (extra stages)

stage('Approve staging') {
  when { branch 'main' }
  steps {
    timeout(time: 30, unit: 'MINUTES') {
      input message: "Deploy ${env.IMAGE} to STAGING?", ok: 'Deploy'
    }
  }
}
stage('STAGING') { when { branch 'main' } steps { deployTo('staging', env.IMAGE) } }
stage('Approve prod') {
  when { branch 'main' }
  agent none
  steps {
    timeout(time: 4, unit: 'HOURS') {
      script {
        def r = input message: "Ship ${env.IMAGE} to PROD?", ok: 'Ship it',
                      submitter: 'release-managers',
                      parameters: [string(name: 'TICKET', description: 'Change ticket id')]
        env.CHANGE_TICKET = r
      }
    }
  }
}

What each line does

• timeout(...) around input — If nobody answers in 30 minutes, the build is aborted, not stuck forever.
• input message: ..., ok: 'Deploy' — Shows a pause with a Deploy button.
• submitter: 'release-managers' — Only that group can approve PROD.
• parameters inside input — The approver must type a change ticket id.
• agent none on the approval stage — No executor is held while waiting for a human.

Green output

[Pipeline] stage (Approve staging)
Deploy registry.example.com/badge-api:8f3a2c1 to STAGING?
Deploy or Abort
Approved by ravi
[Pipeline] stage (STAGING)
deployment "badge-api" successfully rolled out
[Pipeline] stage (Approve prod)
Ship registry.example.com/badge-api:8f3a2c1 to PROD?
Approved by priya
CHANGE_TICKET=CHG-20481

Expected result. Two human gates. PROD needs the right group and a ticket.

Example 3: Stop overlapping PROD deploys with lock and milestone

Stack. Jenkins, Lockable Resources, Milestone

Goal. Stop overlapping PROD deploys with lock and milestone.

Jenkinsfile (PROD stage)

stage('PROD') {
  when { branch 'main' }
  steps {
    milestone(ordinal: 100, label: 'before prod')
    lock(resource: 'badge-prod', inversePrecedence: true) {
      deployTo('prod', env.IMAGE)
      smokeTest('https://badge.example.com')
      echo "PROD now runs ${env.IMAGE} (ticket ${env.CHANGE_TICKET})"
    }
  }
}
post {
  failure { echo 'Deploy failed. Run: kubectl -n badge rollout undo deploy/badge-api' }
}

What each line does

• milestone(ordinal: 100) — If a newer build passes this point first, older builds are cancelled. Old code never overwrites new code.
• lock(resource: 'badge-prod') — Only one build at a time can be inside this block.
• inversePrecedence: true — The newest waiting build gets the lock next.
• post failure — Prints the rollback command when PROD fails.
• kubectl rollout undo — Goes back to the previous ReplicaSet in seconds.

Green output

[Pipeline] milestone
Trying to pass milestone 100
[Pipeline] lock
Trying to acquire lock on [Resource: badge-prod]
Lock acquired on [Resource: badge-prod]
deployment "badge-api" successfully rolled out
SMOKE OK
PROD now runs registry.example.com/badge-api:8f3a2c1 (ticket CHG-20481)
Lock released on resource [Resource: badge-prod]
Finished: SUCCESS

Expected result. PROD is safe from two deploys at once and from old builds landing last.

Verify: did it work?

1. Check what runs in each environment

   for e in dev test staging prod; do kubectl --context $e -n badge get deploy badge-api -o jsonpath="{.spec.template.spec.containers[0].image}"; echo; done
   

   What it does: Prints the image tag in each environment.

   Output

   ...badge-api:8f3a2c1
   ...badge-api:8f3a2c1
   ...badge-api:8f3a2c1
   ...badge-api:8f3a2c1
   

2. See who approved

   Build page > Console, search Approved by
   

   What it does: Audit trail for every promotion.

   Output

   Approved by priya
   

When it is red 1

Rejected by ravi
Finished: ABORTED

Cause. Someone clicked Abort at the input.

Fix, step by step

1. This is the reject path working.
2. Read the reason, fix, and push again.

When it is red 2

Timeout has been exceeded
Finished: ABORTED

Cause. Nobody approved in time.

Fix, step by step

1. Rerun from the stage with Restart from Stage.
2. Agree approval windows with the team.

When it is red 3

ravi is not permitted to submit this input

Cause. The user is not in release-managers.

Fix, step by step

1. Ask a release manager to approve.
2. Check the group name in submitter matches exactly.

Cleanup

1. Release a stuck lock

   Manage Jenkins > Lockable Resources > badge-prod > Reset
   

   What it does: Frees the lock if a build died while holding it.

   Output

   badge-prod free
   

2. Clean old images

   docker image prune -a --filter "until=168h" -f
   

   What it does: Removes week-old images from the agent.

   Output

   Total reclaimed space: 3.2GB
   

Practice tasks

Homework: try at home

Recap

• Build once, deploy the same tag everywhere.
• Automate DEV and TEST.
• Gate STAGING and PROD with input and timeout.
• Limit PROD approval with submitter.
• Use lock and milestone for safe PROD deploys.

FAQ

Should each environment have its own Jenkins?

Usually no. One Jenkins with separate credentials per environment is common.

Can approval come from Slack?

Yes with plugins, but the Jenkins input step is the simple start.

Why not rebuild per environment?

A rebuild can pull different packages. Then you ship something you never tested.

Where do environment configs live?

In ConfigMaps or Helm values per environment, not inside the image.

How do I roll back?

kubectl rollout undo, or redeploy the previous tag with the same job.

Interview questions

1. How do you design a multi-environment Jenkins pipeline?

   Build and push one image tagged by commit, deploy it to DEV and TEST automatically, gate STAGING and PROD with input, timeout and submitter, and lock PROD.

2. What is the input step and its risks?

   It pauses for approval. Without a timeout and agent none it can hold executors forever.

3. How do you stop two PROD deploys overlapping?

   Use lock on a PROD resource and milestone to drop older builds.

4. How do you keep secrets separate per environment?

   One credential id per environment, picked by name in the deploy step.

5. How do you audit who deployed what?

   Console shows Approved by, the input ticket parameter, and build descriptions with env and tag.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards