Chapter 56: Docker CI/CD with Jenkins and ECR, Deploy to Amazon ECS, Troubleshooting, Best Practices and Interview Questions

56.1 CI/CD: a Jenkins pipeline that builds, scans and pushes to Amazon ECR

By Ravindra Bagale Last updated: 25 October 2026

Friends, until now we built images by hand on our laptop and pushed them ourselves. In a real company nobody does that. A machine called Jenkins watches the code, and every time someone merges, it builds the image, scans it for security holes, tags it with the commit and pushes it to Amazon ECR. Today, step by step, we build that pipeline from zero: the ECR repository, the Jenkins agent, the IAM role, the Jenkinsfile, each stage, and how to fix it when it fails.

1. Start from a story

At Infosys, a team of six developers worked on a shop API. Every Friday, one developer named Rohit built the Docker image on his own laptop and pushed it to the registry for the release.

One Friday Rohit was on leave. His colleague built the image from her laptop, but she had an old copy of the code and a different base image cached. The release went out with a bug that had already been fixed.

The team lead said: no more laptop builds. From now on Jenkins builds every image from the main branch, scans it with Trivy, tags it with the Git commit, and pushes it to Amazon ECR. Anyone can see exactly which commit is inside which image.

The following Friday the release took four minutes and nobody touched a laptop. Today we build the same pipeline, one small step at a time.

Picture 1: the pipeline from commit to ECR

2. Prepare AWS step by step

Step 1: Create the ECR repository

Why: The image needs a home in AWS.

What: Amazon ECR is a private Docker registry inside your AWS account. Each application gets one repository. Turn on scan on push and make tags immutable so a tag can never be overwritten.

Example 1: Create the repository

Type this:

aws ecr create-repository --repository-name shop-api --image-tag-mutability IMMUTABLE --image-scanning-configuration scanOnPush=true --region ap-south-1 --query 'repository.repositoryUri' --output text

You will see:

123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api

What it means: This URI is the full image name we will push to.

Example 2: List repositories

Type this:

aws ecr describe-repositories --region ap-south-1 --query 'repositories[].repositoryName' --output text

You will see:

shop-api

What it means: The repository exists and is empty.

Example 3: Check immutability

Type this:

aws ecr describe-repositories --repository-names shop-api --region ap-south-1 --query 'repositories[0].imageTagMutability' --output text

You will see:

IMMUTABLE

What it means: Nobody can push a different image with the same tag.

Step 2: Add a lifecycle policy

Why: Old images cost money.

What: Every build pushes a new image. Without cleanup the repository grows forever. A lifecycle policy tells ECR to keep only the newest images and delete the rest automatically.

Example 4: Write the policy file

Type this:

printf '%s\n' '{"rules":[{"rulePriority":1,"description":"keep last 30","selection":{"tagStatus":"any","countType":"imageCountMoreThan","countNumber":30},"action":{"type":"expire"}}]}' > ecr-policy.json && cat ecr-policy.json

You will see:

{"rules":[{"rulePriority":1,..."countNumber":30}...]}

What it means: The file says: keep the 30 newest images.

Example 5: Apply it

Type this:

aws ecr put-lifecycle-policy --repository-name shop-api --lifecycle-policy-text file://ecr-policy.json --region ap-south-1 --query repositoryName --output text

You will see:

shop-api

What it means: Policy attached to the repository.

Example 6: Read it back

Type this:

aws ecr get-lifecycle-policy --repository-name shop-api --region ap-south-1 --query 'lifecyclePolicyText' --output text | head -c 60

You will see:

{"rules":[{"rulePriority":1,"description":"keep last 30"

What it means: Cleanup now runs on its own.

Step 3: Give the Jenkins agent an IAM role

Why: No access keys on the server.

What: If Jenkins runs on EC2, attach an IAM role to the instance with permission to push to ECR. The AWS CLI then gets short-lived credentials automatically, and no access key is ever stored in Jenkins.

Example 7: Attach the managed policy

Type this:

aws iam attach-role-policy --role-name jenkins-agent-role --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser && echo attached

You will see:

attached

What it means: The role can now push and pull images.

Example 8: Check identity on the agent

Type this:

aws sts get-caller-identity --query Arn --output text

You will see:

arn:aws:sts::123456789012:assumed-role/jenkins-agent-role/i-0abc123

What it means: The agent is using the role, not a personal user.

Example 9: No keys on disk

Type this:

ls ~/.aws/credentials 2>/dev/null || echo 'no credentials file'

You will see:

no credentials file

What it means: Good. The role gives temporary credentials.

3. Prepare the Jenkins agent step by step

Step 4: Install Docker on the agent

Why: Jenkins needs Docker to build.

What: The Jenkins agent is the machine where builds run. It needs Docker, the AWS CLI and Trivy. We give the agent a label called docker so the pipeline can choose it.

Example 10: Install Docker

Type this:

sudo dnf install -y docker && sudo systemctl enable --now docker && docker --version

You will see:

Docker version 25.0.5, build 5dc9bcc

What it means: Docker is running on the agent.

Example 11: Install Trivy

Type this:

sudo rpm -ivh https://get.trivy.dev/rpm/trivy_0.56.2_Linux-64bit.rpm && trivy --version | head -1

You will see:

Version: 0.56.2

What it means: The scanner is ready.

Example 12: Check AWS CLI

Type this:

aws --version

You will see:

aws-cli/2.17.40 Python/3.11.9 Linux/6.1

What it means: The CLI is there to log in to ECR.

Step 5: Let the jenkins user use Docker

Why: The most common first failure.

What: Jenkins runs as the user jenkins. That user must be in the docker group, or every docker command fails with permission denied. Remember the docker group is powerful, so use it only on dedicated build agents.

Example 13: See the error

Type this:

sudo -u jenkins docker ps

You will see:

permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

What it means: The jenkins user cannot reach Docker yet.

Example 14: Add to the group

Type this:

sudo usermod -aG docker jenkins && sudo systemctl restart jenkins && echo done

You will see:

done

What it means: Restart so the new group applies.

Example 15: Try again

Type this:

sudo -u jenkins docker ps --format '{{.Names}}' | wc -l

You will see:

What it means: No error now, just zero running containers.

Step 6: Store a Git credential

Why: Jenkins must read the code.

What: Jenkins needs to clone the repository. Store a deploy key or token in Jenkins Credentials with an ID, and refer to the ID in the job. Never type the token inside the Jenkinsfile.

Example 16: Create a deploy key

Type this:

ssh-keygen -t ed25519 -N '' -f jenkins_deploy -C jenkins@shop && ls jenkins_deploy*

You will see:

jenkins_deploy  jenkins_deploy.pub

What it means: Add the .pub file as a read-only deploy key in your Git server.

Example 17: Safe key permissions

Type this:

chmod 0600 jenkins_deploy && ls -l jenkins_deploy

You will see:

-rw------- 1 me me 411 jenkins_deploy

What it means: Only the owner can read the private key.

Example 18: Test the clone

Type this:

GIT_SSH_COMMAND='ssh -i jenkins_deploy' git ls-remote git@git.example.com:shop/shop-api.git HEAD

You will see:

a1b2c3d4e5f6... HEAD

What it means: The key works. Paste it into Jenkins Credentials with ID git-deploy.

4. Write the Jenkinsfile step by step

Step 7: The skeleton

Why: A pipeline in a file.

What: A Jenkinsfile lives in the root of the repository. It has an agent, an environment section with variables, and stages. Start with an empty Build stage and grow it one stage at a time.

Example 19: Create the file

Type this:

printf '%s\n' 'pipeline {' '  agent { label "docker" }' '  stages {' '    stage("Hello") { steps { sh "docker version --format {{.Server.Version}}" } }' '  }' '}' > Jenkinsfile && cat Jenkinsfile

You will see:

pipeline {
  agent { label "docker" }
  stages {
    stage("Hello") { ... }
  }
}

What it means: The smallest pipeline that runs Docker.

Example 20: Commit and push

Type this:

git add Jenkinsfile && git commit -m 'ci: add Jenkinsfile' && git push origin main

You will see:

[main a1b2c3d] ci: add Jenkinsfile

What it means: Jenkins will notice the new file.

Example 21: Read the console

Type this:

Build Now, then Console Output

You will see:

[Pipeline] sh
+ docker version --format 25.0.5
25.0.5
Finished: SUCCESS

What it means: The agent and Docker are connected.

Step 8: Environment variables

Why: Write names once.

What: Put the registry, image name and tag at the top in environment. The tag is the first seven letters of the Git commit, so every image points to exact code.

Example 22: The environment block

Type this:

environment {
  REGISTRY = '123456789012.dkr.ecr.ap-south-1.amazonaws.com'
  IMAGE    = "${REGISTRY}/shop-api"
  TAG      = "${env.GIT_COMMIT.take(7)}"
}

You will see:

(no output, these are definitions)

What it means: Three names used by every stage.

Example 23: Print them

Type this:

stage('Info') { steps { sh 'echo $IMAGE:$TAG' } }

You will see:

+ echo 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api:a1b2c3d

What it means: The full image name with the commit tag.

Example 24: Same in a shell

Type this:

git rev-parse --short=7 HEAD

You will see:

a1b2c3d

What it means: This is the value Jenkins uses as the tag.

Step 9: The Build stage

Why: Make the image.

What: docker build with --pull always downloads the newest patched base image. We also add a label with the commit so anyone can inspect the image later.

Example 25: Build stage

Type this:

stage('Build') {
  steps { sh 'docker build --pull --label git.commit=$TAG -t $IMAGE:$TAG .' }
}

You will see:

Successfully tagged ...shop-api:a1b2c3d

What it means: The image is built on the agent.

Example 26: Check the label

Type this:

docker inspect --format '{{index .Config.Labels "git.commit"}}' 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api:a1b2c3d

You will see:

a1b2c3d

What it means: The commit is stored inside the image.

Example 27: Check the size

Type this:

docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep shop-api

You will see:

...shop-api:a1b2c3d 142MB

What it means: A quick look before scanning.

Step 10: The Scan stage

Why: Stop bad images early.

What: Trivy looks for known security holes in the image. --exit-code 1 makes it fail the build when it finds CRITICAL ones, so a dangerous image never reaches ECR.

Example 28: Scan stage

Type this:

stage('Scan') {
  steps { sh 'trivy image --exit-code 1 --severity CRITICAL --no-progress $IMAGE:$TAG' }
}

You will see:

Total: 0 (CRITICAL: 0)

What it means: Clean image, pipeline continues.

Example 29: A failing scan

Type this:

trivy image --exit-code 1 --severity CRITICAL node:14 ; echo exit=$?

You will see:

Total: 9 (CRITICAL: 9)
exit=1

What it means: An old base image fails the build. Good.

Example 30: Save a report

Type this:

trivy image --format table --output trivy.txt 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api:a1b2c3d && wc -l trivy.txt

You will see:

48 trivy.txt

What it means: Archive this file in Jenkins for auditors.

Step 11: The Push stage

Why: Log in and push.

What: aws ecr get-login-password gives a 12-hour password. We pipe it into docker login with --password-stdin so it never appears in the process list or the log. Then we push.

Example 31: Push stage

Type this:

stage('Push') {
  steps {
    sh 'aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin $REGISTRY'
    sh 'docker push $IMAGE:$TAG'
  }
}

You will see:

Login Succeeded
a1b2c3d: digest: sha256:9f2c... size: 1784

What it means: The image is now in ECR.

Example 32: Check in ECR

Type this:

aws ecr describe-images --repository-name shop-api --region ap-south-1 --query 'imageDetails[].imageTags[]' --output text

You will see:

a1b2c3d

What it means: The commit tag is in the repository.

Example 33: Pushing the same tag again

Type this:

docker push 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api:a1b2c3d

You will see:

tag invalid: The image tag 'a1b2c3d' already exists ... cannot be overwritten because the repository is immutable

What it means: Immutability protects released images.

Step 12: Clean up and trigger automatically

Why: Keep the agent healthy.

What: A post always block removes the local image after every build so the agent disk does not fill. A webhook from the Git server starts the build on every push to main.

Example 34: Post block

Type this:

post {
  always { sh 'docker image rm $IMAGE:$TAG || true' }
}

You will see:

Untagged: ...shop-api:a1b2c3d

What it means: Runs even when the build fails.

Example 35: Agent disk check

Type this:

df -h /var/lib/docker | tail -1

You will see:

/dev/nvme1n1  50G  9.1G  41G  19% /var/lib/docker

What it means: Plenty of space left.

Example 36: Webhook test

Type this:

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://jenkins.example.com/generic-webhook-trigger/invoke?token=example-token

You will see:

What it means: Jenkins accepted the trigger and started a build.

Picture 2: what happens when a stage fails

5. Common mistakes and how the Infosys team avoids them

Mistake one: storing AWS access keys inside Jenkins. The team uses an IAM role on the agent instead.

Mistake two: tagging every image latest. They tag with the commit, and the repository is immutable.

Mistake three: skipping the scan to save one minute. The scan is a required stage and fails on CRITICAL.

Mistake four: forgetting cleanup, so the agent disk fills in a week. The post always block removes images.

Mistake five: using docker login -p with the password on the command line. They always use --password-stdin.

Check yourself

Q1. Why tag the image with the commit id?

So every image maps to exact code, and tags are never reused.

Q2. How does Jenkins log in to ECR without stored keys?

The agent IAM role, then get-login-password piped to docker login --password-stdin.

Q3. What does trivy --exit-code 1 do?

Fails the stage when it finds holes at the chosen severity.

Q4. Why make ECR tags immutable?

So a released tag can never be replaced by a different image.

Friends, create the ECR repository with immutable tags, give the agent an IAM role, let the jenkins user use Docker, then grow the Jenkinsfile stage by stage: build, scan, push, clean. Did you understand? Let us try it once more. Push one small change to main and watch the image appear in ECR with its commit tag.

Ravindra Bagale's Tip

Build the pipeline one stage at a time and run it after each stage. When it breaks, you know exactly which line broke it.

Lab: try at home

Create an immutable ECR repository with scan on push.
Add a lifecycle policy that keeps 30 images.
Attach an IAM role to the Jenkins agent and check it with sts get-caller-identity.
Fix the docker.sock permission error for the jenkins user.
Write the Jenkinsfile with Build, Scan and Push stages.
Break the scan on purpose with an old base image.
Add the post always cleanup and a webhook trigger.

Recap

ECR is a private registry; one repository per app.
Immutable tags plus a lifecycle policy keep it safe and small.
The agent uses an IAM role, never stored keys.
Tag with the commit, scan with Trivy, push with --password-stdin.
post always cleans the agent after every build.