Ravindra BagaleCourses & study guides मराठी

Chapter 82: DevOps Projects Part 12: GitOps Argo CD Multi-Env Vault External Secrets DevSecOps Kyverno

82.2 Project 86: Multi-environment GitOps (dev, stage, prod)

By Last updated:

Friends, one cluster is not enough. Today we keep separate overlays for dev, stage and prod, and promote by merging git — not by kubectl.

What you will build in this project

  1. Create overlays for three environments.
  2. Wire three Argo apps.
  3. Promote stage to prod by merge.
  4. Prove prod cannot sync from the dev path.

1. Start from a real company story

At Razorpay in Bengaluru, a hot fix meant for stage landed in prod because someone retagged by hand.

Rough cost: about ₹19 lakh in refunds and support.

Imran made promotion a pull request between overlay folders.

Today we build that promotion path.

baseshared YAMLoverlaysdev stage prodPRstage→prodargothree apps
Picture 1: base → overlays/dev|stage|prod → three Argo apps → promote by PR

Ravindra Bagale's Tip

Students point prod Argo at the same path as dev. Separate paths and separate AppProjects. Keep that habit.

2. Start and prepare

Step 1: Why multi-env

Why: Blast radius. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Dev can break; prod must promote on purpose'

You will see:

Dev can break; prod must promote on purpose

What it means: Why.

Example 2: a2

Type this:

mkdir -p /tmp/gitops86/{base,overlays/dev,overlays/stage,overlays/prod} && echo 'Today: overlays, three apps, promote PR'

You will see:

Today: overlays, three apps, promote PR

What it means: Scope.

Example 3: a3

Type this:

echo 'Kustomize overlays patch image tags and replicas per env'

You will see:

Kustomize overlays patch image tags and replicas per env

What it means: Kustomize.

Step 2: Base

Why: Shared manifests. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: base. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

printf '%s\n' 'apiVersion: apps/v1' 'kind: Deployment' 'metadata:' '  name: shop-api' 'spec:' '  replicas: 2' '  template:' '    spec:' '      containers:' '      - name: api' '        image: registry.example.com/shop/api:REPLACE' > /tmp/gitops86/base/deploy.yaml && wc -l /tmp/gitops86/base/deploy.yaml

You will see:

11 /tmp/gitops86/base/deploy.yaml

What it means: Base.

Example 5: b2

Type this:

printf '%s\n' 'resources:' '- deploy.yaml' > /tmp/gitops86/base/kustomization.yaml && cat /tmp/gitops86/base/kustomization.yaml

You will see:

resources:

What it means: Kustomize base.

Example 6: b3

Type this:

echo 'Keep secrets out of base; use External Secrets later'

You will see:

Keep secrets out of base; use External Secrets later

What it means: Secrets.

Step 3: Dev overlay

Why: Low replicas. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: dev. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

printf '%s\n' 'resources:' '- ../../base' 'images:' '- name: registry.example.com/shop/api' '  newTag: dev-a1b2c3d' 'replicas:' '- name: shop-api' '  count: 1' > /tmp/gitops86/overlays/dev/kustomization.yaml && cat /tmp/gitops86/overlays/dev/kustomization.yaml

You will see:

newTag: dev-a1b2c3d

What it means: Dev.

Example 8: c2

Type this:

echo 'kubectl kustomize /tmp/gitops86/overlays/dev | head'

You will see:

kubectl kustomize /tmp/gitops86/overlays/dev | head

What it means: Build.

Example 9: c3

Type this:

echo 'Dev Argo path: overlays/dev'

You will see:

Dev Argo path: overlays/dev

What it means: Path.

3. Build the main pieces

Step 4: Stage overlay

Why: Prod-like. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: stage. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

printf '%s\n' 'resources:' '- ../../base' 'images:' '- name: registry.example.com/shop/api' '  newTag: stage-a1b2c3d' 'replicas:' '- name: shop-api' '  count: 2' > /tmp/gitops86/overlays/stage/kustomization.yaml && cat /tmp/gitops86/overlays/stage/kustomization.yaml

You will see:

newTag: stage-a1b2c3d

What it means: Stage.

Example 11: d2

Type this:

echo 'Stage namespace: shop-stage'

You will see:

Stage namespace: shop-stage

What it means: NS.

Example 12: d3

Type this:

echo 'Smoke tests run against stage before prod PR'

You will see:

Smoke tests run against stage before prod PR

What it means: Smoke.

Step 5: Prod overlay

Why: Locked tag. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: prod. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

printf '%s\n' 'resources:' '- ../../base' 'images:' '- name: registry.example.com/shop/api' '  newTag: v1.4.2' 'replicas:' '- name: shop-api' '  count: 4' > /tmp/gitops86/overlays/prod/kustomization.yaml && cat /tmp/gitops86/overlays/prod/kustomization.yaml

You will see:

newTag: v1.4.2

What it means: Prod.

Example 14: e2

Type this:

echo 'Prod uses immutable release tags not branch SHAs when possible'

You will see:

Prod uses immutable release tags not branch SHAs when possible

What it means: Tags.

Example 15: e3

Type this:

printf '%s\n' 'Acceptance: prod Argo app path is overlays/prod only' > /tmp/gitops86/ACCEPT.txt && cat /tmp/gitops86/ACCEPT.txt

You will see:

Acceptance: prod Argo app path is overlays/prod only

What it means: Gate.

Step 6: Three apps

Why: Register. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: apps. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

printf '%s\n' 'shop-api-dev  -> overlays/dev  ns shop-dev' 'shop-api-stage-> overlays/stage ns shop-stage' 'shop-api-prod -> overlays/prod  ns shop-prod' > /tmp/gitops86/APPS.txt && cat /tmp/gitops86/APPS.txt

You will see:

shop-api-prod -> overlays/prod  ns shop-prod

What it means: Apps.

Example 17: f2

Type this:

echo 'Separate AppProjects: devs sync dev; release managers sync prod'

You will see:

Separate AppProjects: devs sync dev; release managers sync prod

What it means: Projects.

Example 18: f3

Type this:

echo 'argocd app create ... --path overlays/prod --dest-namespace shop-prod'

You will see:

argocd app create ... --path overlays/prod --dest-namespace shop-prod

What it means: Create.

4. Automate and verify

Step 7: Promote PR

Why: Stage to prod. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: pr. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

printf '%s\n' '1. Copy stage tag into prod overlay' '2. Open PR: promote shop-api stage-a1b2c3d to prod as v1.4.3' '3. Merge after checks' '4. Prod Argo syncs' > /tmp/gitops86/PROMOTE.txt && cat /tmp/gitops86/PROMOTE.txt

You will see:

4. Prod Argo syncs

What it means: Promote.

Example 20: g2

Type this:

echo 'CI can open the promote PR after stage smoke passes'

You will see:

CI can open the promote PR after stage smoke passes

What it means: CI.

Example 21: g3

Type this:

echo 'Never kubectl set image in prod when GitOps owns the app'

You will see:

Never kubectl set image in prod when GitOps owns the app

What it means: Rule.

Step 8: Protect prod

Why: Branch rules. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: protect. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

printf '%s\n' 'Require 2 reviews on overlays/prod' 'Block force-push on main' 'CODEOWNERS: platform-team owns overlays/prod' > /tmp/gitops86/PROTECT.txt && cat /tmp/gitops86/PROTECT.txt

You will see:

CODEOWNERS: platform-team owns overlays/prod

What it means: Protect.

Example 23: h2

Type this:

echo 'Argo RBAC: deny sync on prod for developer roles'

You will see:

Argo RBAC: deny sync on prod for developer roles

What it means: Argo RBAC.

Example 24: h3

Type this:

echo 'Optional: manual sync only for prod (no autosync)'

You will see:

Optional: manual sync only for prod (no autosync)

What it means: Manual.

Step 9: Drift check

Why: Self-heal careful. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: drift. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

echo 'Enable self-heal on dev and stage; consider manual on prod'

You will see:

Enable self-heal on dev and stage; consider manual on prod

What it means: Heal.

Example 26: i2

Type this:

echo 'argocd app diff shop-api-prod before sync in a change window'

You will see:

argocd app diff shop-api-prod before sync in a change window

What it means: Diff.

Example 27: i3

Type this:

echo 'Alert on OutOfSync lasting more than 10 minutes in prod'

You will see:

Alert on OutOfSync lasting more than 10 minutes in prod

What it means: Alert.

5. Harden and accept

Step 10: Story Razorpay

Why: Hand retag. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

printf '%s\n' 'At Razorpay in Bengaluru, a stage hotfix was retagged into prod by hand.' 'Imran made promotion a pull request between overlay folders.' > /tmp/gitops86/STORY.txt && wc -l /tmp/gitops86/STORY.txt

You will see:

2 /tmp/gitops86/STORY.txt

What it means: Story.

Example 29: j2

Type this:

echo 'Rough cost: about ₹19 lakh in refunds and support'

You will see:

Rough cost: about ₹19 lakh in refunds and support

What it means: Cost.

Example 30: j3

Type this:

echo 'Imran: PASS'

You will see:

Imran: PASS

What it means: Sign-off.

Step 11: Proof

Why: Tree. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: find. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

find /tmp/gitops86 -type f | sort

You will see:

/tmp/gitops86/ACCEPT.txt\n/tmp/gitops86/APPS.txt\n/tmp/gitops86/PROMOTE.txt\n/tmp/gitops86/PROTECT.txt\n/tmp/gitops86/STORY.txt\n/tmp/gitops86/base/deploy.yaml\n/tmp/gitops86/base/kustomization.yaml\n/tmp/gitops86/overlays/dev/kustomization.yaml\n/tmp/gitops86/overlays/prod/kustomization.yaml\n/tmp/gitops86/overlays/stage/kustomization.yaml

What it means: Tree.

Example 32: k2

Type this:

grep newTag /tmp/gitops86/overlays/prod/kustomization.yaml

You will see:

newTag: v1.4.2

What it means: Prod tag.

Example 33: k3

Type this:

echo 'Next: Vault secrets project 87'

You will see:

Next: Vault secrets project 87

What it means: Bridge.

Step 12: Acceptance

Why: Promote by PR only. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Three overlays exist'

You will see:

Three overlays exist

What it means: Overlays.

Example 35: z2

Type this:

echo 'Three Argo apps'

You will see:

Three Argo apps

What it means: Apps.

Example 36: z3

Type this:

echo 'Prod path isolated'

You will see:

Prod path isolated

What it means: Isolated.

Check yourself

Q1. Who fixed Razorpay retag?

Imran

Q2. Promote how?

pull request

Q3. Overlay count?

three

Q4. Prod shares dev path?

no

Q5. Tool still?

Argo CD

Q6. Hot-fix hand retag?

no

Friends, you promoted with a pull request. Did you understand? Let us try it once more. Open a PR that only changes the prod overlay image tag and sync prod.

Lab: try at home

  1. Overlays.
  2. Three Argo apps.
  3. Promote PR.
  4. Prod path isolated.

Recap

  • Base plus overlays.
  • One app per env.
  • Promote is a merge.
  • Prod path locked.
  • No hand retag.