Chapter 59: Kubernetes Pods in Depth: Architecture, YAML Anatomy and Pod Types

59.5 Init containers: wait for the database, run migrations, seed data

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we deploy the Paytm wallet API. The API pod starts in 2 seconds, but the MySQL pod needs 30 seconds. The API tries to connect, fails, crashes and restarts. After five crashes it is in CrashLoopBackOff, and the on-call engineer gets paged at 2 AM. Another team runs database migrations by hand before every release. One Friday someone forgets, and the new code reads a column that does not exist. Init containers fix both. They run before the app, one at a time, and each must finish successfully. Only then does the app container start. In this lesson we write three init container pods: wait for a database, run a schema migration, and download config before start. Then we break them: an init that never ends, an init that fails, and a migration that is not safe to run twice.

What: every term in plain English

• Init container — A container in spec.initContainers that runs to completion before the app containers start.
• Order — Init containers run one by one, in the order written. The next starts only after the previous exits with code 0.
• Init:0/2 — Pod status while the first of two init containers is running.
• Init:Error — An init container exited with a non-zero code.
• Init:CrashLoopBackOff — An init container keeps failing and the kubelet waits longer between tries.
• Idempotent — Safe to run many times with the same result. Init containers can rerun, so their work must be idempotent.
• Migration — A script that changes the database schema, such as adding a table or column.
• Seeding — Loading starting data, like a list of cities, into a fresh database.
• nc -z — netcat check: is anything listening on this host and port? Exit 0 if yes.
• Shared emptyDir — Init containers can write files into a volume that the app reads later.

How: step by step

1. List what must be true before start. For example: DB is reachable, schema is current, config file exists.
2. Write one init container per job. Small and clear: wait-db, migrate, fetch-config.
3. Make each one idempotent. CREATE TABLE IF NOT EXISTS, not plain CREATE TABLE.
4. Add a timeout. Use a loop with a counter or activeDeadlineSeconds so a pod does not wait forever.
5. Give resources. Init containers need requests and limits too.
6. Apply and watch. kubectl get pod -w shows Init:0/2, Init:1/2, PodInitializing, Running.
7. Read init logs. kubectl logs pod -c init-name.

Example 1: Wait until MySQL accepts connections

Goal. Wait until MySQL accepts connections.

mysql.yaml

apiVersion: v1
kind: Secret
metadata:
  name: wallet-db
  namespace: shop
type: Opaque
stringData:
  MYSQL_ROOT_PASSWORD: "change-me-root"
  MYSQL_DATABASE: "wallet"
---
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  namespace: shop
  labels:
    app: mysql
spec:
  containers:
    - name: mysql
      image: mysql:8.4
      envFrom:
        - secretRef:
            name: wallet-db
      ports:
        - containerPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: shop
spec:
  selector:
    app: mysql
  ports:
    - port: 3306

wallet-api.yaml

apiVersion: v1
kind: Pod
metadata:
  name: wallet-api
  namespace: shop
spec:
  initContainers:
    - name: wait-db
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - for i in $(seq 1 60); do
            nc -z mysql.shop.svc.cluster.local 3306 && echo "DB is up" && exit 0;
            echo "waiting for mysql ($i)"; sleep 2;
          done;
          echo "DB not up after 120s"; exit 1
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}
  containers:
    - name: api
      image: nginx:1.27
      ports:
        - containerPort: 80

What each line does

• Secret stringData — Placeholder password only. Never put a real password in a file you commit.
• Service mysql — Gives the database the DNS name mysql.shop.svc.cluster.local.
• spec.initContainers — A separate list from containers. These run first.
• for i in $(seq 1 60) — Try up to 60 times, so the pod does not wait forever.
• nc -z host 3306 — Exit 0 as soon as the port is open.
• exit 1 at the end — After 120 seconds, fail loudly so the problem is visible.

Green output

$ kubectl apply -f wallet-api.yaml
pod/wallet-api created
$ kubectl get pod wallet-api -n shop
NAME         READY   STATUS     RESTARTS   AGE
wallet-api   0/1     Init:0/1   0          10s
$ kubectl logs -n shop wallet-api -c wait-db --tail=2
waiting for mysql (4)
waiting for mysql (5)
$ kubectl apply -f mysql.yaml
$ kubectl get pod wallet-api -n shop -w
wallet-api   0/1     Init:0/1          0          40s
wallet-api   0/1     PodInitializing   0          58s
wallet-api   1/1     Running           0          59s

Expected result. The API never crashed. It simply waited in Init until MySQL was reachable, and then started once.

Example 2: Run an idempotent migration and seed before the app

Goal. Run an idempotent migration and seed before the app.

migrate-pod.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: wallet-schema
  namespace: shop
data:
  001_schema.sql: |
    CREATE TABLE IF NOT EXISTS wallets (
      id INT PRIMARY KEY AUTO_INCREMENT,
      user_name VARCHAR(50) NOT NULL,
      balance DECIMAL(10,2) NOT NULL DEFAULT 0
    );
  002_seed.sql: |
    INSERT IGNORE INTO wallets (id, user_name, balance) VALUES
      (1, 'asha', 500.00),
      (2, 'ravi', 1200.00);
---
apiVersion: v1
kind: Pod
metadata:
  name: wallet-api-v2
  namespace: shop
spec:
  volumes:
    - name: sql
      configMap:
        name: wallet-schema
  initContainers:
    - name: wait-db
      image: busybox:1.36
      command: ["sh", "-c", "until nc -z mysql 3306; do sleep 2; done"]
    - name: migrate
      image: mysql:8.4
      envFrom:
        - secretRef:
            name: wallet-db
      command: ["sh", "-c"]
      args:
        - for f in /sql/*.sql; do
            echo "applying $f";
            mysql -h mysql -uroot -p"$MYSQL_ROOT_PASSWORD" wallet -e "source $f" || exit 1;
          done
      volumeMounts:
        - name: sql
          mountPath: /sql
  containers:
    - name: api
      image: nginx:1.27

What each line does

• ConfigMap with .sql keys — Each key becomes a file when mounted as a volume.
• CREATE TABLE IF NOT EXISTS — Idempotent: safe if the pod restarts and the init runs again.
• INSERT IGNORE — Skips rows that already exist, so seeding twice does not fail.
• Two init containers — wait-db first, then migrate. Order is the list order.
• short host name mysql — Inside the same namespace, the Service name alone works.
• || exit 1 — Any failed file stops the init, so the app never starts on a half-migrated database.

Green output

$ kubectl apply -f migrate-pod.yaml
$ kubectl get pod wallet-api-v2 -n shop -w
wallet-api-v2   0/1   Init:0/2          0   2s
wallet-api-v2   0/1   Init:1/2          0   4s
wallet-api-v2   0/1   PodInitializing   0   9s
wallet-api-v2   1/1   Running           0   10s
$ kubectl logs -n shop wallet-api-v2 -c migrate
applying /sql/001_schema.sql
applying /sql/002_seed.sql
$ kubectl exec -n shop mysql -- sh -c 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" wallet -e "SELECT * FROM wallets"'
id  user_name  balance
1   asha       500.00
2   ravi       1200.00

Expected result. Status moved Init:0/2, Init:1/2, PodInitializing, Running. Delete and recreate the pod: the migration runs again and nothing breaks.

Example 3: Fetch config into a shared folder before start

Goal. Fetch config into a shared folder before start.

fetch-config.yaml

apiVersion: v1
kind: Pod
metadata:
  name: menu-web
  namespace: shop
spec:
  volumes:
    - name: site
      emptyDir: {}
  initContainers:
    - name: fetch
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - wget -qO /site/index.html https://example.com/ &&
          echo "built at $(date)" >> /site/index.html
      volumeMounts:
        - name: site
          mountPath: /site
  containers:
    - name: web
      image: nginx:1.27
      volumeMounts:
        - name: site
          mountPath: /usr/share/nginx/html
          readOnly: true

What each line does

• emptyDir site — The init writes here; the app reads here.
• wget -qO /site/index.html — Downloads a file into the shared folder. In real life this could be a config file from S3 or a template.
• && — Only add the build line if the download worked.
• readOnly in the app — The app cannot change what the init prepared.

Green output

$ kubectl apply -f fetch-config.yaml
$ kubectl exec -n shop menu-web -- tail -1 /usr/share/nginx/html/index.html
built at Sun Oct 25 06:20:14 UTC 2026

Expected result. The init container is gone after it finishes, but its file stays in the emptyDir for the app.

Every command, step by step (kind lab with kubectl)

1. Check the cluster

   kubectl get nodes
   kubectl get ns shop
   

   What it does: Reuse the ch59 kind cluster.

   NAME                 STATUS   ROLES           AGE
   ch59-control-plane   Ready    control-plane   2h
   ch59-worker          Ready    none            2h
   

2. Start the API first

   kubectl apply -f wallet-api.yaml
   kubectl get pod wallet-api -n shop
   

   What it does: No database yet, so it waits in Init.

   wallet-api   0/1     Init:0/1   0          5s
   

3. Describe the init

   kubectl describe pod wallet-api -n shop | sed -n '/Init Containers/,/Containers:/p'
   

   What it does: Shows State: Running for wait-db.

   Init Containers:
     wait-db:
       State:          Running
   

4. Start MySQL

   kubectl apply -f mysql.yaml
   kubectl wait -n shop --for=condition=Ready pod/mysql --timeout=120s
   

   What it does: Wait until MySQL is ready.

   pod/mysql condition met
   

5. Watch the API start

   kubectl get pod wallet-api -n shop -w
   

   What it does: Ctrl+C after Running.

   wallet-api   1/1     Running   0          61s
   

6. Run migrations

   kubectl apply -f migrate-pod.yaml
   kubectl logs -n shop wallet-api-v2 -c migrate
   

   What it does: Reads each SQL file.

   applying /sql/001_schema.sql
   applying /sql/002_seed.sql
   

7. Rerun to prove idempotent

   kubectl delete pod wallet-api-v2 -n shop
   kubectl apply -f migrate-pod.yaml
   kubectl get pod wallet-api-v2 -n shop
   

   What it does: Second run must also succeed.

   wallet-api-v2   1/1     Running   0          12s
   

8. See init exit codes

   kubectl get pod wallet-api-v2 -n shop -o jsonpath='{range .status.initContainerStatuses[*]}{.name}={.state.terminated.exitCode} {end}{"\n"}'
   

   What it does: 0 means success.

   wait-db=0 migrate=0
   

9. Clean up

   kubectl delete pod -n shop wallet-api wallet-api-v2 menu-web
   

   What it does: Keep mysql for later practice.

   pod "wallet-api" deleted
   

When it is red 1

NAME         READY   STATUS     RESTARTS   AGE
wallet-api   0/1     Init:0/1   0          15m

Cause. The init never finishes. Usually the Service name or port is wrong, or the DB is down.

Fix, step by step

1. kubectl logs pod -c wait-db.
2. kubectl get svc,endpoints -n shop mysql.
3. Fix the host name or start the DB.
4. Add a retry limit so it fails instead of waiting forever.

When it is red 2

wallet-api-v2   0/1   Init:CrashLoopBackOff   4   3m
ERROR 1045 (28000): Access denied for user 'root'

Cause. The migrate container cannot log in.

Fix, step by step

1. Check the Secret keys with kubectl get secret wallet-db -o yaml (values are base64).
2. Make sure envFrom secretRef name matches.
3. Recreate the pod.

When it is red 3

ERROR 1050 (42S01): Table 'wallets' already exists

Cause. The migration is not idempotent, so it fails when the pod restarts.

Fix, step by step

1. Use CREATE TABLE IF NOT EXISTS.
2. Or use a migration tool like Flyway or Liquibase that tracks applied versions.
3. For big releases, run migrations as a Job (lesson 60.6) instead.

When it is red 4

Init:Error
wget: bad address 'example.com'

Cause. The init cannot resolve DNS or reach the internet.

Fix, step by step

1. kubectl exec into a test pod and run nslookup kubernetes.default.
2. Check CoreDNS pods in kube-system (lesson 62.6).
3. Check egress NetworkPolicy.

Practice tasks

kubectl get pod wallet-api -n shop -w

kubectl logs -n shop wallet-api -c wait-db

kubectl get pod wallet-api-v2 -n shop -w

CREATE INDEX idx_user ON wallets (user_name);

for i in $(seq 1 15); do ... done; exit 1

spec:
  activeDeadlineSeconds: 300

nc -z mysql 3307

kubectl get pod wallet-api-v2 -n shop -o jsonpath='{.status.initContainerStatuses[*].state.terminated.exitCode}'

kubectl create deployment ... then edit

args: ["-url=jdbc:mysql://mysql:3306/wallet", "-user=root", "-password=change-me-root", "migrate"]

Recap

• Init containers run before the app, one by one, and each must exit 0.
• Use them to wait for dependencies, migrate schemas and prepare files.
• Status shows Init:N/M, then PodInitializing, then Running.
• Always add retry limits and make the work idempotent.
• Use kubectl logs -c to debug init containers.
• Lesson 59.6 covers sidecars, which run beside the app instead of before it.

FAQ

Q. Can init containers have probes?

No liveness or readiness probes, because they are meant to finish. Use a timeout loop instead.

Q. Do init containers rerun on app restart?

Not when only the app container restarts. They rerun when the whole pod is recreated.

Q. Should migrations run in init containers?

Fine for small apps. With many replicas, two pods may migrate at once, so many teams use a Job or a tool with locking like Flyway.

Q. Can an init container use a different image?

Yes, that is common: busybox for waiting, mysql or flyway for migrations.

Q. Do resources of init containers add up?

The pod's effective request is the bigger of the largest init request and the sum of app requests.

Q. Is waiting in init better than retry in code?

Both are good. Retry in code is still needed, because the DB can go down after start.

Interview questions

1. What are init containers and when do you use them?

   Model answer: Containers that run to completion before the app starts, in order. Used for waiting on dependencies, migrations, fetching config, and setting permissions.

2. How are init containers different from sidecars?

   Model answer: Init containers finish before the app starts; sidecars run alongside the app for its whole life.

3. A pod is stuck at Init:0/1. How do you debug?

   Model answer: kubectl describe pod, kubectl logs -c the init container, then check the Service, endpoints, DNS and the dependency itself.

4. How do you make migrations safe with many replicas?

   Model answer: Use idempotent SQL, a migration tool with a lock table, or a separate Job before the rollout.

5. What happens if an init container fails?

   Model answer: The kubelet restarts it according to restartPolicy; the app containers never start until it succeeds.

Samajla ka?