Chapter 59: Kubernetes Pods in Depth: Architecture, YAML Anatomy and Pod Types

59.6 Sidecar containers: log shippers, native sidecars and proxies

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we run the Ola ride-booking service. Each pod writes logs to a file. When a pod dies, its logs die with it, and nobody can find out why a ride failed. A second problem: the security team wants every call between services encrypted, but there are 200 services in 6 languages. Changing all that code would take a year. The answer to both is a sidecar: a helper container that runs beside the app for its whole life. One sidecar ships logs to a central place. Another sidecar acts as a proxy and adds TLS, retries and metrics without code changes. Since Kubernetes 1.29, sidecars have native support: an init container with restartPolicy: Always. It starts before the app and stops after it. In this lesson we build a log-shipping sidecar, a native sidecar, and a proxy sidecar, then break them with a sidecar that keeps a Job running forever and a sidecar that starts too late.

What: every term in plain English

• Sidecar — A helper container that runs for the whole life of the pod, next to the main app.
• Native sidecar — An entry in initContainers with restartPolicy: Always. Kubernetes starts it first, keeps it running, and stops it last.
• Classic sidecar — A second entry in containers. It works, but there is no start or stop order.
• Log shipper — A sidecar such as Fluent Bit that reads log files and sends them to Elasticsearch, Loki or CloudWatch.
• Proxy sidecar — A sidecar such as Envoy that handles network traffic for the app. Istio injects it automatically.
• Config reloader — A sidecar that watches a ConfigMap and tells the app to reload.
• Job completion problem — A classic sidecar never exits, so a Job pod never completes. Native sidecars fix this.
• SidecarContainers feature — Beta and on by default in Kubernetes 1.29, stable in 1.33.
• Service mesh — A set of proxy sidecars plus a control plane, like Istio or Linkerd.

How: step by step

1. Choose the helper. Logs: Fluent Bit. Proxy: Envoy or nginx. Reload: a small watcher.
2. Share what it needs. A volume for log files, or localhost for a proxy.
3. Prefer native sidecars. Put the sidecar in initContainers with restartPolicy: Always, on Kubernetes 1.29 or newer.
4. Give it resources. Small requests, like 20m CPU and 32Mi memory, and a memory limit.
5. Add probes. A native sidecar may have a readiness or startup probe.
6. Check kubectl version. kubectl version should show server v1.29 or newer for native sidecars.
7. Test shutdown. Delete the pod and see that the app stops before the sidecar.

Example 1: Classic log-shipping sidecar with Fluent Bit

Goal. Classic log-shipping sidecar with Fluent Bit.

fluent-bit-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fb-config
  namespace: shop
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        2
        Log_Level    info
    [INPUT]
        Name         tail
        Path         /var/log/app/*.log
        Tag          rides
    [FILTER]
        Name         modify
        Match        rides
        Add          service ride-api
    [OUTPUT]
        Name         stdout
        Match        rides
        Format       json_lines

ride-api.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ride-api
  namespace: shop
spec:
  volumes:
    - name: logs
      emptyDir: {}
    - name: fb
      configMap:
        name: fb-config
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - while true; do
            echo "ride booked from Hinjewadi to Baner" >> /var/log/app/ride.log;
            sleep 4;
          done
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
    - name: log-shipper
      image: fluent/fluent-bit:3.1
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
          readOnly: true
        - name: fb
          mountPath: /fluent-bit/etc/
      resources:
        requests: {cpu: 20m, memory: 32Mi}
        limits: {memory: 64Mi}

What each line does

• [INPUT] tail — Fluent Bit reads every .log file in the shared folder.
• [FILTER] modify — Adds a field service=ride-api to every record.
• [OUTPUT] stdout — For the lab we print. In production change it to es, loki or cloudwatch_logs.
• log-shipper in containers — Classic style: it is just a second container.
• mountPath /fluent-bit/etc/ — The image reads fluent-bit.conf from here.

Green output

$ kubectl apply -f fluent-bit-config.yaml -f ride-api.yaml
$ kubectl get pod ride-api -n shop
NAME       READY   STATUS    RESTARTS   AGE
ride-api   2/2     Running   0          12s
$ kubectl logs -n shop ride-api -c log-shipper --tail=1
{"date":1792902615.1,"log":"ride booked from Hinjewadi to Baner","service":"ride-api"}

Expected result. The app only writes a file. The sidecar turns it into structured records and sends them on.

Example 2: Native sidecar: starts first, stops last, and lets a Job finish

Goal. Native sidecar: starts first, stops last, and lets a Job finish.

report-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: daily-report
  namespace: shop
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      volumes:
        - name: logs
          emptyDir: {}
      initContainers:
        - name: log-shipper
          image: busybox:1.36
          restartPolicy: Always
          command: ["sh", "-c", "touch /logs/job.log; tail -F /logs/job.log"]
          volumeMounts:
            - name: logs
              mountPath: /logs
      containers:
        - name: report
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - for c in Pune Mumbai Nagpur; do
                echo "report done for $c" >> /logs/job.log; sleep 2;
              done
          volumeMounts:
            - name: logs
              mountPath: /logs

What each line does

• initContainers + restartPolicy: Always — This one line makes it a native sidecar.
• tail -F — Runs forever, like a real shipper.
• report container — Does the real work and exits 0.
• Job restartPolicy: Never — The pod-level policy. The sidecar's own Always overrides it only for itself.
• Why it matters — With a classic sidecar this Job would stay Running forever, because tail never exits.

Green output

$ kubectl apply -f report-job.yaml
$ kubectl get pods -n shop -l job-name=daily-report -w
daily-report-x7k2p   0/2   Init:0/1          0   1s
daily-report-x7k2p   1/2   PodInitializing   0   2s
daily-report-x7k2p   2/2   Running           0   3s
daily-report-x7k2p   0/2   Completed         0   10s
$ kubectl get job daily-report -n shop
NAME           STATUS     COMPLETIONS   DURATION
daily-report   Complete   1/1           10s
$ kubectl logs -n shop job/daily-report -c log-shipper
report done for Pune
report done for Mumbai
report done for Nagpur

Expected result. When the report container finished, Kubernetes stopped the sidecar for us and the Job became Complete.

Example 3: Proxy sidecar: add HTTPS-style front door and access logs without touching the app

Goal. Proxy sidecar: add HTTPS-style front door and access logs without touching the app.

proxy-pod.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-conf
  namespace: shop
data:
  default.conf: |
    server {
      listen 8080;
      location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header X-Request-Id $request_id;
        add_header X-Served-By sidecar-proxy;
      }
      location /healthz { return 200 "ok"; }
    }
---
apiVersion: v1
kind: Pod
metadata:
  name: fare-api
  namespace: shop
  labels:
    app: fare-api
spec:
  volumes:
    - name: conf
      configMap:
        name: proxy-conf
  initContainers:
    - name: proxy
      image: nginx:1.27
      restartPolicy: Always
      ports:
        - containerPort: 8080
      readinessProbe:
        httpGet: {path: /healthz, port: 8080}
      volumeMounts:
        - name: conf
          mountPath: /etc/nginx/conf.d
  containers:
    - name: app
      image: python:3.12-slim
      command: ["python", "-m", "http.server", "5000", "--bind", "127.0.0.1"]

What each line does

• app binds 127.0.0.1:5000 — The app is only reachable from inside the pod.
• proxy listens 8080 — The only port the outside world uses.
• proxy_set_header X-Request-Id — Adds a tracing id to every request, a typical mesh feature.
• readinessProbe on the sidecar — The pod is Ready only when the proxy is ready.
• native sidecar — The proxy is up before the app gets any traffic, and still up while the app drains.

Green output

$ kubectl apply -f proxy-pod.yaml
$ kubectl port-forward -n shop pod/fare-api 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
$ curl -sI http://localhost:8080/ | grep -i served
X-Served-By: sidecar-proxy
$ kubectl logs -n shop fare-api -c proxy --tail=1
127.0.0.1 - - [25/Oct/2026:06:30:02 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.5.0"

Expected result. Every request goes through the proxy. This is exactly how Istio's Envoy sidecar works, just automated.

Every command, step by step (kind lab with kubectl)

1. Check server version

   kubectl version
   

   What it does: Native sidecars need server v1.29 or newer. Recent kind images are fine.

   Client Version: v1.31.2
   Server Version: v1.31.2
   

2. Apply the classic sidecar

   kubectl apply -f fluent-bit-config.yaml -f ride-api.yaml
   kubectl get pod ride-api -n shop
   

   What it does: Two containers in containers.

   ride-api   2/2     Running   0          9s
   

3. Read shipped logs

   kubectl logs -n shop ride-api -c log-shipper -f
   

   What it does: Ctrl+C to stop following.

   {"log":"ride booked from Hinjewadi to Baner","service":"ride-api"}
   

4. Try a classic sidecar in a Job

   printf '%s\n' \
     'apiVersion: batch/v1' \
     'kind: Job' \
     'metadata:' \
     '  name: daily-report-classic' \
     '  namespace: shop' \
     'spec:' \
     '  backoffLimit: 1' \
     '  template:' \
     '    spec:' \
     '      restartPolicy: Never' \
     '      volumes:' \
     '        - name: logs' \
     '          emptyDir: {}' \
     '      containers:' \
     '        - name: log-shipper' \
     '          image: busybox:1.36' \
     '          command: ["sh", "-c", "touch /logs/job.log; tail -F /logs/job.log"]' \
     '          volumeMounts:' \
     '            - name: logs' \
     '              mountPath: /logs' \
     '        - name: report' \
     '          image: busybox:1.36' \
     '          command: ["sh", "-c"]' \
     '          args:' \
     '            - for c in Pune Mumbai Nagpur; do' \
     '                echo "report done for $c" >> /logs/job.log; sleep 2;' \
     '              done' \
     '          volumeMounts:' \
     '            - name: logs' \
     '              mountPath: /logs' \
     > report-job-classic.yaml
   kubectl apply -f report-job-classic.yaml
   kubectl get job daily-report-classic -n shop
   

   What it does: The full file is written by the first command. It is report-job.yaml with one change: log-shipper moved from initContainers into containers, and it has no restartPolicy: Always. The report container exits 0, but tail -F never exits, so the pod never finishes.

   NAME                   STATUS    COMPLETIONS   DURATION
   daily-report-classic   Running   0/1           5m
   

5. Delete the stuck Job

   kubectl delete job daily-report-classic -n shop
   

   What it does: Remember this pain; it is why native sidecars exist.

   job.batch "daily-report-classic" deleted
   

6. Run the native version

   kubectl apply -f report-job.yaml
   kubectl wait -n shop --for=condition=complete job/daily-report --timeout=60s
   

   What it does: Completes in about 10 seconds.

   job.batch/daily-report condition met
   

7. Spot a native sidecar

   kubectl get pod fare-api -n shop -o jsonpath='{.spec.initContainers[0].restartPolicy}{"\n"}'
   

   What it does: Always means sidecar; empty means normal init.

   Always
   

8. Watch shutdown order

   kubectl delete pod fare-api -n shop
   kubectl get events -n shop --field-selector involvedObject.name=fare-api | tail -3
   

   What it does: The app is killed first, then the proxy.

   Normal   Killing   pod/fare-api   Stopping container app
   Normal   Killing   pod/fare-api   Stopping container proxy
   

9. Clean up

   kubectl delete pod ride-api -n shop
   kubectl delete job daily-report -n shop
   

   What it does: Keep the namespace.

   pod "ride-api" deleted
   

When it is red 1

NAME                   STATUS    COMPLETIONS
daily-report-classic   Running   0/1

Cause. A classic sidecar in a Job never exits, so the Job never completes.

Fix, step by step

1. Move the sidecar into initContainers with restartPolicy: Always.
2. On old clusters, make the app touch a file at the end and have the sidecar exit when it sees it.

When it is red 2

spec.initContainers[0].restartPolicy: Unsupported value: "Always"

Cause. The cluster is older than 1.28 or the SidecarContainers feature is off.

Fix, step by step

1. Check kubectl version.
2. Upgrade the cluster, or use a classic sidecar.

When it is red 3

app: connect: connection refused 127.0.0.1:15001

Cause. With a classic proxy sidecar, the app started before the proxy was ready.

Fix, step by step

1. Make the proxy a native sidecar with a readiness or startup probe.
2. Native sidecars must be ready before app containers start.

When it is red 4

log-shipper  OOMKilled  Exit Code: 137

Cause. The shipper buffers too much data in memory.

Fix, step by step

1. Raise its memory limit a little.
2. Set Mem_Buf_Limit in the Fluent Bit INPUT.
3. Use storage.type filesystem for buffering.

Practice tasks

kubectl logs -n shop ride-api -c log-shipper -f

kubectl version

kubectl get pod fare-api -n shop -o jsonpath='{.spec.initContainers[*].restartPolicy}'

kubectl get job -n shop -w

initContainers:
  - name: log-shipper
    restartPolicy: Always

Add city Pune

add_header X-Team rides;

startupProbe:
  httpGet: {path: /healthz, port: 8080}

[OUTPUT]
    Name loki
    Host loki.monitoring
    Port 3100

shareProcessNamespace: true

Recap

• A sidecar runs beside the app for the whole pod life.
• Native sidecars are initContainers with restartPolicy: Always (1.29+).
• Native sidecars start first, stop last, and let Jobs complete.
• Common sidecars: Fluent Bit for logs, Envoy or nginx as proxy, config reloaders.
• Give sidecars small resources and probes.
• Lesson 59.4 showed adapter and ambassador; 59.7 covers static pods.

FAQ

Q. Sidecar or DaemonSet for logs?

A DaemonSet (one shipper per node, lesson 60.5) is cheaper for stdout logs. Use a sidecar when the app writes files or needs a special config.

Q. Do I need a service mesh?

Not for small setups. Start with plain Services and add Istio or Linkerd when you need mTLS and traffic control across many services.

Q. Can a native sidecar have a liveness probe?

Yes. Unlike normal init containers, native sidecars support probes.

Q. Does a sidecar slow down my app?

A proxy adds a little latency, usually about a millisecond. Log shippers add a little CPU.

Q. What does Istio injection do?

A webhook adds the Envoy sidecar to every pod in labelled namespaces.

Q. Can I kubectl exec into a sidecar?

Yes, with -c sidecar-name.

Interview questions

1. What is a sidecar container?

   Model answer: A helper container that runs alongside the main container for the whole pod life, adding logging, proxying or config reload without changing the app.

2. What are native sidecars?

   Model answer: Since 1.29, init containers with restartPolicy: Always. They start before app containers, can have probes, and terminate after them, which also fixes Jobs.

3. Why did sidecars break Jobs before?

   Model answer: The pod completes only when all containers exit; a long-running sidecar kept the Job running forever.

4. Sidecar versus DaemonSet for log collection?

   Model answer: DaemonSet is one agent per node for stdout logs, cheaper; sidecar is per pod, used for file logs or per-app config.

5. How does a service mesh use sidecars?

   Model answer: It injects a proxy into each pod that handles mTLS, retries, routing and metrics, controlled centrally.

Samajla ka?