53.12 Wait-for-DB and retry patterns beyond depends_on
Friends, depends_on with service_healthy makes the API wait for the database at start. But what happens when the database restarts at 3 am while the API is running? depends_on does nothing then. Today we learn the patterns that keep apps alive: good healthchecks, wait scripts, retries with backoff in code, connection pools, and restart policies.
मित्रांनो, depends_on आणि service_healthy मुळे start वेळी API database साठी थांबतो. पण API चालू असताना रात्री 3 वाजता database restart झाला तर? तेव्हा depends_on काहीच करत नाही. आज app जिवंत ठेवणारे patterns शिकू: चांगले healthchecks, wait scripts, code मध्ये backoff सह retry, connection pool आणि restart policy.
दोस्तों, depends_on और service_healthy से start के समय API database का इंतज़ार करता है. पर API चलते समय रात 3 बजे database restart हो जाए तो? तब depends_on कुछ नहीं करता. आज app को ज़िंदा रखने वाले patterns सीखेंगे: अच्छे healthchecks, wait scripts, code में backoff के साथ retry, connection pool और restart policy.
1. Start from a story
At PhonePe, a reports service ran next to MySQL in Compose. On a normal day it started fine because of depends_on with service_healthy.
One night the MySQL container was recreated during a routine image update. It was down for 40 seconds. The reports service lost its connections and every request after that failed with Connection refused, even after MySQL came back.
Nobody had told the app to reconnect. It held dead connections until someone restarted it in the morning.
Meera, the backend lead, added three things: a retry loop with backoff when connecting, a connection pool that tests connections before use, and a healthcheck on the app itself so Docker restarts it if it gets stuck. Today we build all three.
2. Why depends_on is not enough
Step 1: depends_on only acts at start
Why: It does not watch later.
What: Compose orders start-up. Once everything is running, it does not stop or pause the API when the database goes away.
Example 1: Start
Type this:
docker compose up -d --wait && docker compose ps --format '{{.Service}} {{.Status}}'
You will see:
api Up 20 seconds (healthy)
db Up 26 seconds (healthy)
What it means: Good start.
Example 2: Kill db
Type this:
docker compose restart db && docker compose logs --tail 2 api
You will see:
api-1 | Error: Connection terminated unexpectedly
api-1 | Error: connect ECONNREFUSED 172.20.0.2:5432
What it means: API sees the outage.
Example 3: After db is back
Type this:
sleep 15 && curl -s localhost:3000/orders
You will see:
{"error":"connection closed"}
What it means: A naive app never recovers.
Step 2: Healthy is not the same as ready
Why: Check the right thing.
What: pg_isready says the server accepts connections. Your app may also need the database, user and tables to exist.
Example 4: Basic check
Type this:
docker compose exec db pg_isready -U shop -d shop
You will see:
/var/run/postgresql:5432 - accepting connections
What it means: Server is up.
Example 5: Stronger check
Type this:
printf ' healthcheck:\n test: ["CMD-SHELL", "psql -U shop -d shop -c \x27select 1\x27 \x3e/dev/null"]\n interval: 5s\n timeout: 3s\n retries: 12\n start_period: 20s\n' && echo 'runs a real query'
You will see:
healthcheck:
test: ["CMD-SHELL", "psql -U shop -d shop -c 'select 1' >/dev/null"]
interval: 5s
timeout: 3s
retries: 12
start_period: 20s
runs a real query
What it means: Proves login and database work.
Example 6: MySQL version
Type this:
printf 'test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-p$$MYSQL_ROOT_PASSWORD"]\n'
You will see:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-p$$MYSQL_ROOT_PASSWORD"]
What it means: $$ passes a literal $ to the container.
Step 3: service_completed_successfully
Why: Wait for a migration job.
What: A one-shot migrate service runs first; the API starts only if it exits with 0.
Example 7: Compose
Type this:
printf 'services:\n migrate:\n image: shop/api:1.4\n command: npm run migrate\n depends_on: {db: {condition: service_healthy}}\n api:\n image: shop/api:1.4\n depends_on:\n migrate: {condition: service_completed_successfully}\n' > deps.yaml && echo written
You will see:
written
What it means: Chain: db, then migrate, then api.
Example 8: Run
Type this:
docker compose -f compose.yaml -f deps.yaml up -d --wait 2>&1 | grep -E 'migrate|api'
You will see:
✔ Container shop-migrate-1 Exited
✔ Container shop-api-1 Healthy
What it means: migrate finished first.
Example 9: Failed migration
Type this:
docker compose logs migrate | tail -1; docker compose ps -a api --format '{{.State}}'
You will see:
migrate-1 | Error: relation already exists
created
What it means: API never starts on a broken schema.
3. Wait scripts
Step 4: A tiny shell wait loop
Why: Wait inside the container.
What: Before starting the app, loop until the database port answers, with a time limit.
Example 10: Script
Type this:
printf '#!/bin/sh\nhost="${DB_HOST:-db}"; port="${DB_PORT:-5432}"\nfor i in $(seq 1 30); do\n nc -z "$host" "$port" && exec "$@"\n echo "waiting for $host:$port ($i)"; sleep 2\ndone\necho "gave up"; exit 1\n' > wait-for.sh && chmod 755 wait-for.sh && echo ok
You will see:
ok
What it means: exec replaces the shell with your app.
Example 11: Use it
Type this:
printf 'COPY wait-for.sh /usr/local/bin/\nENTRYPOINT ["wait-for.sh"]\nCMD ["node","server.js"]\n'
You will see:
COPY wait-for.sh /usr/local/bin/
ENTRYPOINT ["wait-for.sh"]
CMD ["node","server.js"]
What it means: Wait first, then run CMD.
Example 12: Output
Type this:
docker compose logs api | head -3
You will see:
api-1 | waiting for db:5432 (1)
api-1 | waiting for db:5432 (2)
api-1 | listening on 3000
What it means: Starts as soon as the port opens.
Step 5: Ready-made tools
Why: dockerize and wait4x.
What: Small tools do the same with HTTP and TCP checks and timeouts.
Example 13: wait4x
Type this:
docker run --rm --network shop_default atkrad/wait4x:2.14 tcp db:5432 --timeout 60s && echo ready
You will see:
ready
What it means: Exits 0 when reachable.
Example 14: HTTP wait
Type this:
docker run --rm --network shop_default atkrad/wait4x:2.14 http http://api:3000/health --expect-status-code 200 && echo api-ready
You will see:
api-ready
What it means: Wait for a real health endpoint.
Example 15: In CI
Type this:
docker compose up -d && docker run --rm --network shop_default atkrad/wait4x:2.14 postgresql 'postgres://shop:example-only@db:5432/shop?sslmode=disable' && npm test
You will see:
Tests: 42 passed
What it means: Example password only.
Step 6: Port open is not database ready
Why: Pick the right check.
What: Postgres opens the port a moment before it accepts logins during start-up. A real query is the safest check.
Example 16: Port only
Type this:
nc -z db 5432 && echo port-open
You will see:
port-open
What it means: May still be starting.
Example 17: Login
Type this:
PGPASSWORD=example-only psql -h db -U shop -d shop -c 'select 1' 2>&1 | tail -1
You will see:
FATAL: the database system is starting up
What it means: Port open, not ready.
Example 18: A moment later
Type this:
sleep 3; PGPASSWORD=example-only psql -h db -U shop -d shop -tc 'select 1'
You will see:
1
What it means: Now ready.
4. Retry in code
Step 7: Retry with backoff in Node
Why: Connect patiently.
What: Each failed try waits twice as long, up to a maximum, with a little random jitter so many apps do not retry at once.
Example 19: Code
Type this:
printf 'const {Pool}=require("pg");\nconst pool=new Pool({connectionString:process.env.DATABASE_URL});\nasync function connect(max=8){\n for(let i=0;i!==max;i++){\n try{await pool.query("select 1");return console.log("db ok")}\n catch(e){const ms=Math.min(30000,1000*2**i)+Math.random()*300;\n console.log("db retry",i+1,Math.round(ms));await new Promise(r=>setTimeout(r,ms))}\n }\n process.exit(1);\n}\nconnect();\n' > db.js && node --check db.js && echo ok
You will see:
ok
What it means: Exit after eight tries so Docker can restart it.
Example 20: Output
Type this:
docker compose logs api | grep db
You will see:
api-1 | db retry 1 1102
api-1 | db retry 2 2210
api-1 | db ok
What it means: Connected on try three.
Example 21: Exit on give-up
Type this:
docker compose ps -a api --format '{{.Status}}'
You will see:
Restarting (1) 3 seconds ago
What it means: restart policy tries again.
Step 8: Retry in Python
Why: Same idea with tenacity.
What: The tenacity library adds retries with a decorator.
Example 22: Install
Type this:
pip install tenacity==9.0.0 psycopg[binary]==3.2.3 | tail -1
You will see:
Successfully installed psycopg-3.2.3 tenacity-9.0.0
What it means: Pinned.
Example 23: Code
Type this:
printf 'import os, psycopg\nfrom tenacity import retry, wait_exponential, stop_after_attempt\n@retry(wait=wait_exponential(min=1, max=30), stop=stop_after_attempt(8))\ndef connect():\n return psycopg.connect(os.environ["DATABASE_URL"])\nconn = connect()\nprint("db ok")\n' > db.py && python3 -m py_compile db.py && echo ok
You will see:
ok
What it means: One decorator.
Example 24: Spring Boot
Type this:
printf 'spring.datasource.hikari.initialization-fail-timeout=60000\nspring.datasource.hikari.connection-test-query=select 1\n'
You will see:
spring.datasource.hikari.initialization-fail-timeout=60000
spring.datasource.hikari.connection-test-query=select 1
What it means: Hikari waits up to 60 seconds.
Step 9: Pools that heal
Why: Drop dead connections.
What: After a database restart, old connections are dead. Good pools test or replace them automatically.
Example 25: pg pool events
Type this:
printf 'pool.on("error",e=>console.error("idle client error",e.message));\n' >> db.js && echo added
You will see:
added
What it means: Without this handler, Node crashes on a dead idle client.
Example 26: Test
Type this:
docker compose restart db && sleep 10 && curl -s localhost:3000/orders | head -c 40
You will see:
[{"id":1,"item":"Masala Dosa"}
What it means: Requests work again.
Example 27: Timeouts
Type this:
printf 'new Pool({connectionTimeoutMillis:5000, idleTimeoutMillis:30000, max:10})\n'
You will see:
new Pool({connectionTimeoutMillis:5000, idleTimeoutMillis:30000, max:10})
What it means: Fail fast instead of hanging forever.
Step 10: Retries between services
Why: API to API calls too.
What: When one service calls another over HTTP, a short retry with backoff hides brief restarts. Retry only safe requests like GET, never a payment POST without an idempotency key.
Example 28: curl retry
Type this:
curl -s --retry 5 --retry-delay 2 --retry-connrefused http://api:3000/health
You will see:
{"db":"up"}
What it means: curl itself can retry.
Example 29: Node fetch
Type this:
printf 'async function get(u,n=4){for(let i=0;i!==n;i++){try{const r=await fetch(u);if(r.ok)return r.json()}catch{}await new Promise(r=>setTimeout(r,500*2**i))}throw new Error("down")}\n' > get.js && node --check get.js && echo ok
You will see:
ok
What it means: Four tries, growing waits.
Example 30: Idempotency key
Type this:
printf 'POST /payments\nIdempotency-Key: 7f3c9a10-example\n'
You will see:
POST /payments
Idempotency-Key: 7f3c9a10-example
What it means: The server ignores a repeated key, so a retry cannot charge twice.
Step 11: Test the recovery
Why: Prove it works.
What: Run traffic, restart the database, and count errors. A healthy setup shows a short burst of errors and then none.
Example 31: Traffic
Type this:
for i in $(seq 60); do curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/orders; sleep 1; done > codes.txt &
You will see:
[1] 48213
What it means: Runs for one minute in the background.
Example 32: Restart db
Type this:
sleep 10 && docker compose restart db
You will see:
✔ Container shop-db-1 Started
What it means: Outage in the middle.
Example 33: Count
Type this:
sleep 55 && sort codes.txt | uniq -c
You will see:
54 200
6 503
What it means: Six seconds of errors, then full recovery.
Step 12: App healthcheck and restart policy
Why: The last safety net.
What: If the app gets stuck, a healthcheck marks it unhealthy. restart: unless-stopped brings it back when it exits.
Example 34: Health endpoint
Type this:
printf 'app.get("/health",async(q,r)=>{try{await pool.query("select 1");r.json({db:"up"})}catch{r.status(503).json({db:"down"})}});\n' >> server.js && echo added
You will see:
added
What it means: Returns 503 when the database is down.
Example 35: Compose
Type this:
printf ' api:\n restart: unless-stopped\n healthcheck:\n test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/health"]\n interval: 10s\n retries: 3\n'
You will see:
api:
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/health"]
interval: 10s
retries: 3
What it means: Docker reports unhealthy after 30 seconds.
Example 36: Note
Type this:
printf 'Plain Docker marks unhealthy but does not restart. Exit the process or use an orchestrator.\n'
You will see:
Plain Docker marks unhealthy but does not restart. Exit the process or use an orchestrator.
What it means: So the code should exit when it truly cannot recover.
5. Common mistakes and how Meera avoids them
Mistake one: trusting depends_on for the whole life of the app. Meera adds retries in code.
Mistake two: retrying forever with no wait. That floods the database when it comes back. She uses backoff with jitter and a maximum.
Mistake three: a healthcheck that only checks the port. She runs a real query.
Mistake four: sleep 30 in the entrypoint. Sometimes it is too short, always it is slow. She waits for the real condition.
Mistake five: no idle error handler on the pool. One dead connection should not crash the whole app.
Check yourself
Q1. Does depends_on help when db restarts later?
No, it only orders start-up.
Q2. What is exponential backoff?
Each retry waits twice as long, up to a maximum.
Q3. Why is an open port not enough?
The database may still be starting and refuse logins.
Q4. Which condition waits for a migration job?
service_completed_successfully.
Friends, depends_on for start, retries for life, pools that heal, and a health endpoint as the safety net. Did you understand? Let us try it once more. Restart your database while the app runs and make sure the app recovers on its own.
मित्रांनो, start साठी depends_on, आयुष्यभर retry, स्वतः बरे होणारे pools, आणि safety net म्हणून health endpoint. समजलं का? चला, पुन्हा एकदा करून पाहू. App चालू असताना database restart करा आणि app स्वतःहून recover होतो याची खात्री करा.
दोस्तों, start के लिए depends_on, पूरी ज़िंदगी retry, खुद ठीक होने वाले pools, और safety net के रूप में health endpoint. समझ आया? चलिए, एक बार फिर करके देखते हैं। App चलते समय database restart कीजिए और पक्का कीजिए कि app खुद recover हो जाता है.
Ravindra Bagale's Tip
Add a chaos step to your test plan: docker compose restart db while a load test runs. If errors stop within a minute, your retry code works.
Test plan मध्ये एक chaos step जोडा: load test चालू असताना docker compose restart db. एका मिनिटात errors थांबले तर retry code बरोबर आहे.
Test plan में एक chaos step जोड़िए: load test चलते समय docker compose restart db. एक मिनट में errors रुक जाएँ तो retry code सही है.
Lab: try at home
- Restart db and see a naive app fail.
- Write a real-query healthcheck.
- Add a migrate job with service_completed_successfully.
- Write a wait-for.sh with a time limit.
- Add retry with backoff to your connect code.
- Add a /health endpoint and a restart policy.
Recap
- depends_on only orders start-up.
- Healthchecks should run a real query.
- Wait scripts need a time limit and exec.
- Retry with exponential backoff and jitter.
- Pools must handle dead connections.
- restart policies are the last safety net.
Revision card
Interview card