53.13 Watchtower: automatic image updates for Compose apps
Friends, every time a new image is pushed, someone has to SSH to the server, pull and restart. On a small server you can let a tool do this: Watchtower. It checks the registry, pulls new images and restarts only the containers that changed. Today we run it, control it with labels and schedules, add notifications, and learn when not to use it.
मित्रांनो, प्रत्येक वेळी नवीन image push झाली की कोणाला तरी server वर SSH करून pull आणि restart करावं लागतं. छोट्या server वर हे काम एका tool ला देता येतं: Watchtower. तो registry तपासतो, नवीन images pull करतो आणि फक्त बदललेले containers restart करतो. आज तो चालवू, labels आणि schedule ने control करू, notifications जोडू, आणि तो कधी वापरू नये हेही शिकू.
दोस्तों, हर बार नई image push होने पर किसी को server पर SSH करके pull और restart करना पड़ता है. छोटे server पर यह काम एक tool को दे सकते हैं: Watchtower. वह registry जाँचता है, नई images pull करता है और सिर्फ़ बदले हुए containers restart करता है. आज उसे चलाएँगे, labels और schedule से control करेंगे, notifications जोड़ेंगे, और यह भी सीखेंगे कि उसे कब इस्तेमाल न करें.
1. Start from a story
At a small Zoho partner firm in Chennai, Rahul looked after twelve client demo servers. Each ran a Compose app. Every Friday he pushed new demo images and then spent two hours SSH-ing into all twelve servers to pull and restart.
Twice he forgot a server, and a client saw last week's demo. Once he restarted the database by mistake while updating the app.
He added Watchtower to each server with two rules: only containers with a label are updated, and checks happen once a night at 2 am. The database had no label, so it was never touched.
Now Friday's push reaches all twelve servers by Saturday morning, and Rahul gets a Slack message for each update. Today we set up the same thing.
2. Run Watchtower
Step 1: What Watchtower needs
Why: Access to the Docker socket.
What: Watchtower talks to the Docker engine through /var/run/docker.sock. That is root-level power, so run only a trusted image with a pinned tag.
Example 1: Run
Type this:
docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower:1.7.1 --interval 300
You will see:
f2a91c7b3e04
What it means: Checks every 5 minutes.
Example 2: Logs
Type this:
docker logs watchtower | tail -2
You will see:
Watchtower 1.7.1
Scheduling first run: 2026-10-25 11:35:00 +0530 IST
What it means: Running.
Example 3: Status
Type this:
printf 'Check the project page for maintenance status before production use.\n'
You will see:
Check the project page for maintenance status before production use.
What it means: Pin the version you tested.
Step 2: Inside Compose
Why: Manage it with the app.
What: Add Watchtower as a service so it starts and stops with the stack.
Example 4: Service
Type this:
printf 'services:\n watchtower:\n image: containrrr/watchtower:1.7.1\n volumes: ["/var/run/docker.sock:/var/run/docker.sock"]\n command: --interval 300 --cleanup\n restart: unless-stopped\n' > watchtower.yaml && docker compose -f compose.yaml -f watchtower.yaml up -d watchtower
You will see:
✔ Container shop-watchtower-1 Started
What it means: --cleanup removes old images after update.
Example 5: See it
Type this:
docker compose -f compose.yaml -f watchtower.yaml ps watchtower --format '{{.Status}}'
You will see:
Up 12 seconds
What it means: Running beside the app.
Example 6: One-off run
Type this:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower:1.7.1 --run-once shop-api-1
You will see:
Found new shop/api:stable image (4be1c0a)
Stopping /shop-api-1
Creating /shop-api-1
What it means: Update once, then exit.
Step 3: Which tag it follows
Why: Moving tags only.
What: Watchtower updates when the same tag points to a new digest. A fixed tag like 1.4.0 never changes, so it is never updated.
Example 7: Moving tag
Type this:
printf 'image: shop/api:stable\n'
You will see:
image: shop/api:stable
What it means: CI moves stable to each new build.
Example 8: Push new
Type this:
docker tag shop/api:1.4.1 shop/api:stable && docker push shop/api:stable | tail -1
You will see:
stable: digest: sha256:9ac3e2... size: 1784
What it means: Same tag, new digest.
Example 9: Watchtower sees it
Type this:
docker logs watchtower 2>&1 | grep -m1 Found
You will see:
Found new shop/api:stable image (9ac3e2d)
What it means: Updates within the interval.
3. Control what gets updated
Step 4: Label opt-in
Why: Update only chosen containers.
What: With --label-enable, Watchtower updates only containers that have the enable label set to true.
Example 10: Watchtower
Type this:
printf ' command: --interval 300 --cleanup --label-enable\n'
You will see:
command: --interval 300 --cleanup --label-enable
What it means: Opt-in mode.
Example 11: Label the app
Type this:
printf ' api:\n image: shop/api:stable\n labels:\n com.centurylinklabs.watchtower.enable: "true"\n'
You will see:
api:
image: shop/api:stable
labels:
com.centurylinklabs.watchtower.enable: "true"
What it means: Only api will update.
Example 12: Database untouched
Type this:
docker compose up -d && docker logs watchtower 2>&1 | grep -c shop-db-1
You will see:
0
What it means: db has no label, so it is skipped.
Step 5: Label opt-out
Why: Exclude a few.
What: Without --label-enable, everything updates except containers labelled false.
Example 13: Exclude db
Type this:
printf ' db:\n labels:\n com.centurylinklabs.watchtower.enable: "false"\n'
You will see:
db:
labels:
com.centurylinklabs.watchtower.enable: "false"
What it means: Never auto-update a database.
Example 14: Check labels
Type this:
docker inspect shop-db-1 --format '{{index .Config.Labels "com.centurylinklabs.watchtower.enable"}}'
You will see:
false
What it means: Label is set.
Example 15: Monitor only
Type this:
printf ' com.centurylinklabs.watchtower.monitor-only: "true"\n'
You will see:
com.centurylinklabs.watchtower.monitor-only: "true"
What it means: Notify about updates without applying them.
Step 6: Schedule
Why: Update at quiet hours.
What: --schedule takes a 6-field cron expression with seconds. TZ sets the time zone.
Example 16: 2 am daily
Type this:
printf ' command: --schedule "0 0 2 * * *" --cleanup --label-enable\n environment: {TZ: Asia/Kolkata}\n'
You will see:
command: --schedule "0 0 2 * * *" --cleanup --label-enable
environment: {TZ: Asia/Kolkata}
What it means: Every night at 2:00 IST.
Example 17: Logs
Type this:
docker logs watchtower 2>&1 | grep Scheduling
You will see:
Scheduling first run: 2026-10-26 02:00:00 +0530 IST
What it means: Next run time.
Example 18: Sunday only
Type this:
printf '"0 30 3 * * 0" means 3:30 am every Sunday\n'
You will see:
"0 30 3 * * 0" means 3:30 am every Sunday
What it means: Field order: sec min hour day month weekday.
Step 7: Scope per project
Why: Several stacks on one host.
What: A scope label lets each Watchtower manage only its own project.
Example 19: Scope
Type this:
printf ' command: --scope shop --label-enable\n labels: {com.centurylinklabs.watchtower.scope: shop}\n'
You will see:
command: --scope shop --label-enable
labels: {com.centurylinklabs.watchtower.scope: shop}
What it means: Watchtower itself is in scope shop.
Example 20: App label
Type this:
printf ' com.centurylinklabs.watchtower.scope: shop\n'
You will see:
com.centurylinklabs.watchtower.scope: shop
What it means: App belongs to shop.
Example 21: Other project
Type this:
docker inspect blog-web-1 --format '{{index .Config.Labels "com.centurylinklabs.watchtower.scope"}}'
You will see:
blog
What it means: Shop's Watchtower ignores it.
4. Private registries, notifications and safety
Step 8: Private registries
Why: Log in once.
What: Watchtower reads Docker's config.json. Mount it read-only after docker login on the host.
Example 22: Login
Type this:
echo "$REG_TOKEN" | docker login registry.example.com -u deploy --password-stdin
You will see:
Login Succeeded
What it means: Token from an environment variable, never typed in the file.
Example 23: Mount
Type this:
printf ' volumes:\n - /var/run/docker.sock:/var/run/docker.sock\n - /home/ubuntu/.docker/config.json:/config.json:ro\n'
You will see:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/ubuntu/.docker/config.json:/config.json:ro
What it means: Read-only.
Example 24: ECR
Type this:
printf 'For Amazon ECR, use the ecr credential helper; ECR tokens expire every 12 hours.\n'
You will see:
For Amazon ECR, use the ecr credential helper; ECR tokens expire every 12 hours.
What it means: Plain login stops working after 12 hours.
Step 9: Notifications
Why: Know what changed.
What: Watchtower uses shoutrrr URLs for Slack, email, Teams, Telegram and webhooks.
Example 25: Slack
Type this:
printf ' environment:\n WATCHTOWER_NOTIFICATION_URL: slack://hook:T000/B000/XXXX-example@watchtower\n'
You will see:
environment:
WATCHTOWER_NOTIFICATION_URL: slack://hook:T000/B000/XXXX-example@watchtower
What it means: Example token only; keep the real one in .env.
Example 26: From .env
Type this:
printf 'WATCHTOWER_NOTIFICATION_URL=${WT_SLACK_URL}\n' && echo 'real value lives in .env with chmod 600'
You will see:
WATCHTOWER_NOTIFICATION_URL=${WT_SLACK_URL}
real value lives in .env with chmod 600
What it means: Secret stays out of git.
Example 27: Message
Type this:
printf 'Watchtower updates on demo-07\nFound new shop/api:stable image (9ac3e2d)\nStopping /shop-api-1\nCreating /shop-api-1\n'
You will see:
Watchtower updates on demo-07
Found new shop/api:stable image (9ac3e2d)
Stopping /shop-api-1
Creating /shop-api-1
What it means: What arrives in Slack.
Step 10: Trigger from CI with the HTTP API
Why: Update right after a push.
What: Watchtower can listen for an HTTP call with a token, so CI tells it to update immediately instead of waiting for the schedule.
Example 28: Enable
Type this:
printf ' command: --http-api-update --http-api-periodic-polls --label-enable\n environment: {WATCHTOWER_HTTP_API_TOKEN: "${WT_TOKEN}"}\n ports: ["127.0.0.1:8088:8080"]\n'
You will see:
command: --http-api-update --http-api-periodic-polls --label-enable
environment: {WATCHTOWER_HTTP_API_TOKEN: "${WT_TOKEN}"}
ports: ["127.0.0.1:8088:8080"]
What it means: Bound to localhost only; the token comes from .env.
Example 29: Call it
Type this:
curl -s -H "Authorization: Bearer $WT_TOKEN" http://127.0.0.1:8088/v1/update && echo triggered
You will see:
triggered
What it means: CI runs this over SSH after pushing.
Example 30: Result
Type this:
docker logs shop-watchtower-1 2>&1 | tail -2
You will see:
Found new shop/api:stable image (9ac3e2d)
Creating /shop-api-1
What it means: Updated within seconds.
Step 11: Rollback
Why: Undo a bad image.
What: Point the moving tag back to the last good digest. Watchtower then rolls every server back on its next run.
Example 31: Find good digest
Type this:
docker buildx imagetools inspect shop/api:1.4.0 | grep -m1 Digest
You will see:
Digest: sha256:4be1c0a7...
What it means: The tested version.
Example 32: Retag
Type this:
docker buildx imagetools create -t shop/api:stable shop/api:1.4.0 && echo retagged
You will see:
retagged
What it means: stable now points to 1.4.0 again.
Example 33: Force now
Type this:
docker exec shop-watchtower-1 /watchtower --run-once 2>&1 | grep -m1 Found
You will see:
Found new shop/api:stable image (4be1c0a)
What it means: Servers return to 1.4.0.
Step 12: When not to use Watchtower
Why: Know the limits.
What: Watchtower is great for small servers and demos. For production fleets, use CI/CD that deploys fixed tags, or Kubernetes with rollouts.
Example 34: Databases
Type this:
printf 'Never auto-update Postgres or MySQL: major versions need migration steps.\n'
You will see:
Never auto-update Postgres or MySQL: major versions need migration steps.
What it means: Label them false.
Example 35: No health gate
Type this:
printf 'Watchtower does not wait for health or roll back on failure by itself.\n'
You will see:
Watchtower does not wait for health or roll back on failure by itself.
What it means: Tests must pass before stable moves.
Example 36: Alternative
Type this:
printf 'Diun: notify-only. You still deploy by hand or by CI.\n'
You will see:
Diun: notify-only. You still deploy by hand or by CI.
What it means: Good when you want alerts, not changes.
5. Common mistakes and how Rahul avoids them
Mistake one: watching every container. Rahul uses --label-enable and labels only app containers.
Mistake two: auto-updating the database. Its label is false, always.
Mistake three: using latest. He uses a stable tag that CI moves only after tests pass.
Mistake four: updating in the middle of the day. He schedules 2 am IST.
Mistake five: silent updates. Every update sends a Slack message, so he knows what changed.
Check yourself
Q1. What does Watchtower compare to decide an update?
The image digest behind the same tag.
Q2. Which flag limits updates to labelled containers?
--label-enable.
Q3. Why never auto-update a database?
Major versions need migrations and testing.
Q4. How do you roll back?
Retag the moving tag to the last good digest.
Friends, Watchtower saves time on small servers when you give it clear rules: opt-in labels, a schedule, notifications and a stable tag. Did you understand? Let us try it once more. Add Watchtower to a test stack, push a new stable image and watch it update only the API.
मित्रांनो, स्पष्ट नियम दिले तर Watchtower छोट्या servers वर वेळ वाचवतो: opt-in labels, schedule, notifications आणि stable tag. समजलं का? चला, पुन्हा एकदा करून पाहू. Test stack मध्ये Watchtower जोडा, नवीन stable image push करा आणि तो फक्त API update करतो ते बघा.
दोस्तों, साफ़ नियम दें तो Watchtower छोटे servers पर समय बचाता है: opt-in labels, schedule, notifications और stable tag. समझ आया? चलिए, एक बार फिर करके देखते हैं। Test stack में Watchtower जोड़िए, नई stable image push कीजिए और देखिए कि वह सिर्फ़ API update करता है.
Ravindra Bagale's Tip
Move the stable tag only from CI, after tests pass. Then Watchtower can never ship an image that nobody tested.
Stable tag फक्त CI मधून, tests pass झाल्यावरच हलवा. मग test न झालेली image Watchtower कधीच ship करू शकत नाही.
Stable tag सिर्फ़ CI से, tests pass होने के बाद ही हिलाइए. फिर बिना test हुई image Watchtower कभी ship नहीं कर सकता.
Lab: try at home
- Run Watchtower with --run-once.
- Add it to Compose with --cleanup.
- Switch to opt-in labels and protect the database.
- Schedule updates for 2 am IST.
- Add a Slack notification URL from .env.
- Push a bad image and roll back by retagging.
Recap
- Watchtower updates containers when a tag's digest changes.
- It needs the Docker socket, so pin a trusted version.
- --label-enable makes updates opt-in.
- --schedule runs at quiet hours.
- Notifications tell you what changed.
- Databases and production fleets need other tools.
Lab
- Write the 3-tier compose.yaml and bring it up with --wait.
- Break the DB password and watch depends_on stop the API.
- Add an adminer profile.
- Create dev and prod override files and compare docker compose config.
- Scale api to 4 and test through nginx.
Practice task
Convert any 3 docker run commands you use into one compose.yaml with a healthcheck, a secret, an internal network, a named volume and a tools profile. Show docker compose config passes.
Ravindra Bagale's Tip
In production always use -f compose.yaml -f compose.prod.yaml, and run docker compose config first to see the final file.
Production मध्ये नेहमी -f compose.yaml -f compose.prod.yaml वापरा, आणि आधी docker compose config चालवून final file बघा.
Production में हमेशा -f compose.yaml -f compose.prod.yaml इस्तेमाल कीजिए, और पहले docker compose config चलाकर final file देखिए.
In short
- Compose turns many docker run lines into one file.
- depends_on with service_healthy fixes start order.
- .env interpolates YAML; env_file sets container env; secrets mount files.
- Profiles hide optional services.
- Override files separate dev and prod.
- Scale services without fixed host ports.
Revision card
Interview card