Docker Compose healthcheck: wetin dey really work

Docker Compose healthcheck dey do wetin exactly

Docker Compose healthcheck na one command wey Docker dey run inside the container at intervals. Docker no dey read your logs, monitor your port, or inspect your process list. E dey run the command, read the exit code, and store one state for the container: starting, healthy, or unhealthy. Exit code 0 mean say e healthy. Any other exit code mean say e unhealthy, and Docker reserve exit code 2, so no ever return am intentionally.

Na all the mechanism be this. Almost every healthcheck problem na the same kind problem: the command wey you write dey answer different question from the one wey you mean to ask. This guide assume say you sabi already how to write compose file for VPS, and e continue from the point where the stack start in wrong order.

services:
  api:
    image: ghcr.io/example/api:1.4.0
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/healthz"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 30s

The test value get two useful forms. A list wey start with CMD dey run the command directly, without shell, so pipes, &&, and variable expansion no go work. A list wey start with CMD-SHELL dey pass the remaining part as one string to /bin/sh -c inside the container. Na this one you need whenever the check require shell syntax. A plain string dey treated as CMD-SHELL. A list of exactly ["NONE"] dey remove healthcheck wey the image bake in through im Dockerfile.

The check dey run inside the container, so every binary wey e name must dey inside that image. Verify this first, because slim image wey no get curl go produce container wey dey unhealthy permanently for reason wey no go ever show for application log. Test am by hand:

docker compose exec api curl --version

A missing binary go answer with OCI runtime exec failed: exec: "curl": executable file not found in $PATH: unknown. Alpine based images usually dey ship BusyBox wget instead, so the check go become ["CMD", "wget", "-q", "-O", "-", "http://localhost:8080/healthz"].

How , and join together

Settings five dey control the timing. Docker Engine dey provide their defaults, no be Compose.

• interval: time between two checks after container don pass im start period. Default na 30s.
• timeout: how long one check fit run before Docker kill am and count that run as failure. Default na 30s.
• retries: number of failures wey must happen one after another before state change to unhealthy. Default na 3.
• start_period: grace period after container start. Default na 0s.
• start_interval: how often the check run during start period. Default na 5s, and e need Docker Engine 25.0 or newer.

The rule wey matter be say: during start period, failing check no dey count toward retries, and container remain for starting. The first time the check succeed, container become healthy and start period end immediately, even if plenty of the time still remain. If start period finish while check still dey fail, normal countdown go start, and container need retries failures one after another before dem mark am as unhealthy.

So, the worst-case time from container start to unhealthy na start_period plus retries multiplied by interval plus timeout. With the values for the file above, na 30 plus 5 times 13, wey equal 95 seconds. Write that number down before you set deploy timeout, because rollout wey give up after 60 seconds no go ever see this container reach final state.

The common mistake here na to increase retries to cover slow start. E go work once, then e go cause problem forever: service wey need 8 retries to boot go now tolerate 8 consecutive failures for production before anything notice. Use start_period instead, because e apply only before the first success.

Why depends_on by itself no dey guarantee anything

The short form of depends_on na the main reason for most of the confusion.

  api:
    depends_on:
      - db

This one mean only one thing: start the db container before the api container. Compose go wait make the container create and start. E no go wait for PostgreSQL to finish the first-time initialization, and e no go wait for port 5432 to accept connection. Your app go start about one second later, connect to port wey nothing dey listen to yet, then stop. For the log, you go see Connection refused, or FATAL: the database system is starting up when the server don come up but e still dey recover.

The long form na wetin people really want:

  api:
    depends_on:
      db:
        condition: service_healthy
        restart: true
      migrate:
        condition: service_completed_successfully

condition get three values. service_started na the same thing as the short form. service_healthy go hold the dependent service back until the dependency report say e healthy. This one only make sense when that dependency define a healthcheck, either for the compose file or for its image. service_completed_successfully dey wait for a one-shot container, like a database migration, to exit with status 0.

Two extra fields dey beside condition. restart: true tell Compose to restart this service after e update the dependency service. required: false change a missing dependency from error to warning.

Now, na this limit dey catch people. Compose evaluate these conditions when the stack come up. Dem control startup order, not ongoing supervision. If the database restart at three in the morning, nothing go evaluate service_healthy again, and nothing go restart your app to satisfy am again. Your application code still need reconnect by itself. docker compose up --no-deps api skip the whole mechanism by design, and starting a container directly with docker start do the same thing.

Write check wey go test readiness, no be say process dey

Check like pgrep nginx dey prove say process table entry dey. E no dey prove say service fit answer request. Web application fit keep im listening socket open long after im database pool don die, and process check go still show green throughout the whole outage.

Make container do the work wey e exist to do:

• For HTTP service, request real endpoint. curl -fsS dey exit non zero for any status of 400 or above because of -f, so 500 from broken app na failed check.
• For PostgreSQL, use pg_isready. E dey exit 0 when server dey accept connections, 1 when e dey reject dem, 2 when e no respond at all, and 3 when the parameters wey you pass wrong.
• For Redis, use redis-cli ping. E dey print PONG and exit 0.
• For MariaDB, the official image dey ship with healthcheck.sh script, and healthcheck.sh --connect --innodb_initialized na the form wey im maintainers document.

pg_isready get one trap wey you need know. For the very first start with empty data directory, the official postgres image dey run im initialisation against temporary server wey dey listen only on Unix socket. pg_isready without host argument dey use that socket, so e fit answer “accepting connections” while TCP port 5432 still dey closed to your application. Point the check to TCP explicitly and the problem go comot, because temporary server no dey answer there.

    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 30s

The doubled dollar signs no be typo. Compose dey expand $VAR by itself while e dey read the file, and that one go put value from your host environment inside the check. $$ dey escape am down to one $, so the shell inside the container go expand am against the container own environment.

Postgres and app stack wey start for correct order

services:
  db:
    image: postgres:17.5
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env}
      POSTGRES_DB: appdb
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 30s
    restart: unless-stopped

  api:
    image: ghcr.io/example/api:1.4.0
    environment:
      DATABASE_URL: postgres://appuser:${DB_PASSWORD}@db:5432/appdb
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/healthz"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 30s
    ports:
      - "127.0.0.1:8080:8080"
    restart: unless-stopped

volumes:
  pgdata:

Bring am up and monitor how the states dey change:

docker compose up -d
docker compose ps

The STATUS column dey carry the health state inside brackets. Healthy pair go show Up 41 seconds (healthy) for both rows. While the database still dey initialise, db go show Up 4 seconds (health: starting) and api no go dey for the list, because Compose never create am yet.

To see why check pass or fail, read the health log:

docker inspect --format '{{json .State.Health}}' "$(docker compose ps -q db)"

Docker dey keep the last few results, each one with start time, end time, an ExitCode, and the command's Output. The stored output dey truncated, so if check print large page body, the log entry no go useful. Make checks quiet.

Wetin Docker dey do when container become unhealthy

Nothing. Na this answer dey surprise people pass.

Docker Engine for one host no dey restart unhealthy container. The restart: unless-stopped policy dey react when main process exit, and unhealthy container never exit. E fit remain for unhealthy for one week while Compose leave am alone. Swarm mode dey replace unhealthy tasks, but plain Compose stack for one server no dey do am.

That one leave two honest options. Make the process exit when e know say e don break, so restart policy get wetin e fit act on. Or monitor the state from outside and alert when e happen. If you point an Uptime Kuma monitor to the same endpoint wey your healthcheck dey call, broken dependency go show for both places, and monitor go tell you instead of user. If traffic dey reach the app through a Traefik reverse proxy, remember say the proxy own view of backend dey separate from Docker health state, so one no cover the other.

Debugging check wey no dey ever healthy

Run the exact command by yourself, inside the same container, then check the exit code:

docker compose exec api curl -fsS http://localhost:8080/healthz; echo "exit=$?"

exit=0 for here while the container still dey report unhealthy, e mean say your compose test no be the same as wetin you just type. Most times, na because dem use CMD where shell syntax suppose dey.

Two mistakes dey cause most of the remaining cases. The first one na wrong port. Healthcheck dey run inside the container, so e must use the container port, never the published host port. With ports: - "8080:3000", the application dey listen on 3000. Check against http://localhost:8080 go fail forever, even when the site dey work well for browser. The second one na wrong host. Inside the check, localhost na that same container. E correct when you dey check the container itself, but e wrong when you dey check another container. For that one, use the service name, like db.

One last case get special name: healthcheck dey pass while users dey see errors. This one happen when the endpoint returns static 200 without checking anything real. Readiness endpoint wey never query the database no fit tell you say the database don go down. Make am run one cheap real query.

FAQ

Why my app still dey fail to connect when depends_on talk say database healthy?

Because condition: service_healthy dey evaluate one time, when stack start. E no dey supervise anything afterwards. If database container restart later, Compose no go restart your application to satisfy the condition again, so your application code need get im own reconnect and retry logic. The condition still no dey do anything when you start one container with docker start or with docker compose up --no-deps.

I need healthcheck if the image already define one?

Usually no, and overriding am often na step backward, because image maintainer sabi wetin readiness mean for that software. Add your own only when image check no correct for your setup, for example when e dey probe port wey you move. To turn image healthcheck off, set test: ["NONE"] or disable: true for the service.

Healthcheck suppose use curl or wget?

Use the one wey already dey inside the image, and confirm am with docker compose exec <service> curl --version before you rely on am. Many Debian based images no get either of dem. Alpine based images get BusyBox wget. No add package to an image only to run healthcheck when the software ship im own client, like pg_isready or redis-cli.

Unhealthy container go restart automatically?

Docker Engine no dey do am automatically for one host. Restart policies dey react when process exit, not when health state change, so unhealthy container go remain up and remain broken until another thing act on am. Either make the process exit when e detect the failure, or run external monitor wey go alert when the state change.

How long start_period suppose be?

Make e long enough for the slowest valid first start wey you don measure, plus some margin. Time am with docker compose up against empty volume, because database first start dey far slower than every start after am. start period wey too long only dey delay the first unhealthy verdict. Retries wey too high dey weaken the check for the whole life of the container, and na the worse failure.