Container Healthchecks and Restart Policies: What Docker Actually Does

A common belief is that if you add a healthcheck and restart: unless-stopped, Docker will notice a hung container and restart it. It will not. Docker marks the container unhealthy, shows it in docker ps, and carries on. The restart policy only reacts to the process exiting. These two features are independent, and most surprises in single-host container setups come from assuming they are wired together.

This article explains what each mechanism does, how they interact in Compose, and how to build a setup that recovers from the failures you actually see.

Restart Policies: Reacting to Exit

A restart policy tells the Docker daemon what to do when the main process of a container exits. The four values:

• no (default): never restart.
• on-failure[:max]: restart only when the exit code is non-zero, optionally capped at max attempts.
• always: restart whatever the exit code, and also start the container when the daemon starts.
• unless-stopped: like always, except a container you explicitly stopped stays stopped after a daemon restart.

services:
  api:
    image: example/api:1.0
    restart: unless-stopped

Some details that matter:

• Backoff. Docker delays restarts, starting small and doubling up to a cap, so a crash-looping container does not spin the CPU. The delay resets once the container has run successfully for a while.
• Crash loops hide behind “Up”. With always, a container that dies seconds after each start is retried indefinitely. It looks “Up” most of the time while a broken release sits underneath.
• Manual stops win. If you run docker stop, the policy does not bring the container back until the daemon restarts, and then only for always.
• Killed by hand is not a crash. A container you stopped or killed yourself is treated as stopped by you, so the policy may not revive it. Do not rely on restart policies to undo manual intervention.
• Out-of-memory kills count as exits. The kernel OOM killer terminating the process produces an exit, so the policy restarts it, and you get a loop if the memory limit is simply too low.

In Compose, use the service-level restart: key for plain docker compose up. The deploy.restart_policy block belongs to Swarm-style deploy configuration and supports condition, delay, max_attempts and window. Mixing the two styles in one file is a common source of confusion, so pick one and stay consistent. Docker Compose in Production shows a fuller production file.

Healthchecks: Reporting State

A healthcheck is a command that Docker runs inside the container on a schedule. Its exit code determines the health state:

• exit 0: healthy
• exit 1: unhealthy
• exit 2 is reserved and should not be used

You can define it in the Dockerfile or in Compose. The Compose definition overrides the image’s.

HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD curl -fsS http://localhost:8080/healthz || exit 1

services:
  api:
    image: example/api:1.0
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 20s
      start_interval: 2s

The parameters:

Field	Meaning
interval	Time between checks.
timeout	A check taking longer than this counts as a failure.
retries	Consecutive failures before the state flips to unhealthy.
start_period	Grace time during which failures do not count toward retries.
start_interval	Faster polling during the start period, in recent Docker versions.

The state machine has three values: starting, healthy and unhealthy. A container begins in starting. A success at any time moves it to healthy, even within the start period. Failures during start_period are ignored. After the start period, retries consecutive failures produce unhealthy, and a single success returns it to healthy.

Check the state yourself:

docker inspect --format '{{json .State.Health}}' api | jq
docker ps --filter health=unhealthy

The inspect output includes the last several check results with exit codes and captured output, which is the first place to look when a check fails for a reason that is not obvious.

The shell form gotcha

test: ["CMD", ...] runs the binary directly. test: ["CMD-SHELL", "..."] runs through /bin/sh -c. Use CMD-SHELL when you need pipes or ||. The check runs inside the container, so the tool must exist there. A curl check in a distroless or scratch image fails permanently, and unhealthy forever is the result. Options are to ship a tiny probe binary, add a --healthcheck subcommand to your application, or use a minimal tool where the image has one:

# Minimal check in an image that ships busybox wget
wget -q --spider http://localhost:8080/healthz

What Docker Does Not Do With Unhealthy

Here is the central point. In plain Docker and Compose, an unhealthy container is not restarted. Nothing happens apart from the label and a health event on the daemon’s event stream. Compose does not act on it either, except in two places:

• depends_on with condition: service_healthy waits for a dependency to become healthy before starting the dependent.
• docker compose up --wait blocks until services are running and healthy, which is useful in deployment scripts.

Docker Swarm is the exception: a Swarm service task that becomes unhealthy is replaced by the orchestrator. Kubernetes has similar behaviour with its liveness probes, though the semantics differ. On a single host with Compose, you have to build the reaction yourself.

Making Compose React to Unhealthy

There are several practical approaches.

1. Let the process die when it is broken. The cleanest fix is to make the application exit on fatal conditions, such as a connection pool it cannot rebuild or a deadlock caught by an internal watchdog. The restart policy then does exactly what it is good at.

2. A watcher container. A small sidecar listens to Docker events and restarts containers that turn unhealthy. Community images exist for this. If you use one, scope it with labels so it only touches containers you opted in, and remember that mounting the Docker socket is effectively handing out root on the host.

3. A host-level script. A cron job or systemd timer can inspect specific containers and restart them. Keep it explicit. A script that restarts everything matching a broad filter can turn a partial outage into a full one, so name each container.

#!/usr/bin/env bash
# restart-unhealthy.sh: restart explicitly named containers that report unhealthy
for name in api worker; do
  state=$(docker inspect --format '{{.State.Health.Status}}' "$name" 2>/dev/null)
  if [ "$state" = "unhealthy" ]; then
    echo "$(date -Is) restarting $name"
    docker restart "$name"
  fi
done

4. Alert instead of auto-restart. Restarting hides problems. For stateful services, an automatic restart can make things worse: a database that is slow because of a long recovery should not be restarted mid-recovery. Alerting a human, or alerting and restarting only after a threshold, is often wiser.

Ordering Startup With depends_on

Without conditions, depends_on only controls start order, not readiness. A web app that connects to its database on boot will race the database. Use health conditions:

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
    restart: unless-stopped

  api:
    image: example/api:1.0
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

This only gates the initial start. If the database becomes unhealthy later, Compose does not stop or restart the API. Your application still needs its own retry logic for lost connections. Treat depends_on as a convenience for boot, not as a guarantee of runtime availability.

Designing a Good Healthcheck

A healthcheck that is too shallow reports healthy for a broken service. One that is too deep causes false alarms and cascading failures. Guidelines:

• Check what this container owns. A web app’s check should confirm the process can serve a request and reach the dependencies it cannot function without. Avoid checking third-party services, which would mark you unhealthy during someone else’s outage.
• Separate liveness from readiness in your head. Docker has a single check. If it fails whenever the database is down and your remedy is a restart, you will restart the app in a loop for a problem a restart cannot fix. Prefer a check that reflects “this process is wedged”, and handle dependency loss in application logic.
• Keep it cheap. The command runs every interval for the container’s lifetime. A heavy query as a check becomes load.
• Mind the timeout. A check that sometimes takes longer than timeout under load will flap and flag healthy-but-busy containers as unhealthy.
• Set start_period from observation. If an app takes 40 seconds to warm up, a 20-second grace period leads to spurious unhealthy states on every deploy.
• Do not log noise. Health endpoints should not write access log lines every few seconds, or they will dominate your logs.

Common Mistakes

• Assuming restart: always restarts unhealthy containers. It does not.
• A healthcheck command that is missing from the image, producing a permanent unhealthy state.
• Using localhost inside the check for a service that binds only to a specific interface or a different port.
• No start_period, so slow starters are flagged during boot and service_healthy dependencies time out.
• Using always for a job that is meant to run once, so it re-runs forever after finishing successfully. Use on-failure for batch work.
• Restart loops hiding a broken release. Watch restart counts, not just “is it up”.

Observing Restarts and Health

The container’s restart count is available from inspect:

docker inspect --format '{{.RestartCount}} {{.State.Status}} {{.State.OOMKilled}}' api

For continuous monitoring, container metrics exporters let you alert on a container that stops reporting or restarts repeatedly. For endpoints visible to users, an external probe is a better judge of health than Docker’s internal check.

When Not to Rely on This

Healthchecks plus restart policies give you a modest, useful level of self-healing on one host. They do not give you scheduling, rolling updates, or failover to another machine. If the host dies, nothing restarts your containers elsewhere. If you need those properties, you are into orchestrator territory. For many small deployments the single-host approach, with careful checks and good alerting, is simpler and more predictable. See Zero-Downtime Deploys Without Kubernetes for the deploy side.

Key Takeaways

• Restart policies react to process exit only; a healthcheck result never triggers a restart in plain Docker or Compose.
• Health states are starting, healthy and unhealthy; tune start_period, timeout and retries from real startup behaviour.
• depends_on with service_healthy orders boot only; applications still need retry logic for runtime dependency loss.
• To act on unhealthy, make the process exit on fatal states, or add an explicit watcher that restarts named containers, and avoid broad filters.
• Keep healthchecks cheap and scoped to what the container owns, and never include third-party dependencies.
• Monitor restart counts and probe from outside, because a container looping under always still looks up most of the time.