Docker Compose Multiple Files: How Override Merge Dey Work

Compose dey do wetin with more than one file

Docker Compose fit build one project from many files. E dey read dem in the order wey e receive dem and merge dem into one model, so later file go win for any value wey conflict. Two mechanisms dey do this from the command line: an override file wey Compose load by itself, and the -f flag wey you pass by hand. Third one dey inside the file itself, the include element, and e work differently from the other two.

The merge no be plain overwrite. Mappings dey merge key by key, sequences dey append, and small set of fields dey replace whole. Na this difference dey cause surprises, and the ports list na the one wey catch almost everybody.

Everything below assume say na Compose v2, the docker compose plugin, no be the old docker-compose script. Run docker compose version to check. If you never write Compose file, start with Docker Compose basics guide and come back.

Compose je load override file wey nobody tell am about

Run docker compose up without -f flag. Compose go search working directory, then the parent directories, for compose.yaml or docker-compose.yaml. If override file dey beside the base file, Compose go load that second file by itself.

ls compose.yaml compose.override.yaml
docker compose up -d

When both files dey, na the same thing as typing both files by hand.

docker compose -f compose.yaml -f compose.override.yaml up -d

The names wey Compose recognises na compose.override.yaml, compose.override.yml, and the older docker-compose.override.yml and docker-compose.override.yaml. Any other name, like compose.dev.yaml, go load only when you specify am with -f.

As soon as you pass one -f, automatic loading go stop. docker compose -f compose.yaml up go read exactly that one file and ignore the override. Na this property the dev and prod pattern later for this guide dey use.

This fit work for both sides on a server. If you leave override file for the deploy directory, every bare docker compose command wey you run from that directory go load am, including the one your cron job runs. Na so production stack fit end up bind-mounting source directory wey nobody plan to ship. Run docker compose config after every deploy and read wetin come out.

Ordering with -f, and where relative paths resolve

Compose dey build the configuration according to the order wey you supply the files. Files wey come later dey override and add to the ones before dem. From left to right, na the last one dey win.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml up -d

Every command for that project need the same file list. If you run up with two files and logs with one, you dey talk to different merged model. This fit quickly cause Compose to say say service no dey. Set the list once instead with the COMPOSE_FILE environment variable.

export COMPOSE_FILE=compose.yaml:compose.prod.yaml
docker compose config
docker compose up -d

The separator na : for Linux, and COMPOSE_PATH_SEPARATOR dey change am. COMPOSE_FILE fit also dey inside the project .env file. This one make am part of the checkout instead of your shell history. Anything wey you set directly for the command line go override the environment variable.

Now na the rule wey dey break bind mounts. When you use multiple files with -f, all relative paths inside all the files dey resolve against the directory of the first file. Dem no dey resolve against the file wey contain dem. If you write ./data:/var/lib/postgresql/data inside deploy/prod/compose.prod.yaml, Compose still dey look for ./data beside the base file. Docker go then create an empty directory for that wrong path, and the container go start with nothing inside am. E go look like data loss, but e no be. Pass --project-directory to set the base path yourself, or use include. That one dey resolve each file against its own directory.

The project name dey come from that same base directory. So, if you change which file come first, you fit rename the project. When project name change, new container names and new volume names go dey created. The old volume still dey for disk under the old name. Pin the name instead with top-level name: inside the base file.

name: myapp

Which fields dey merge, and which ones dem dey replace

Compose dey merge based on the value type, not the field name.

• Fields wey get only one value dey replace. image, command, entrypoint and mem_limit go take the later value directly. You no fit append one argument to command, because the override dey rewrite the whole line.
• Mappings dey merge key by key. environment, labels, volumes and devices go keep every key from both files, and the later file go win for any key wey dey inside both files. For environment and labels, the key na the variable or label name. For volumes and devices, the key na the container path.
• Sequences dey append. dns, dns_search, expose, tmpfs and external_links dey join together. If base file get expose: ["3000"] and override file get ["4000", "5000"], the result go be ["3000", "4000", "5000"].

Four sequences get identity key, so entries wey match for that key dey merge instead of appending. volumes, secrets and configs dey match for target. ports dey match based on the combination of ip, target, published and protocol.

Read that ports rule two times, because na there the common mistake dey. Two port entries na the same entry only when all four parts agree. If you change any one of them, Compose go see am as another unrelated port, so e go keep both.

Why your port still dey published after the override

A base file wey dey publish one service for every interface:

services:
  web:
    image: nginx:1.27
    ports:
      - "8080:80"

An override wey bind am to localhost only, because one reverse proxy go sit for front of am:

services:
  web:
    ports:
      - "127.0.0.1:8080:80"

Check the result before you assume say e work.

docker compose -f compose.yaml -f compose.prod.yaml config

Both entries dey for the output. The ip part no be the same, 0.0.0.0 against 127.0.0.1, so dem be two different ports as far as the merge be concerned, and the public binding wey you try remove still dey for the model. This one matter pass for Docker than for other places, because one published port dey write into iptables before your firewall rules. The mechanism dey covered for why published Docker ports dey bypass ufw.

Two fixes dey. The explicit one na the !override tag, wey replace the whole attribute and skip the merge rules:

services:
  web:
    ports: !override
      - "127.0.0.1:8080:80"

!override need Compose v2.24.4 or newer. The portable fix no need any tag: keep ports comot from the base file completely and declare am only for the environment-specific files. If nothing dey to merge, nothing go leak. Na this pattern dem use for the worked example below.

Delete value wey base file set

!reset dey remove an attribute, e put am back to default or null. E dey accept value but e no dey use am, so write something wey valid and empty.

services:
  web:
    ports: !reset []
    environment:
      DEBUG: !reset null

!reset need Compose v2.24 or newer. Use am when you no fit edit the base file, like vendor fragment wey you pull in.

include, for stacks wey dem assemble from parts

include dey pull another Compose application enter your model. E be top-level element, no be flag.

include:
  - path: ../commons/compose.yaml

Every path for include dey load as im own Compose application model, with im own project directory. So relative paths inside that file dey resolve against that file own directory. Na this be the real difference from -f, and na why include be the correct tool when the fragment dey another folder or another repository.

The long form get sub-options.

include:
  - path:
      - ../monitoring/compose.yaml
      - ../monitoring/compose.vps.yaml
    project_directory: ../monitoring
    env_file: ../monitoring/.env

path dey accept a list, and dem dey merge those files together with the normal rules before the result join your model. project_directory dey set the base path wey dem use resolve relative paths inside the included file. env_file dey give the included file im own variables for interpolation. This stops shared fragment from quietly reading your project own .env. include need Compose v2.20.0 or newer.

If your file and included file get duplicate resource names, dem go report am as error instead of quietly merging dem, and na intentional. To change wetin an included file declare, put the change for compose.override.yaml. The override dey apply to the assembled model, so e fit touch included resources without colliding with dem.

The short version: include dey compose separate applications, while -f dey layer configuration on top one application.

Dev and prod split for one VPS

Na three files dey show the whole pattern here. The base file dey declare wetin dey true everywhere, and e no publish any port.

name: myapp

services:
  app:
    image: ghcr.io/example/app:1.4.2
    environment:
      DATABASE_URL: postgres://app:${POSTGRES_PASSWORD}@db:5432/app
      LOG_LEVEL: info
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: app
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  db_data:

The depends_on condition na wetin make the app wait for database wey dey answer, instead of container wey only dey exist. See healthchecks and depends_on conditions for explanation. POSTGRES_PASSWORD dey come from the project .env file through interpolation. That file must never enter git. See env files and Compose secrets for safer options.

Next na compose.override.yaml, wey Compose loads by itself. Na the developer file be this.

services:
  app:
    build: .
    command: npm run dev
    environment:
      LOG_LEVEL: debug
    ports:
      - "3000:3000"
    volumes:
      - ./src:/app/src

  db:
    ports:
      - "127.0.0.1:5432:5432"

For laptop, bare docker compose up dey merge those two files. command replaces the image default because na one value e get. LOG_LEVEL replaces info because environment dey merge by key. The bind mount and the two published ports na additions only. The database port dey bind to localhost, so laptop wey dey the same network no go expose PostgreSQL to everybody.

Last na compose.prod.yaml. Its name no be one wey Compose dey look for, so Compose no go load am by mistake.

services:
  app:
    ports:
      - "127.0.0.1:8000:3000"
    deploy:
      resources:
        limits:
          memory: 512M

For the VPS, name both files. Na this naming exactly dey prevent the override from loading.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml up -d
docker compose -f compose.yaml -f compose.prod.yaml ps

ps suppose list both services as running, with db showing (healthy). Because you pass -f, compose.override.yaml no read. So the dev command, the source bind mount, and the public port 3000 no fit reach production, even though the file dey the same directory. Port 8000 dey localhost only, ready for proxy. See running several apps behind Traefik when you add the second service.

Set COMPOSE_FILE=compose.yaml:compose.prod.yaml inside the server .env, and your other commands go return to plain docker compose logs -f app.

Read the merged model before you deploy

docker compose config dey print the model wey don merge finish and interpolate finish. E no be preview. Na the exact input wey Compose go use, so if the output no match wetin you expect, na the output correct.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml config --no-interpolate
docker compose -f compose.yaml -f compose.prod.yaml config --services

--no-interpolate no dey expand ${VAR}. Use am before you paste the output anywhere, because plain config dey print every resolved secret as clear text. --services dey list only the service names. Na quick way to confirm say an include pull wetin you expect.

Failure modes, and wetin you go see

no configuration file provided: not found. Compose no find anything to read. You dey outside the project directory, or COMPOSE_FILE name a path wey no exist. Compose dey search parent directories for the default base file, but e no dey search anywhere for a file wey you name by yourself.

WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. Interpolation dey resolve against the project .env file and the shell environment. The project directory for here na the directory of the first -f file. If you deploy from another directory instead of the one wey hold .env, you go see this warning, and then get database wey go reject every connection.

Your override edit no show for docker compose config. Either you pass -f, wey dey turn off automatic override loading, or Compose find compose.yaml for a parent directory and your override file no dey beside am. If you run docker compose config without any other arguments, e go tell you the model wey Compose dey really build.

A bind mount empty, and Docker create directory wey you no ask for. The relative path resolve against the first file directory. Correct the path, pass --project-directory, or move the fragment behind include.

Containers come back with new names, and one volume look blank. The project name don change because the project name follow the first file directory. Add top-level name: to the base file, and the naming no go dey change again. The old volume still dey under the old prefix, and docker volume ls go show am.

A port wey you remove for the override still dey open. The ports merge append instead of replacing. Confirm am with docker compose config, then either use !override or move ports out of the base file.

FAQ

Compose dey load compose.override.yaml automatically?

Yes, when you run docker compose without -f flag. Compose dey search the working directory and its parent directories for compose.yaml or docker-compose.yaml. If override file dey beside am, Compose go load that file second. The recognised names na compose.override.yaml, compose.override.yml, docker-compose.override.yml and docker-compose.override.yaml. If you pass any -f, e dey disable this behaviour, so docker compose -f compose.yaml up go read only one file.

Which order multiple -f files dey merge?

Na from left to right. Compose dey build the configuration according to the order wey you supply the files. Each file dey override and add to the files before am, so the last file for the command line wins any conflict. You must use the same list for every command for that project. Na COMPOSE_FILE=compose.yaml:compose.prod.yaml dey handle this.

Why my port still dey published after I override am?

Because ports entries dey identified by the complete set of ip, target, published and protocol. If you override 127.0.0.1:8080:80 against base value of 8080:80, the ip part no be the same. So Compose dey treat am as another port and e keeps both. Run docker compose config and you go see the two entries. Use ports: !override with Compose v2.24.4 or newer. Another option na to remove ports from the base file, so nothing dey available to merge against.

Wetin be the difference between include and -f?

-f dey layer several files onto one application. Every relative path for every file dey resolve against the first file directory. include dey bring in a separate Compose application. Each included path keeps its own project directory, so its relative paths dey resolve against itself. Use -f for environment layers of your own stack, and include for a fragment wey somebody dey maintain elsewhere. include needs Compose v2.20.0 or newer.

How I fit remove value wey the base file set?

Use the !reset tag with Compose v2.24 or newer. Write ports: !reset [] or MY_VAR: !reset null for the overriding file, and the attribute go return to its default or null. The value wey you give the tag na required, but Compose no dey use am. If you want replace an attribute instead of clearing am, !override go do that, and e needs v2.24.4 or newer.