Docker Compose: How Multiple Files Really Merge

Compose dey do wetin if file pass one

Docker Compose fit build one project from plenty files. E dey read dem for the order wey e receive dem, then merge dem into one model. So, later file go win for any value wey conflict. Two mechanisms dey do this from command line: override file wey Compose load by itself, and the -f flag wey you pass manually. The third one dey inside the file itself, na the include element, and e dey work differently from the other two.

The merge no be simple overwrite. Mappings dey merge key by key, sequences dey append, and small group of fields dey replace completely. Na this difference dey cause surprise, and the ports list na the one wey dey catch almost everybody.

Everything for here assume say na Compose v2 you dey use, the docker compose plugin instead of the old docker-compose script. Run docker compose version to check. If you never write Compose file before, start with Docker Compose basics guide and come back.

The override file wey Compose load even when you no tell am

Run docker compose up without -f flag, and Compose go search the working directory, then the parent directories, for compose.yaml or docker-compose.yaml. If override file dey beside the base file, Compose go load that second file by itself.

ls compose.yaml compose.override.yaml
docker compose up -d

When both files dey present, na the same thing as typing both files out by hand.

docker compose -f compose.yaml -f compose.override.yaml up -d

The names wey Compose recognise na compose.override.yaml, compose.override.yml, and the older docker-compose.override.yml and docker-compose.override.yaml. Any other name, like compose.dev.yaml, go load only when you name am with -f.

As soon as you pass one -f, automatic loading go stop. docker compose -f compose.yaml up go read exactly that one file and ignore the override. Na this property the dev and prod pattern later for this guide dey use.

This setup fit work for both good and bad ways on server. If override file remain for deploy directory, every bare docker compose command wey you run from that directory go load am, including the one wey your cron job run. Na so production stack fit end up bind-mounting source directory wey nobody plan to ship. Run docker compose config after every deploy and read the output. If that deploy no dey attended by person, this check go only help when something tell you say e fail. A push channel like self-hosted ntfy server fit do this. Cron job or systemd OnFailure unit fit post notification go there.

Ordering with -f, and where relative paths resolve

Compose dey build the configuration according to the order wey you supply the files. Later files override and add to the files before dem. From left to right, the last one win.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml up -d

Every command for that project need the same file list. Run up with two files and logs with one file, and you dey talk to another merged model. This na quick way to get a service wey Compose talk say e no exist. The risk dey higher for stack wey upgrades run as one-off commands, like the database migration step for self-hosted Chatwoot support desk, where docker compose run wey you run with wrong file list quietly targets another model from the one wey your services dey already use. Set the list once instead with COMPOSE_FILE environment variable.

export COMPOSE_FILE=compose.yaml:compose.prod.yaml
docker compose config
docker compose up -d

The separator na : for Linux, and COMPOSE_PATH_SEPARATOR changes am. COMPOSE_FILE fit also dey inside the project .env file. This makes am part of the checkout instead of your shell history. Anything wey you set directly for command line get priority over the environment variable.

Now na the rule wey dey break bind mounts. When you use multiple files with -f, all relative paths for all the files resolve against the directory of the first file. Dem no resolve against the file wey contains dem. If you write ./data:/var/lib/postgresql/data inside deploy/prod/compose.prod.yaml, Compose still dey look for ./data beside the base file. Docker then creates empty directory for that wrong path, and the container starts without anything inside am. E fit look like data loss, but e no be. Pass --project-directory to set the base path yourself, or use include, wey resolves each file against its own directory.

The project name dey come from that same base directory. So, if you change which file comes first, you fit rename the project. Renamed project means new container names and new volume names. The old volume still dey for disk under the old name. Pin the name instead with top-level name: for the base file.

name: myapp

Wetin fields merge, and which ones dem replace

Compose dey merge based on the value type, no be the field name.

• Fields wey get one value dey replace. image, command, entrypoint and mem_limit go take the later value directly. You no fit append one argument to command, because the override dey rewrite the whole line.
• Mappings dey merge key by key. environment, labels, volumes and devices keep every key from both files, and the later file win for any key wey dey inside both. For environment and labels, the key na the variable or label name. For volumes and devices, the key na the container path.
• Sequences dey append. dns, dns_search, expose, tmpfs and external_links dey join together. If base get expose: ["3000"] and override get ["4000", "5000"], the result go produce ["3000", "4000", "5000"].

Four sequences get identity key, so entries wey match for that key dey merge instead of appending. volumes, secrets and configs dey match for target. ports dey match based on the combination of ip, target, published and protocol.

Read that ports rule two times, because na there the trap dey. Two port entries na the same entry only when all four parts match. If you change any one of dem, Compose go see another unrelated port, so e go keep both.

Why your port still dey published after the override

A base file wey publish service for every interface:

services:
  web:
    image: nginx:1.27
    ports:
      - "8080:80"

An override wey bind am to localhost only, because reverse proxy go sit for front of am:

services:
  web:
    ports:
      - "127.0.0.1:8080:80"

Check the result before you assume say e work.

docker compose -f compose.yaml -f compose.prod.yaml config

Both entries dey for the output. The ip part no be the same, 0.0.0.0 against 127.0.0.1, so merge see dem as two different ports, and the public binding wey you try remove still dey for the model. This matter pass for Docker, because published port dey write into iptables before your firewall rules. The mechanism dey explained for why published Docker ports dey pass ufw.

Two fixes dey. The explicit one na the !override tag. E replace the whole attribute and bypass the merge rules:

services:
  web:
    ports: !override
      - "127.0.0.1:8080:80"

!override need Compose v2.24.4 or newer. The portable fix no need any tag: remove ports completely from the base file and declare am only for environment-specific files. If nothing dey merge, nothing fit leak. Na this pattern the worked example below use.

Base file wey set value, how to delete am

!reset dey remove one attribute and return am to default value or null. E dey take one value but e no use the value, so write any valid empty value.

services:
  web:
    ports: !reset []
    environment:
      DEBUG: !reset null

!reset need Compose v2.24 or newer. Use am when base file no be your own to edit, like vendor fragment wey you pull in. Published upstream stack na exactly this kind case: Compose file behind self-hosted AFFiNE workspace declare four containers wey you no write, and !reset let you clear one attribute for one of dem without forking the file or taking responsibility to track am.

include, for stacks wey dem assemble from parts

include dey pull another Compose application enter your model. E na top-level element, no be flag.

include:
  - path: ../commons/compose.yaml

Every path for include dey load as im own Compose application model, with im own project directory. So relative paths inside that file go resolve against that file own directory. Na this be the real difference from -f, and na why include be the correct tool when the fragment dey another folder or another repository. Na so vendor stack wey you no write usually be: the multi-service Compose file behind self-hosted Authentik SSO install fit dey for im own directory with im own relative paths intact, while your file remain about your own services.

The long form get sub-options.

include:
  - path:
      - ../monitoring/compose.yaml
      - ../monitoring/compose.vps.yaml
    project_directory: ../monitoring
    env_file: ../monitoring/.env

path fit accept list, and dem go merge those files together with the normal rules before the result join your model. project_directory dey set the base path wey dem use resolve relative paths for the included file. env_file dey give the included file im own variables for interpolation, so shared fragment no go quietly read your project .env. include need Compose v2.20.0 or newer. These same options fit work for one-container add-on to stack wey you already dey run, like Halcyon, wey dey reskin Jellyfin library as 90s rental store: im file keep im own image tag and im own env_file, so upgrading am no mean say you must touch the file wey your media stack dey use.

If your file and included file get duplicate resource names, Compose go report error instead of quietly merging dem, and na deliberate behaviour. If you wan change wetin included file declare, put the change for compose.override.yaml: Compose go apply the override to the assembled model, so e fit touch included resources without colliding with dem. This habit dey useful pass for stack wey upstream file dey rewrite for every release, like the multi-container photo servers wey dem compare for PhotoPrism versus Immich, where localhost binding or extra volume suppose dey your override, no be inside the file wey next upgrade go replace.

The short version: include dey compose separate applications, while -f dey layer configuration onto one application.

Dev and prod split for one VPS

Na here the full pattern dey for three files. The base file dey declare wetin dey true everywhere, and e no publish any port at all.

name: myapp

services:
  app:
    image: ghcr.io/example/app:1.4.2
    environment:
      DATABASE_URL: postgres://app:${POSTGRES_PASSWORD}@db:5432/app
      LOG_LEVEL: info
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: app
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  db_data:

Na the depends_on condition make the app wait for database wey dey answer, instead of container wey just dey exist. healthchecks and depends_on conditions explain this. POSTGRES_PASSWORD dey interpolate from the project .env file, and that file no suppose dey git. See env files and Compose secrets for safer options.

Next, compose.override.yaml. Compose dey load this file by itself. Na the developer file be this.

services:
  app:
    build: .
    command: npm run dev
    environment:
      LOG_LEVEL: debug
    ports:
      - "3000:3000"
    volumes:
      - ./src:/app/src

  db:
    ports:
      - "127.0.0.1:5432:5432"

For laptop, bare docker compose up go merge both files. command go replace the image default because e get only one value. LOG_LEVEL go replace info because environment dey merge by key. The bind mount and the two published ports na additions only. The database port dey bind to localhost, so laptop wey dey shared network no go expose PostgreSQL to everybody for the room.

Last, compose.prod.yaml. Compose no dey look for this name, so e no go load am by mistake.

services:
  app:
    ports:
      - "127.0.0.1:8000:3000"
    deploy:
      resources:
        limits:
          memory: 512M

For the VPS, you name both files. Na this naming exactly exclude the override.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml up -d
docker compose -f compose.yaml -f compose.prod.yaml ps

ps suppose list both services as running, while db dey show (healthy). Because you pass -f, compose.override.yaml no read. So the dev command, source bind mount, and public port 3000 no fit reach production, even though the file dey the same directory. Port 8000 dey localhost only and ready for proxy. See running several apps behind Traefik when you add the second service.

Set COMPOSE_FILE=compose.yaml:compose.prod.yaml inside the server's .env. After that, the rest of your commands go return to plain docker compose logs -f app.

A single-service stack get the same structure, because a self-hosted openGym workout tracker must answer through TLS behind a proxy before you enrol the first passkey. A base file wey no get ports inside am na wetin stop stray public binding from reach the service before the proxy.

Read the merged model before you deploy

docker compose config dey print the fully merged, fully interpolated model. E no be preview. Na the exact input wey Compose go act on, so when the output no agree with wetin you expect, na the output correct.

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose -f compose.yaml -f compose.prod.yaml config --no-interpolate
docker compose -f compose.yaml -f compose.prod.yaml config --services

--no-interpolate leave ${VAR} unexpanded. Use am before you paste the output anywhere, because plain config dey print every resolved secret as clear text. --services dey list only the service names, and this na quick way to confirm say an include pull in wetin you expect.

Failure modes, and wetin you go see

no configuration file provided: not found. Compose no find anything to read. You dey outside the project directory, or COMPOSE_FILE name a path wey no dey exist. Compose dey search parent directories for the default base file, but e no dey search anywhere for file wey you name by yourself.

WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. Interpolation dey resolve against the project .env file and the shell environment, and the project directory here na the directory of the first -f file. If you deploy from directory wey different from the one wey hold .env, you go see this warning, then database wey dey reject every connection.

Your override edit no show for docker compose config. Either you pass -f, wey dey turn off automatic override loading, or Compose find compose.yaml for parent directory and your override file no dey beside am. If you run docker compose config without any other arguments, e go tell you the model wey Compose really dey build.

Bind mount empty, and Docker create directory wey you no request. The relative path resolve against the first file directory. Correct the path, pass --project-directory, or move the fragment behind include.

Containers come back with new names, and volume look empty. The project name don change, because the project name follow the first file directory. Add top-level name: to the base file, and the name no go dey change again. The old volume still dey under the old prefix, and docker volume ls go show am.

Port wey you remove for the override still dey open. The ports merge append instead of replace. Confirm am with docker compose config, then either use !override or move ports comot from the base file.

FAQ

Compose dey load compose.override.yaml by itself?

Yes, when you run docker compose without -f flag. Compose dey search the working directory and its parent directories for compose.yaml or docker-compose.yaml. If override file dey beside am, Compose go load that file second. The recognised names na compose.override.yaml, compose.override.yml, docker-compose.override.yml and docker-compose.override.yaml. If you pass any -f, this behaviour go stop, so docker compose -f compose.yaml up go read only one file.

Which order Compose dey use to merge multiple -f files?

Na from left to right. Compose dey build the configuration according to the order wey you supply the files. Each file dey override and add to the files before am, so the last file for the command line wins any conflict. You must use the same list for every command for that project. Na COMPOSE_FILE=compose.yaml:compose.prod.yaml dey provide this.

Why port still dey published after I override am?

Because ports entries dey identify with the complete set of ip, target, published and protocol. Override of 127.0.0.1:8080:80 against base value of 8080:80 get different ip part. Because of this, Compose dey treat am as another port and keep both. Run docker compose config and you go see the two entries. Use ports: !override for Compose v2.24.4 or newer, or remove ports from the base file so Compose no get anything to merge against.

Wetin be the difference between include and -f?

-f dey layer several files onto one application. Every relative path for every file dey resolve against the first file directory. include dey bring in a separate Compose application. Each included path keep its own project directory, so its relative paths dey resolve against itself. Use -f for environment layers wey belong to your stack, and include for fragment wey somebody maintain for another place. include need Compose v2.20.0 or newer.

How I fit remove value wey base file set?

Use !reset tag for Compose v2.24 or newer. Write ports: !reset [] or MY_VAR: !reset null for the overriding file, and the attribute go return to its default or to null. The value wey you give the tag na required, but Compose go ignore am. If you want replace an attribute instead of clearing am, !override go do that. E need v2.24.4 or newer.