How to Use Ansible Vault to Encrypt Git Secrets

Wetin Ansible Vault dey protect, and wetin e no dey protect

Ansible Vault dey encrypt secrets inside your playbook repository, so wetin git store na ciphertext instead of plaintext password. The ansible-vault command fit encrypt whole file or one value inside file, using symmetric key wey come from password wey you choose. Ansible dey decrypt that content for memory when play dey run, so the variable dey behave like any other variable.

That model get one clear boundary. Vault dey protect secret when e dey rest inside repository, and nothing more. Once task run, the value na plaintext for memory, rendered template, module arguments, and run output unless you stop am. Everybody wey fit run the playbook get the vault password, so vault dey give secrecy from people outside the team, but e no give access control per person inside the team.

If you never write playbook yet, start with one first Ansible playbook against VPS and come back when that playbook need password.

Encrypt whole file, or na one string?

ansible-vault encrypt dey replace file with ciphertext. File go turn to one block of base64 text under header line wey start with $ANSIBLE_VAULT. Use am when file no contain anything apart from secrets.

ansible-vault encrypt_string dey encrypt one value and print YAML snippet wey you fit paste inside normal vars file. Variable name go remain readable, and na only the value be ciphertext. Use am when secrets dey together with plaintext settings.

The main difference for everyday work na the diff. Vault file dey re-encrypt with fresh random salt every time you save am, so every byte for ciphertext dey change. git diff go then show one unreadable block replaced with another unreadable block. This mean say reviewer no fit know whether you rotate one password or rewrite the whole file. With encrypt_string, each secret na im own block inside plaintext file, so diff go show exactly which variable change and leave the rest of the file untouched.

Inline form get one cost, and e go show when you rotate secrets: ansible-vault rekey no dey touch inline blocks. Choose file form when secret list long and e no dey change often. Choose inline form when file mix secrets with normal variables and you want code review to show meaningful changes.

The group_vars/<group>.yml layout wey dey show wetin dem protect

Ansible go load group_vars/<group>.yml, and e go also load every file inside group_vars/<group>/ directory. Na the directory form you suppose use, because e allow one group carry plaintext file and encrypted file side by side.

inventory/
  hosts.ini
group_vars/
  all/
    vars.yml
    vault.yml
  web/
    vars.yml
    vault.yml
host_vars/
  db01/
    vars.yml
    vault.yml
playbooks/
  site.yml

Every vault.yml dey encrypted. Every vars.yml na plaintext. Reader fit see which values dem protect without opening anything, because filename dey show am.

The second part of the pattern na indirection. Inside the encrypted file, put vault_ before every variable.

vault_db_password: "a real password"
vault_grafana_admin_token: "a real token"

Then reference those names from the plaintext file wey dey beside am.

db_password: "{{ vault_db_password }}"
grafana_admin_token: "{{ vault_grafana_admin_token }}"

Roles and templates go use db_password, and dem no go know where the value come from. This keep the separation between playbook and role clean. The plaintext vars.yml also dey work as searchable index: grep -r vault_ group_vars/ lists every secret wey repository expect, without decrypting anything. The cost na one extra name for every secret. If you make typo for vault_ name, e go show at run time as undefined variable instead of syntax error.

Encrypt one variable with encrypt_string

ansible-vault encrypt_string --vault-id prod@~/.ansible/vault-prod.txt \
  --stdin-name 'vault_db_password'

Secret type ma, then press Ctrl-D. --stdin-name dey read the value from standard input, so e no go enter your shell history file. The other form dey put the value for command line, where shell dey record am:

ansible-vault encrypt_string --vault-id prod@~/.ansible/vault-prod.txt \
  'a real password' --name 'vault_db_password'

For both cases, command go print one YAML block. Paste am inside vars file exactly as e print, because the indentation under !vault tag na part of the value.

vault_db_password: !vault |
          $ANSIBLE_VAULT;1.2;AES256;prod
          6638643965323633646262656665306333616466396630323136393465356136396436383331
          3131303163306665326539353837343663313762616561306534373963383531613664393332

!vault tag dey tell YAML loader say the scalar na ciphertext, no be ordinary text. The header get format version, cipher, and vault ID label wey encrypt am. Value wey you encrypt without vault ID get 1.1 header without label. E still dey work, but e go give you less information about where the password come from.

Vault password dey live for where?

E dey outside the repository. Na the one rule wey no get exception.

--ask-vault-pass dey ask once for each run and e no store anything. E good for laptop, but e no good for cron job or CI runner.

Password file na plain text file wey password dey for the first line. Create am empty with tight permissions, then fill am inside editor, so password no go enter your shell history:

mkdir -p ~/.ansible
install -m 600 /dev/null ~/.ansible/vault-prod.txt
$EDITOR ~/.ansible/vault-prod.txt

Point any command to am with --vault-password-file:

ansible-playbook -i inventory/hosts.ini playbooks/site.yml \
  --vault-password-file ~/.ansible/vault-prod.txt

To repeat that flag for every command easy to forget, so set am once inside ansible.cfg for the root of the repository.

[defaults]
inventory = inventory/hosts.ini
vault_password_file = ~/.ansible/vault-prod.txt

The same setting dey read from environment variable ANSIBLE_VAULT_PASSWORD_FILE, and na so CI job normally dey provide am. The job writes password from its own credential store into file for temporary directory, exports the variable, then deletes the file when the run finish. Add the filename pattern to .gitignore too, because path inside ansible.cfg dey committed, and sooner or later person go create the real file inside the checkout.

If password file get executable permission, Ansible go run am and read password from its standard output instead of reading the file as text. Na so you fit collect vault password from system keyring or cloud secret manager without writing am to disk at all. Script wey you use through --vault-id get extra requirements: its name must end with -client or with -client plus an extension, e must be executable, e must accept a --vault-id option, and e must print password to standard output.

Vault ID two: staging and production

Vault ID na label wey dem attach to vault password, and dem dey write am as label@source. The source fit be prompt, wey be path to password file, or path to client script. Labels let one repository keep secrets under more than one password, so staging password no go open production file.

ansible-vault encrypt --vault-id staging@~/.ansible/vault-staging.txt \
  group_vars/staging/vault.yml
ansible-vault encrypt --vault-id prod@~/.ansible/vault-prod.txt \
  group_vars/prod/vault.yml

Pass every ID wey one run fit need:

ansible-playbook playbooks/site.yml \
  --vault-id staging@~/.ansible/vault-staging.txt \
  --vault-id prod@~/.ansible/vault-prod.txt

Or list dem once for ansible.cfg:

[defaults]
vault_identity_list = staging@~/.ansible/vault-staging.txt, prod@~/.ansible/vault-prod.txt

One behaviour dey surprise people. By default, the label na hint, e no be lock. Ansible dey try every secret wey e currently get against the file until one decrypt am, so file wey get staging label still fit open if production password happen to be the correct key. Set vault_id_match = True under [defaults], or use environment variable ANSIBLE_VAULT_ID_MATCH, and Ansible go use only the secret wey label match the file header. This check need 1.2 header, so e apply only to content wey dem encrypt with vault ID from the beginning.

When more than one ID dey loaded, ansible-vault encrypt no longer know which password to use for encryption. Name the ID with --encrypt-vault-id prod, or set vault_encrypt_identity inside ansible.cfg so repository get default.

The main benefit na deployment scope. CI job wey dey deploy staging go receive staging password only, so if attacker breach the runner, e no fit read production credentials. Once you dey run plays across plenty Linux servers from one control machine, this separation fit be the difference between small incident and very serious one.

Rekey the vault when person leave

Rekeying dey change vault password and re-encrypt the content with the new one. E no dey undo anything. Anybody wey don ever hold the old password fit still decrypt any repository copy wey dem keep, including every old commit inside that copy. So treat vault password as exposed immediately the holder leave, then rotate am for this order.

1. Change the real credentials for the servers and third-party services. Na this step actually revoke access.
2. Put the new values inside the vault files with ansible-vault edit.
3. Rekey every encrypted file with a new vault password.
4. Give the new vault password to people wey still need am, through channel wey no be the repository.

ansible-vault rekey --vault-id prod@~/.ansible/vault-prod-old.txt \
  --new-vault-id prod@prompt \
  group_vars/prod/vault.yml host_vars/db01/vault.yml

rekey fit accept several files for one command, and --new-vault-id prod@prompt go ask for the new password once instead of reading am from disk. Keep the same label unless you get reason to change am, because the label dey written inside the header of every file wey the command rewrite.

Na here inline form dey cost you. ansible-vault rekey dey work on fully encrypted files, so any !vault block wey dey inside plaintext vars file go remain unchanged. Find dem first, then regenerate each one with encrypt_string under the new password:

grep -rl '!vault' group_vars/ host_vars/

Na the complete trade-off be this. Inline blocks give you diffs wey you fit read, but dem make you do manual pass during rotation. Fully encrypted files fit rotate with one command, but review no go show you anything useful.

Why secret still dey appear for your output

Vault don finish as soon as value don decrypt. Ansible dey report task result, and module wey dey echo its arguments go carry credential enter that report. Verbose run, a --diff for template task, failed task wey dump its arguments, or callback plugin wey write output to file, all of dem fit hold plaintext. Encrypting the file no solve any of dem.

no_log: true na the switch. Set am for any task wey receive credential.

- name: Write the application environment file
  ansible.builtin.template:
    src: app.env.j2
    dest: /etc/myapp/app.env
    owner: myapp
    group: myapp
    mode: "0600"
  no_log: true

Ansible go then hide result of that task from output. The log go record say task run, but e no go record wetin task handle. Set am for loops especially, because loop dey report one result for each item. Loop over credential list go therefore report the whole list.

Four other places decrypted secret fit escape. None of dem dey covered by no_log:

• File wey template render inherit the mode and owner wey you give am. Set mode: "0600" and specific owner for anything wey hold credential. Otherwise secret go become world readable for target host.
• Secret wey you pass to ansible.builtin.command or ansible.builtin.shell go appear for process list on target host while command dey run. Any local user fit read am. Pass am through file or environment variable instead.
• Fact caching dey write gathered facts to disk for control machine. So registered variable wey hold secret fit end up inside cache file wey nobody dey treat as sensitive.
• The same secret usually dey another place too, like environment file wey container dey read. Rules for that place separate, and how to keep credentials out of Compose env files cover that side.

no_log dey make debugging harder, and na exactly why dem create am. Remove am temporarily for test host when task get problem. Put am back before the change reach production.

Read and edit encrypted files without leaving plaintext behind

ansible-vault view group_vars/prod/vault.yml dey decrypt into pager and e no write anything for disk. ansible-vault edit dey decrypt into temporary file, open your $EDITOR, then encrypt am again when you close am. Prefer both instead of ansible-vault decrypt, wey dey leave plaintext file for working tree. If decrypted vault file enter staging by mistake, na the most common way real credential dey reach public repository.

Git fit render readable diff for files wey fully encrypted by decrypting dem as e dey go:

git config --local diff.ansible-vault.textconv "ansible-vault view --vault-password-file ~/.ansible/vault-prod.txt"
printf '%s\n' 'group_vars/**/vault.yml diff=ansible-vault' >> .gitattributes

Understand wetin this one dey do before you enable am. git diff go now print production secrets for your terminal, and this one go put dem for scrollback and any screen share. Na local convenience for one person on one machine, so keep the git config local, and expect other people's checkouts to behave differently unless dem set the same thing up.

When vault no be the correct tool again

Vault na file format wey get one password for each label, and this structure decide where e no fit continue. Move go a real secret store when any of these conditions apply.

• You need access for each person separately. Everybody wey run the playbook dey hold the same password, and vault IDs fit split access by environment only, never by person.
• You need audit trail. Vault no record who decrypt wetin, or when e happen.
• You need rotation according to schedule. Vault no get expiry or versioning, so nothing go tell you say credential never change for two years.
• The application itself need the secret during run time. Service wey dey read its database password during boot no suppose dey read am from your deployment repository.

The pattern go turn around from there. Ansible go stop to store secrets and start to fetch dem during run time through a lookup plugin, against HashiCorp Vault (another product wey get almost the same confusing name), a cloud provider's secret manager, or a keyring for the control machine. The repository go hold path, the store go hold value, and the store go keep access log. For small team, self-hosted password manager wey get API, like one Vaultwarden server, fit handle the same work for smaller setup.

One credential dey outside all this. The SSH key wey your control machine dey use to reach the servers no be vault problem, because Ansible need am before any play fit run. Handle am with agent and passphrase, based on the basics of SSH key management.

FAQ

I suppose encrypt the whole vars file or just the secret string?

Encrypt the whole file when e no hold anything except secrets, because one command go rotate everything and the layout go remain simple. Use ansible-vault encrypt_string when secrets dey beside ordinary variables, because na only the encrypted value go change for diff and reviewer fit see which variable dem touch. The trade-off na rotation. ansible-vault rekey covers whole files and leaves inline !vault blocks untouched, so you must regenerate those by hand with the new password.

Where I suppose store the Ansible Vault password file?

Store am outside the repository, with mode 0600, for path like ~/.ansible/vault-prod.txt. Point to am with --vault-password-file, or set vault_password_file under [defaults] inside ansible.cfg, or set ANSIBLE_VAULT_PASSWORD_FILE for the environment. For CI, make the job write the password from its own credential store into temporary file, export the variable, then delete the file when the job finish. If the file get execute permission, Ansible go run am and read the password from standard output. This one let you source am from keyring instead of storing am for disk.

How I fit use different vault passwords for staging and production?

Give each password label with --vault-id staging@/path/to/file and --vault-id prod@/path/to/file, then encrypt each environment files under its own label. Pass both IDs when you run am, or list dem inside vault_identity_list under [defaults]. By default, Ansible go try every secret wey e hold until one decrypt the file. So set vault_id_match = True if you want make e try only the secret wey label match the file header. When several IDs dey loaded, choose the one wey encrypt am with --encrypt-vault-id.

Ansible Vault dey stop password from showing for run output?

No. Vault dey protect the secret only when e dey at rest inside the repository. Once task run, the value don become plaintext, and verbose run or failed task fit carry am enter the log. Add no_log: true to every task wey handle credential. Set restrictive mode and owner on any file wey you template out. Avoid passing secrets as command arguments, because dem dey visible for the process list on the target host while the command dey run.