Back up to two restic repositories

What two restic repositories give you

A second restic repository is a second copy of your data that fails on its own. You create it with its own restic init run and its own password. Snapshot IDs inside it have no meaning in the first repository, and neither repository knows the other exists unless you tell it.

This is the 3-2-1 rule (three copies of your data, on two kinds of storage, with one copy off site) written as a scheduling problem. You already have one repository that works, so the job is to add a destination without adding a way to lose both at once. This guide assumes you already run a working restic backup on a VPS, and that the new destination is a VPS used as an offsite backup target reached over SFTP or the restic REST server.

There are two honest designs. Back up twice from the source, once into each repository. Or back up once and use restic copy to move snapshots from the first repository into the second. Both are below, with what each one costs.

Why copying the repository directory is the trap

The tempting shortcut is rsync -a /srv/restic/local/ user@host:/srv/restic/copy/. It is fast and it needs no new password. It is also not a second repository. It is the same repository in two places, with the same repository ID and the same keys.

restic stores each unique chunk of your data once, so a single damaged pack file can be referenced by every snapshot that contains that data. Copy the directory and you copy the damaged pack file with it. Run restic check against the copy and it reports what the original reports, because it is reading the same bytes. A mistake travels the same way: a restic forget --prune that removed more than you meant is mirrored to the copy on the next sync run.

The password is the quieter problem. One password file opens both locations, so a leaked key is a leaked key for every copy you hold.

A directory copy is still worth something. It protects you against the disk under the first repository dying. It does nothing about anything that happens inside the repository, which is where most real losses come from. If that distinction is new, the difference between provider snapshots and real backups is the same argument at the level of the whole server.

Design A: back up twice from the source

Create each repository on its own, with its own key.

sudo install -d -m 700 /etc/restic
sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /etc/restic/local.pass'
sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /etc/restic/offsite.pass'
sudo chmod 600 /etc/restic/local.pass /etc/restic/offsite.pass

sudo restic init --repo /srv/restic/local --password-file /etc/restic/local.pass
sudo restic init --repo sftp:backup@203.0.113.10:/srv/restic/offsite \
  --password-file /etc/restic/offsite.pass

Set the mode explicitly instead of trusting umask alone, because sudo applies its own umask to the file the shell creates under it. Copy both passwords into a password manager before you go further. A repository whose password exists only on the machine being backed up cannot be opened after that machine is gone, which is the exact situation you are preparing for.

Then run the backup once per repository.

sudo restic --repo /srv/restic/local --password-file /etc/restic/local.pass \
  backup /etc /srv /home --exclude-file=/etc/restic/excludes.txt

sudo restic --repo sftp:backup@203.0.113.10:/srv/restic/offsite \
  --password-file /etc/restic/offsite.pass \
  backup /etc /srv /home --exclude-file=/etc/restic/excludes.txt

Run restic snapshots against each repository. Each should list a snapshot from today, and the two snapshot IDs will not match. That is correct: a snapshot ID is a hash over the snapshot's contents, including the moment it was taken, and the two runs happened at different moments. Never carry an ID you read in one repository over to the other.

What this design costs: the source files are read and encrypted twice, and the new data is uploaded twice. Deduplication happens inside a repository, so neither repository ends up larger than it would be alone, but your CPU and your uplink do the work two times. On a busy host, or on a connection where the first backup already takes hours, that is the deciding number.

What it buys: nothing is shared. If the offsite host is unreachable, the local backup still runs and still succeeds. If the offsite repository is damaged, the local one is untouched, because each was written from the source independently. Retention becomes independent without any extra work.

Design B: back up once, then use restic copy

restic copy transfers snapshots from one repository into another. Initialise the second repository with the chunker parameters of the first.

sudo restic init \
  --repo sftp:backup@203.0.113.10:/srv/restic/offsite \
  --password-file /etc/restic/offsite.pass \
  --from-repo /srv/restic/local \
  --from-password-file /etc/restic/local.pass \
  --copy-chunker-params

The chunker parameters decide where restic splits a file into chunks. Every restic init picks its own set, so two repositories created separately cut the same file in different places. copy does not re-chunk the data it transfers, so without --copy-chunker-params the copied data will not deduplicate against data the destination already holds, and the destination grows more than it should. The flag only works at init time, so decide before you create the repository.

Then copy, after the local backup has finished.

sudo restic --repo sftp:backup@203.0.113.10:/srv/restic/offsite \
  --password-file /etc/restic/offsite.pass \
  copy --from-repo /srv/restic/local \
  --from-password-file /etc/restic/local.pass --verbose

The two repositories have different keys, so restic reads and re-encrypts everything it transfers. Each copied snapshot is saved under a new ID in the destination, and the output names both the new snapshot and the source snapshot it came from. Later runs skip snapshots that were copied before, and --verbose makes restic say which ones it skipped, so running copy twice in a row should transfer nothing the second time. That is your check that the skip logic is working.

What this design costs: the source is read once, and the transfer runs between the two repositories. Where the first repository lives decides the bill. A local disk is free to read, while object storage charges egress on every byte the copy pulls out. The processor work does not disappear, since decrypting and re-encrypting happens per byte.

What it risks: the second repository can only hold what the first one holds. An exclude pattern that skipped a directory, or a backup that never ran, is missing from both. You are testing one backup job instead of two.

Data damage does not spread quietly through copy. restic names every blob by the hash of its contents, so data that does not match its hash fails to load and the run reports an error rather than writing it into the destination. Judge the run by its exit status, not by the last line on screen: check echo $? while you are testing by hand, and the unit result once it is scheduled.

Pick design A when the source can afford to be read twice and you want the two repositories fully independent. Pick design B when reading the source is the expensive part, or when both repositories must hold the same set of snapshots. If you are still choosing the tool underneath all this, the comparison of restic and BorgBackup covers where each one fits.

One systemd unit per repository

The scheduling rule is short: the offsite job must not be able to fail the local job. One unit that runs both backups in a single ExecStart stops at the first failure, so an unreachable offsite host means no local backup that night. Use one templated unit with one environment file per destination.

Write /etc/restic/local.env:

RESTIC_REPOSITORY=/srv/restic/local
RESTIC_PASSWORD_FILE=/etc/restic/local.pass

And /etc/restic/offsite.env:

RESTIC_REPOSITORY=sftp:backup@203.0.113.10:/srv/restic/offsite
RESTIC_PASSWORD_FILE=/etc/restic/offsite.pass

Both files point at a secret, so sudo chmod 600 /etc/restic/*.env. Now /etc/systemd/system/restic-backup@.service:

[Unit]
Description=restic backup to %i
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/%i.env
ExecStart=/usr/bin/restic backup /etc /srv /home --exclude-file=/etc/restic/excludes.txt
Nice=10
IOSchedulingClass=idle

And /etc/systemd/system/restic-backup@.timer:

[Unit]
Description=restic backup timer for %i

[Timer]
OnCalendar=*-*-* 01:30
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

A templated timer starts the service with the same instance name, so restic-backup@offsite.timer starts restic-backup@offsite.service. Enable one instance per destination and give the offsite one a later hour, so the two runs do not compete for the same disk and the same uplink.

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup@local.timer
sudo systemctl enable --now restic-backup@offsite.timer
sudo systemctl start restic-backup@offsite.service

For design B the offsite instance runs a copy instead of a backup. restic reads the source repository from RESTIC_FROM_REPOSITORY and RESTIC_FROM_PASSWORD_FILE, so put those two lines in /etc/restic/offsite.env next to the destination variables, and write a separate restic-copy.service whose ExecStart is /usr/bin/restic copy --verbose. Schedule its timer an hour or more after the local backup. If it ever starts early it copies whatever exists at that moment, which delays a snapshot by one day rather than breaking anything.

Check the schedule and the last result:

systemctl list-timers 'restic-*'
journalctl -u restic-backup@offsite.service -n 50 --no-pager

list-timers should show both instances with a next run time. The journal shows how the last run ended. Neither is the real proof. A unit can exit 0 and still have backed up nothing you care about if the paths were wrong, so the check that counts is restic snapshots against each repository, showing a snapshot from the expected time with the expected paths.

A backup job that fails silently for two months is the normal way this goes wrong. Add OnFailure= to each service, pointing at a unit that notifies you somewhere you actually read, such as a self-hosted ntfy server for push notifications. Add a second check that alerts when the newest snapshot is older than expected, because a timer that never fires never fails either.

Two retention policies, one per repository

restic forget works on one repository at a time, so each destination gets its own policy. The usual split keeps more short-term history locally, where space is cheap and restores are frequent, and fewer but longer-lived snapshots offsite, where you pay per gigabyte per month.

sudo restic --repo /srv/restic/local --password-file /etc/restic/local.pass \
  forget --keep-daily 14 --keep-weekly 8 --dry-run

sudo restic --repo /srv/restic/local --password-file /etc/restic/local.pass \
  forget --keep-daily 14 --keep-weekly 8 --prune

Run --dry-run first and read the list of snapshots it would remove. forget removes snapshots, prune removes the data that no remaining snapshot references, and --prune runs the second step when the first actually removed something. Give pruning its own weekly timer instead of attaching it to every backup, because prune takes an exclusive lock on the repository, so a backup that starts during it stops with a message that the repository is locked. Do not clear a lock with restic unlock until you have confirmed that no restic process is running against that repository.

One trap belongs to design B alone. copy decides what to transfer by looking at what the destination already has, so a snapshot you forget in the second repository comes back on the next copy run while it still exists in the first. Make the offsite policy the longer one, or narrow what you copy with --tag or explicit snapshot IDs. Confirm the outcome with restic snapshots on the destination after the next copy has run, not on the day you ran forget.

Test a restore from the second repository

The second repository is the one you have never restored from, which makes it the one most likely to surprise you. A repository nobody has restored from is a hope, not a backup. Test it from a machine that is not the source, using the password out of your password manager, because that exercises the credentials and the network path as well as the data.

export RESTIC_REPOSITORY=sftp:backup@203.0.113.10:/srv/restic/offsite
export RESTIC_PASSWORD_FILE=./offsite.pass
restic snapshots
restic restore latest --target /tmp/restore-test --include /etc/ssh

Then look at what came back:

ls -l /tmp/restore-test/etc/ssh
diff -r /etc/ssh /tmp/restore-test/etc/ssh

diff should report only files that changed since the snapshot was taken. A file missing entirely points at an exclude pattern you did not intend, which is worth finding now rather than during a real restore. Restoring as a normal user cannot set ownership, so run the test under sudo if uid and gid matter for the files you picked.

Add an integrity check on a schedule:

restic check
restic check --read-data-subset=5%

Plain check verifies the structure of the repository, that every referenced pack file exists and that the index agrees with it. It does not read your file data. --read-data-subset=5% downloads and verifies part of the data, and that is the version which catches a storage backend returning bad bytes. On an offsite repository each of those bytes is billed, so pick a subset you are willing to pay for every month.

What still fails when you have two repositories

Both repositories are written by one machine that holds both passwords. Anything running as root on that machine can delete both, which is what ransomware does on purpose. The answer is a destination the source cannot delete from. restic's REST server has an --append-only mode, and object storage backends can enforce an object lock, at the cost of not being able to prune inside the lock window. Keep the two repositories with different providers or in different physical sites as well, because one suspended account can otherwise take both copies at the same time.

FAQ

Can I just rsync my restic repository to a second server?

That gives you a second copy of one repository, not a second repository. It has the same repository ID and the same keys, and a damaged pack file is copied along with the good ones. It protects you against the disk dying. It does not protect you against a forget --prune that removed too much or against a corrupted pack file, because both exist identically in both places. Run restic init for the second destination, then either back up twice from the source or use restic copy.

Does restic copy keep the same snapshot IDs?

No. Each copied snapshot gets a new ID in the destination repository, and restic's output names the source snapshot it came from. Later copy runs skip snapshots that are already there, and with --verbose they print which ones were skipped. Find snapshots by date and path with restic snapshots inside the repository you intend to restore from, and never carry an ID from one repository to the other.

Should both repositories use the same password?

No. A single password file that opens both destinations removes much of the point of having two. Generate a separate key for each repository, keep both outside the machine being backed up, and give each systemd unit only the one it needs through RESTIC_PASSWORD_FILE. restic key add adds a second key to a repository, which is how a recovery account gets access without sharing the everyday password.

How do I stop an offsite failure from failing my local backup?

Give each destination its own service and its own timer, so each run carries its own exit status. A single unit that runs both backups in sequence stops at the first error, so an unreachable offsite host also costs you that night's local backup. Confirm with systemctl list-timers 'restic-*' that both instances are scheduled, and verify each one separately with restic snapshots against its own repository.

How often should I test a restore from the second repository?

Every quarter, and again after any change to the units or to the repository location. A restore test is the only check that covers the credentials as well as the data. Restore into a scratch directory with --target, compare the result with diff -r, and run restic check --read-data-subset=5% on whatever schedule the egress cost allows.