VPS snapshots vs backups vs clones

What a snapshot, a backup, and a clone actually are

A VPS snapshot is a disk image of your server, held by your provider, on your provider's infrastructure, inside your account. A backup is an independent copy of your data that you can restore somewhere else, with no help from the provider that held the original. A clone is a new instance deployed from a snapshot, so it starts life as an exact copy of the original, identity included.

They solve different problems. A snapshot rolls a broken upgrade back in minutes, and it does nothing for a closed account. A backup survives the provider going away, and it takes longer to restore because you rebuild the machine first. A clone gives you a second running server in one step, and it also gives you two machines that believe they are the same machine.

Why a VPS snapshot is not a backup

The problem is the failure domain, not the quality of the image. A snapshot sits on your provider's storage platform, usually in the same region as the server it came from, and always inside the same account. One event can take the server and its snapshot together.

• The account is suspended, a payment fails, or someone steals the login.
• A person or a script with API access deletes the instance. On many providers, deleting an instance deletes its snapshots with it. Read your provider's documented behaviour before you assume otherwise.
• The region has a bad day and everything in it is unreachable at once.
• Something running as root on the server finds the provider API token you left in /root, and removes the snapshots before it touches the disk.

A backup is the copy that survives all four. The test is one question: if your provider account stopped existing this afternoon, what could you still restore, and where would you restore it? Anything that fails that question is a rollback tool. Keep taking snapshots, because nothing restores faster. Then keep a second copy on storage your provider does not control.

The old rule still holds: three copies of the data, on two kinds of storage, one of them off the platform. A provider snapshot plus a restic backup repository on separate infrastructure covers that with two moving parts.

Why a snapshot of a running database can restore broken

A provider snapshot copies the block device as it is at one instant. It does not ask your applications to stop first, and it cannot see anything still sitting in the page cache. So the image is crash-consistent at best. It looks exactly like the disk would look if someone pulled the power cable.

Most of the stack handles that. ext4 and XFS replay their journal at mount, so the filesystem comes up. PostgreSQL replays its write-ahead log on start, and the log says so:

LOG:  database system was not properly shut down; automatic recovery in progress

InnoDB does the same and prints its own crash recovery lines during startup. That recovery is the database working as designed, so a single-volume snapshot of a quiet PostgreSQL or MySQL usually restores fine.

The cases where crash-consistent is not enough are real, and they are the ones that hurt. If your data spans two volumes, the root disk and a separate data disk are snapshotted at different instants, so the data files and the log directory can disagree and recovery has nothing correct to replay. Any file an application writes without calling fsync, such as a half-received upload or a queue file, can come back truncated. Anything the application holds in memory and flushes on a timer is simply not in the image.

So write a dump to disk before you take the snapshot. Then the image contains one file you know is internally consistent, whatever state the live data files are in.

sudo -u postgres pg_dumpall --clean --file=/var/backups/pg-$(date +%F).sql
sudo mysqldump --single-transaction --routines --all-databases > /var/backups/mysql-$(date +%F).sql

--single-transaction gives a consistent dump of InnoDB tables without blocking writers, because the dump runs inside one repeatable-read transaction. It does not cover MyISAM tables, which need a lock or a stopped server. Check the dump is not empty and not truncated before you trust it: tail -n 1 /var/backups/mysql-$(date +%F).sql on a complete mysqldump ends with a Dump completed comment.

If you have a separate data volume, you can freeze it for the seconds the snapshot needs:

sudo fsfreeze -f /srv
# take the snapshot from your provider's panel or API
sudo fsfreeze -u /srv

Freeze a data volume only. Never freeze /. A frozen root filesystem blocks every write on the box, including the shell you would use to type the unfreeze command, so you lock yourself out and wait for a hard reset.

The offsite half: restic or Borg

The snapshot is the fast half. The offsite copy is the half that survives your provider. restic is a good default because it deduplicates, encrypts client side, and writes to S3-compatible object storage, SFTP, or a plain directory. A storage VPS as the offsite target works well here, because backup repositories want capacity rather than IOPS.

sudo apt update && sudo apt install -y restic
sudo sh -c 'umask 077; head -c 24 /dev/urandom | base64 > /root/.restic-pass'
sudo chmod 600 /root/.restic-pass
sudo cat /root/.restic-pass

Copy that passphrase into a password manager now, on a device that is not this server. A restic repository cannot be opened without it and there is no recovery path. If the only copy of the password was on the box you just lost, the backup is encrypted noise.

export RESTIC_REPOSITORY="s3:https://s3.example.com/vps-backups"
export RESTIC_PASSWORD_FILE=/root/.restic-pass
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
sudo -E restic init
sudo -E restic backup /etc /srv /var/backups
sudo -E restic snapshots

sudo -E keeps those variables, since without it root gets a clean environment and restic reports that no repository location was specified. restic snapshots should list the run you just made, with its host and paths. Verify the repository itself on a schedule, and read some of the data back rather than only checking the structure:

sudo -E restic check --read-data-subset=5%
sudo -E restic restore latest --target /tmp/restore-check
sudo -E restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

An untested backup is a guess. Restore to a different VPS at least once, time it, and write the time down, because that number is your real recovery target. Borg is the other solid choice and stores its repository over SSH instead of object storage; the trade-offs are covered in the restic and BorgBackup comparison.

What to fix before a cloned VPS goes near production

A clone is an exact replica. That is the selling point and the problem. Everything that made the original unique is duplicated, and duplicates collide.

Regenerate the SSH host keys. The clone carries the original's /etc/ssh/ssh_host_* files, so two servers present the same host identity. Anyone who controls one can impersonate the other to every client that has accepted that key, and SSH raises no warning, because the key is the one the client expected.

sudo rm -f /etc/ssh/ssh_host_*
sudo ssh-keygen -A
sudo systemctl restart ssh
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

ssh-keygen -A writes a fresh key of every type the daemon expects. The fingerprint from the last command must differ from the one on the original. Your current session survives the restart, because restarting sshd does not close established connections. Do this before anyone connects to the clone. If you leave it until later, every client that already trusted the inherited key gets WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! and has to run ssh-keygen -R <host> first.

Reset the machine ID. /etc/machine-id is a unique identifier that systemd generates once, on first boot, and the clone inherits it.

sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
sudo reboot

An empty /etc/machine-id tells systemd to generate a new value on the next boot, which is why you truncate the file instead of deleting it. Two things break while it is duplicated. On images that take their address by DHCP, systemd-networkd derives its DHCP client identifier from the machine ID by default, so both clones ask for a lease as the same client and the server hands them the same address. And journald stamps every entry with the machine ID, so a central log collector files both servers under one machine. Run cat /etc/machine-id after the reboot and confirm the value changed.

Change the hostname.

sudo hostnamectl set-hostname web-02
grep 127.0.1.1 /etc/hosts

hostnamectl writes /etc/hostname and applies the name immediately. It does not touch /etc/hosts, so edit the 127.0.1.1 line to match. Skip that and the new name resolves nowhere, so every sudo call waits on a failed lookup and prints sudo: unable to resolve host web-02: Name or service not known.

Rotate every credential baked into the image. The clone holds the original's secrets, and now two machines can act as the original. Work through the SSH authorized_keys files, provider and DNS API tokens, application .env files, database passwords, TLS private keys, monitoring enrolment tokens, and the restic repository password. This finds most of them:

sudo grep -rIlE 'PASSWORD|SECRET|TOKEN|API_KEY' /etc /srv /opt /home 2>/dev/null
sudo find / -name '.env' -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null

If the clone is a test copy that will never serve traffic, revoke rather than rotate. A staging box holding a live production API token is a production box with worse patching.

Turn off the jobs that now run twice. Two servers running the same crontab hit the same external systems at the same minute.

systemctl list-timers --all
sudo crontab -l
sudo ls -l /etc/cron.d /etc/cron.daily

The restic case is worth spelling out, because it corrupts your retention rather than only failing loudly. restic tags each snapshot with the host name, and restic forget --keep-daily 7 applies its policy per host. Two machines reporting the same host name are treated as one host, so seven "daily" snapshots can all come from the clone while the original's snapshots are pruned away. Fix the hostname before the first backup run, or stop the timer on the clone. The certbot case is simpler: two servers renewing the same names hit the certificate authority's duplicate certificate rate limit, and the losing run fails with an error about too many certificates already issued for that exact set of names. A clone whose domain still points at the original cannot pass an HTTP challenge anyway, so disable renewal there.

Deal with the monitoring agent. Most agents identify by hostname or by an ID file written at install time, so two agents reporting as one host interleave their metrics into a single series. CPU graphs then show values that no single machine produced, and alerts flap. Stop and remove the agent on the clone, or re-enrol it under the new hostname using your vendor's documented procedure.

Check the network configuration for the original's address. If the image carries a static address in netplan, the clone claims an IP that belongs to another machine.

ip -br addr
sudo grep -r addresses /etc/netplan/

Clear cloud-init state if this clone becomes a template.

sudo cloud-init clean --logs

That removes cloud-init's state under /var/lib/cloud, so the next boot runs the first-boot modules again, which includes generating SSH host keys when none are present. Some versions also offer a flag to reset the machine ID. Run cloud-init clean --help on your own image to see what yours supports rather than trusting a flag list from elsewhere.

When to use which

Rolling back a risky upgrade: take a snapshot. Take it minutes before the change, run the upgrade, and restore the image if it goes wrong. Restoring throws away every write since the snapshot, so on a server taking live traffic, dump the database first and know exactly what window you would lose. For a do-release-upgrade on a box you can take offline for ten minutes, a snapshot is the whole plan.

Migrating to a bigger plan: deploy a clone. Build the clone from a snapshot onto the larger plan, work through the identity list above, then test it on its own IP before any traffic moves. Lower the DNS TTL a day ahead so the cutover is quick, and keep the original running until the new box has held real traffic. Confirm the larger plan is actually faster for your workload first, using the same benchmark method on both servers, because more vCPUs on busier hardware is not always an upgrade.

Building a template: snapshot a cleaned machine. Install and harden one server, then remove everything unique before you image it. No host keys, an empty machine ID, no personal authorized_keys, no credentials, cloud-init cleaned. Snapshot that. Every instance deployed from it generates its own identity on first boot, so the checklist above stops being a checklist. Pair it with the standard first ten minutes on a new VPS so the template already contains the work you would otherwise repeat.

FAQ

Is a VPS snapshot a backup?

No, because it shares a failure domain with the server it came from. The snapshot sits on your provider's storage, in your account, usually in the same region. An account suspension, a stolen API key, or an accidental instance deletion can remove the server and its snapshots in one action, and on many providers deleting an instance deletes its snapshots by design. A snapshot is the fastest rollback you have, so keep taking them, and keep a second encrypted copy on infrastructure your provider does not control.

Do I need to stop my database before taking a snapshot?

Not always, but you do need to accept what you get. A provider snapshot is crash-consistent, meaning the image matches what the disk would look like after a power cut. PostgreSQL and InnoDB recover from that on start, and PostgreSQL logs database system was not properly shut down; automatic recovery in progress while it does. Recovery is not guaranteed when your data spans two volumes that were snapshotted at different instants, or when an application writes without fsync. Write a pg_dumpall or a mysqldump --single-transaction to disk first, so the image contains one file you know is consistent.

Why do two cloned servers fight over the same IP address?

Because they share /etc/machine-id. On images that use DHCP, systemd-networkd builds its DHCP client identifier from the machine ID by default, so both clones request a lease as the same client and the DHCP server offers both the same address. Truncate /etc/machine-id to zero bytes, remove /var/lib/dbus/machine-id, symlink it back to /etc/machine-id, and reboot so systemd generates a new value. The other common cause is a static address written into /etc/netplan/, which the clone copied verbatim; check with ip -br addr.

What is the fastest way to check a clone is safe to put in production?

Compare four things against the original. Run ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on both and confirm the fingerprints differ. Run cat /etc/machine-id on both and confirm the values differ. Run hostnamectl status and confirm the name is new and resolves, so sudo does not warn. Then run systemctl list-timers --all and stop every timer that talks to a shared system, such as backups, certificate renewal, or a monitoring agent, until you have decided which machine owns that job.