VPS snapshot, backup, or clone: which one better?

Wetin snapshot, backup, and clone really be

VPS snapshot na disk image of your server wey your provider dey keep for the provider infrastructure, inside your account. Backup na independent copy of your data wey you fit restore for another place, without help from the provider wey hold the original. Clone na new instance wey dem deploy from snapshot, so e start as exact copy of the original, including the identity.

Dem dey solve different problems. Snapshot fit roll broken upgrade back within minutes, but e no help if dem close your account. Backup fit survive if the provider stop operation, but restore go take longer because you first rebuild the machine. Clone give you second running server with one step, but e also give you two machines wey believe say dem be the same machine.

Why VPS snapshot no be backup

The problem na failure domain, no be image quality. Snapshot dey for your provider storage platform, usually for the same region where the server dey, and always inside the same account. One event fit take the server and the snapshot together.

• Dem suspend the account, payment fail, or person steal the login.
• Person or script wey get API access delete the instance. For many providers, deleting an instance dey delete its snapshots too. Read your provider documented behaviour before you assume otherwise.
• The region get serious problem and everything inside am become unreachable at once.
• Something wey dey run as root for the server find the provider API token wey you leave for /root, then remove the snapshots before e touch the disk.

Backup na the copy wey survive all four situations. The test na one question: if your provider account stop to exist this afternoon, wetin you fit still restore, and where you go restore am? Anything wey fail this question na rollback tool. Continue to take snapshots, because nothing dey restore faster. Then keep second copy for storage wey your provider no control.

The old rule still dey valid: three copies of the data, for two types of storage, with one of dem outside the platform. Provider snapshot plus restic backup repository for separate infrastructure cover this with two moving parts.

Why snapshot of database wey dey run fit restore broken

Provider snapshot dey copy block device as e dey for one exact moment. E no ask your applications to stop first, and e no fit see anything wey still dey inside page cache. So at best, the image dey crash-consistent. E go look exactly like how disk for look if person pull power cable.

Most of the stack fit handle this. ext4 and XFS dey replay their journal when dem mount, so filesystem go come up. PostgreSQL dey replay its write-ahead log when e start, and the log dey talk am:

LOG:  database system was not properly shut down; automatic recovery in progress

InnoDB dey do the same thing and e print its own crash recovery lines during startup. That recovery na database dey work as e suppose, so snapshot of one volume for quiet PostgreSQL or MySQL normally dey restore fine.

The cases wey crash-consistent no reach dey real, and na dem dey cause serious problem. If your data dey across two volumes, root disk and separate data disk go get snapshot for different moments. Because of that, data files and log directory fit no agree, and recovery no get correct thing to replay. Any file wey application write without calling fsync, like half-received upload or queue file, fit come back truncated. Anything wey application hold for memory and flush by timer no dey inside the image at all.

So write dump to disk before you take the snapshot. Then image go contain one file wey you know say e internally consistent, no matter the state of the live data files.

sudo -u postgres pg_dumpall --clean --file=/var/backups/pg-$(date +%F).sql
sudo mysqldump --single-transaction --routines --all-databases > /var/backups/mysql-$(date +%F).sql

--single-transaction dey give consistent dump of InnoDB tables without blocking writers, because dump dey run inside one repeatable-read transaction. E no cover MyISAM tables, wey need lock or stopped server. Check say dump no empty and no truncated before you trust am: tail -n 1 /var/backups/mysql-$(date +%F).sql for complete mysqldump dey end with Dump completed comment.

If you get separate data volume, you fit freeze am for the few seconds wey snapshot need:

sudo fsfreeze -f /srv
# take the snapshot from your provider's panel or API
sudo fsfreeze -u /srv

Freeze data volume only. Never freeze /. Frozen root filesystem dey block every write for the box, including the shell wey you for use type the unfreeze command. So you go lock yourself out and wait for hard reset.

The offsite half: restic or Borg

The snapshot na the fast half. The offsite copy na the half wey fit survive your provider. restic na good default because e dey deduplicate, encrypt for client side, and write go S3-compatible object storage, SFTP, or plain directory. Storage VPS as offsite target dey work well here, because backup repositories need capacity pass IOPS.

sudo apt update && sudo apt install -y restic
sudo sh -c 'umask 077; head -c 24 /dev/urandom | base64 > /root/.restic-pass'
sudo chmod 600 /root/.restic-pass
sudo cat /root/.restic-pass

Copy that passphrase enter password manager now, for device wey no be this server. You no fit open restic repository without am, and no recovery path dey. If the only copy of the password dey for the box wey you just lose, the backup na encrypted noise.

export RESTIC_REPOSITORY="s3:https://s3.example.com/vps-backups"
export RESTIC_PASSWORD_FILE=/root/.restic-pass
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
sudo -E restic init
sudo -E restic backup /etc /srv /var/backups
sudo -E restic snapshots

sudo -E dey keep those variables, because without am root go get clean environment and restic go report say no repository location dey specified. restic snapshots suppose list the run wey you just make, together with its host and paths. Verify the repository itself on schedule, and read some data back instead of only checking the structure:

sudo -E restic check --read-data-subset=5%
sudo -E restic restore latest --target /tmp/restore-check
sudo -E restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

Backup wey nobody don test na guesswork. Restore am to another VPS at least once, time am, and write the time down, because that number na your real recovery target. Borg na the other solid choice, and e dey store repository over SSH instead of object storage; the trade-offs dey covered for restic and BorgBackup comparison.

Wetin to fix before cloned VPS go near production

Clone na exact copy. Na this make am useful, and na this cause problem. Everything wey make the original unique dey duplicated, and the duplicates go clash.

Generate the SSH host keys again. The clone carry the original /etc/ssh/ssh_host_* files, so both servers dey present the same host identity. Anybody wey control one fit pretend to be the other for every client wey don accept that key. SSH no go raise warning because na the key wey client expect.

sudo rm -f /etc/ssh/ssh_host_*
sudo ssh-keygen -A
sudo systemctl restart ssh
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

ssh-keygen -A dey write fresh key for every type wey daemon expect. Fingerprint from the last command must different from the one for the original. Your current session go remain active after restart because restarting sshd no dey close established connections. Do this before anybody connect to the clone. If you leave am for later, every client wey don trust the inherited key go get WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! and must run ssh-keygen -R <host> first.

Reset the machine ID. /etc/machine-id na unique identifier wey systemd generate once for first boot, and clone inherit am.

sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
sudo reboot

Empty /etc/machine-id dey tell systemd make e generate new value for next boot. Na why you truncate the file instead of deleting am. Two things go spoil while the value remain duplicated. For images wey dey get address through DHCP, systemd-networkd dey derive DHCP client identifier from machine ID by default. Both clones go request lease as the same client, so server go give dem the same address. Also, journald dey stamp every entry with machine ID, so central log collector go file both servers under one machine. Run cat /etc/machine-id after reboot and confirm say the value don change.

Change the hostname.

sudo hostnamectl set-hostname web-02
grep 127.0.1.1 /etc/hosts

hostnamectl dey write /etc/hostname and apply the name immediately. E no touch /etc/hosts, so edit the 127.0.1.1 line to match. If you skip am, the new name no go resolve anywhere. Every sudo call go wait for failed lookup and print sudo: unable to resolve host web-02: Name or service not known.

Rotate every credential wey image already contain. The clone hold the original secrets, and now two machines fit act as the original. Check the SSH authorized_keys files, provider and DNS API tokens, application .env files, database passwords, TLS private keys, monitoring enrolment tokens, and restic repository password. This go find most of dem:

sudo grep -rIlE 'PASSWORD|SECRET|TOKEN|API_KEY' /etc /srv /opt /home 2>/dev/null
sudo find / -name '.env' -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null

If na test copy wey no go ever serve traffic, revoke the credentials instead of rotating dem. Staging box wey hold live production API token na production box wey patching worse.

Turn off jobs wey now dey run twice. Two servers wey run the same crontab go hit the same external systems for the same minute.

systemctl list-timers --all
sudo crontab -l
sudo ls -l /etc/cron.d /etc/cron.daily

The restic case need clear explanation because e fit spoil your retention instead of only failing openly. restic dey tag each snapshot with hostname, and restic forget --keep-daily 7 dey apply policy per host. Two machines wey report the same hostname go count as one host. So seven “daily” snapshots fit all come from the clone while the original snapshots dey pruned away. Fix the hostname before the first backup run, or stop the timer on the clone. certbot case simpler: two servers renewing the same names go hit certificate authority duplicate certificate rate limit. The run wey lose go fail with error say too many certificates don already issue for that exact set of names. Clone wey domain still point to original no fit pass HTTP challenge anyway, so disable renewal there.

Handle the monitoring agent. Most agents dey identify themselves with hostname or ID file wey dem write during installation. So two agents wey report as one host go mix their metrics into one series. CPU graphs go then show values wey no single machine produce, and alerts go flap. Stop and remove the agent from the clone, or enrol am again under the new hostname using your vendor documented procedure.

Check network configuration for the original address. If image carry static address inside netplan, clone go claim IP wey belong to another machine.

ip -br addr
sudo grep -r addresses /etc/netplan/

Clear cloud-init state if this clone go become template.

sudo cloud-init clean --logs

This removes cloud-init state under /var/lib/cloud, so next boot go run the first-boot modules again. This includes generating SSH host keys when none dey present. Some versions also get flag wey fit reset machine ID. Run cloud-init clean --help for your own image to see wetin e support, instead of trusting flag list from elsewhere.

Which time you go use which one

If you wan roll back risky upgrade: take a snapshot. Take am few minutes before you make the change, run the upgrade, and restore the image if anything go wrong. Restoring go throw away every write wey happen since the snapshot, so if na server wey dey receive live traffic, dump the database first and know exactly which time window you fit lose. For a do-release-upgrade on a box wey you fit take offline for ten minutes, snapshot na the complete plan.

If you dey migrate go bigger plan: deploy a clone. Build the clone from a snapshot onto the larger plan, work through the identity list above, then test am with its own IP before you move any traffic. Lower the DNS TTL one day before so cutover go quick, and keep the original running until the new box don carry real traffic. Confirm first say the larger plan actually faster for your workload, using the same benchmark method for both servers, because more vCPUs on busier hardware no always mean upgrade.

If you dey build template: snapshot a cleaned machine. Install and harden one server, then remove everything wey unique before you create the image. No host keys, empty machine ID, no personal authorized_keys, no credentials, and clean cloud-init. Snapshot am. Every instance wey you deploy from am go generate its own identity for first boot, so the checklist above no longer need to remain a checklist. Pair am with the standard first ten minutes for a new VPS so the template don already contain the work wey you for otherwise repeat.

FAQ

VPS snapshot na backup?

No, because e share the same failure domain with the server wey create am. The snapshot dey for your provider storage, inside your account, and usually for the same region. Account suspension, stolen API key, or accidental instance deletion fit remove the server and all its snapshots with one action. For many providers, deleting an instance dey also delete its snapshots by design. Snapshot na the fastest rollback wey you get, so continue to take dem, and keep another encrypted copy for infrastructure wey your provider no control.

I need stop my database before I take snapshot?

No be always, but you need accept wetin you go get. Provider snapshot dey crash-consistent, meaning say the image match how the disk for look after power cut. PostgreSQL and InnoDB fit recover from that when dem start, and PostgreSQL logs database system was not properly shut down; automatic recovery in progress while e dey do am. Recovery no dey guaranteed when your data spread across two volumes wey snapshot at different times, or when application dey write without fsync. Write pg_dumpall or mysqldump --single-transaction to disk first, so the image get one file wey you know say e consistent.

Why two cloned servers dey fight over the same IP address?

Because dem share /etc/machine-id. For images wey use DHCP, systemd-networkd dey build the DHCP client identifier from machine ID by default. So both clones dey request lease as the same client, and DHCP server dey offer both of dem the same address. Truncate /etc/machine-id to zero bytes, remove /var/lib/dbus/machine-id, symlink am back to /etc/machine-id, then reboot so systemd fit generate new value. The other common cause na static address wey dem write inside /etc/netplan/, and the clone copy am exactly; check with ip -br addr.

Which one be the fastest way to check say clone safe to put for production?

Compare four things against the original. Run ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub for both and confirm say the fingerprints different. Run cat /etc/machine-id for both and confirm say the values different. Run hostnamectl status and confirm say the name new and e dey resolve, so sudo no go warn. Then run systemctl list-timers --all and stop every timer wey dey talk to shared system, like backups, certificate renewal, or monitoring agent, until you decide which machine own that job.