Immutable Linux distros on a server

What an immutable Linux distro is

An immutable Linux distro ships the operating system as one image, so you replace the system instead of patching it in place. There is no apt upgrade rewriting files under /usr on a running box. You build or pull a new image, the machine stages it beside the one it is running, and the next reboot swaps which one is active. The previous image is still on disk, so undoing a bad update is a reboot.

The word "immutable" oversells it. Nothing physically stops root from writing to the disk. What these systems do is mount the system directories read only and give ownership of them to the image. Persistent data lives in /var. Machine-specific configuration lives in /etc. Everything under /usr belongs to the image, which is why two servers running the same image tag hold identical system files.

Red Hat's names for the two models are the clearest ones: package mode and image mode. Package mode is a running system plus a package manager that edits it. Image mode is a build step somewhere else that produces an artifact, and a server whose only job is to boot the artifact you point it at. Everything below follows from that one difference.

Why a read-only system matters more on a server

A server that has been running for two years has a history nobody wrote down. A make install from a hurried evening. A third-party repository added for one package. A config file edited during an outage and never put back into your configuration management. The name for this is configuration drift, and it is why rebuilding "the same" server from your notes so often produces a box that behaves differently. The notes hold the intent. The disk holds the truth.

Image mode removes the place where drift collects. /usr is read only at runtime, so an install done by hand either fails outright or is recorded as a layer you can list with one command. That makes the difference between two machines visible instead of archaeological. It is the same problem a regular Linux server maintenance checklist attacks with discipline, handled by the filesystem instead.

Rollback is a reboot, and that is the whole pitch

The failure this model is built for is the one we already document: a VPS that will not boot after a kernel update. In package mode you recover from the provider's rescue console. You mount the disk, chroot in, and remove a kernel package by hand. That works because the bootloader keeps old kernels, but only the kernel is versioned that way. The glibc update and the systemd changes that arrived in the same transaction are already applied, and no single command steps them back together.

In image mode the unit is the whole system. On a bootc host:

sudo bootc status
sudo bootc rollback
sudo systemctl reboot

bootc rollback swaps the bootloader ordering back to the previous boot entry, which is the image you were running an hour ago, kernel and userspace together. Nothing is downloaded and nothing is rebuilt, because the old image never left the disk.

Fedora CoreOS does the same thing under different names:

sudo systemctl stop zincati.service
sudo rpm-ostree rollback -r

Stop Zincati first. Zincati is the agent that keeps a Fedora CoreOS machine on the newest release, so if you leave it running it will stage the update you just backed out. -r reboots once the rollback is staged. To keep a deployment you trust from being garbage collected:

sudo ostree admin pin 0
rpm-ostree status

rpm-ostree status lists deployments in the order the bootloader will offer them, marks the running one with a dot, and shows Pinned: yes on the one you pinned.

Talos does it as one API call from your workstation:

talosctl rollback --nodes 10.20.30.40

Flatcar keeps two /usr partitions and switches between them. Each slot carries a priority and a try counter in the partition table, so a slot that never boots successfully runs out of tries and the bootloader picks the other one. Check which slot you are on and whether it was marked good:

sudo cgpt show "$(rootdev -s /usr)" | grep successful=1

A healthy running slot prints a line containing priority=1 tries=0 successful=1. No matching line means the current slot was never confirmed, which is the state a machine sits in between an update and its first clean boot.

What replaces "install a package": bootc and a Containerfile

bootc is the tool that generalised this pattern. It describes itself as transactional, in-place operating system updates using OCI (open container initiative) container images, and it is a CNCF Sandbox project. Your server becomes a Containerfile. As of August 2026 the Fedora base image is quay.io/fedora/fedora-bootc:44 and the CentOS Stream base is quay.io/centos-bootc/centos-bootc:stream10.

FROM quay.io/fedora/fedora-bootc:44
RUN dnf -y install nginx && dnf clean all
RUN systemctl enable nginx

Build and push it like any other image:

sudo podman build -t registry.example.com/edge/web:2026-08-19 .
sudo podman push registry.example.com/edge/web:2026-08-19

Then on the server:

sudo bootc upgrade --check
sudo bootc upgrade --apply

bootc upgrade queries the image source and queues the new image for the next boot. --check reports whether an update is available and changes nothing. --apply reboots into it. bootc switch registry.example.com/edge/web:next points the machine at a different image while preserving /etc and /var, which is how you move a server between image streams without reinstalling it.

For unattended updates, enable the timer the project ships:

sudo systemctl enable --now bootc-fetch-apply-updates.timer

That is the image-mode answer to unattended upgrades on Ubuntu and to dnf-automatic on Rocky and Alma. The difference is what lands. A package-mode timer applies whatever versions the repository holds that night, so the resulting set is slightly different on every machine. An image-mode timer applies one artifact you have already booted somewhere else.

Two build rules follow from that Containerfile. Writable data belongs under /var, so software that insists on writing inside its own install directory needs a symlink or a systemd BindPaths= line added at build time. And /etc is three-way merged on update, which means a file you never touched picks up the image's new version while a file you edited locally is kept.

When you need a tool on a live box for one debugging session:

sudo bootc usr-overlay
sudo dnf -y install strace

That adds a transient writable overlay on /usr that is discarded at the next reboot. It is for looking at a problem, not for fixing one. You cannot change the kernel this way, and everything you install disappears on reboot by design.

Fedora CoreOS: provisioned once, updated forever

Fedora CoreOS has no interactive installer. You write a Butane YAML file, transpile it to Ignition JSON, and hand that to the machine at first boot:

podman run --interactive --rm quay.io/coreos/butane:release \
       --pretty --strict < your_config.bu > transpiled_config.ign

Ignition runs in the initramfs on the first boot only. This is the part that catches people arriving from cloud-init. If the config carries no SSH key, the machine boots with no way in, and the fix is to provision it again from scratch. Test the config on a throwaway machine before you point it at a server you care about.

Installing to a disk from a live environment:

sudo coreos-installer install /dev/sda \
    --ignition-url https://example.com/example.ign

Updates are automatic by default. You control when, not whether. Drop a TOML file at /etc/zincati/config.d/55-updates-strategy.toml that selects the periodic strategy:

[updates]
strategy = "periodic"

Under that strategy you add one maintenance window per array-of-tables entry, so each window opens with the name updates.periodic.window written inside double square brackets as its header, followed by three keys:

• days, a list of day names, such as "Sat" and "Sun".
• start_time, the moment the window opens, written as "22:30".
• length_minutes, how long it stays open, such as 60.

Those times are UTC. To stop updates entirely, run sudo systemctl disable --now zincati.service, and accept that you now own the patching schedule.

Package layering exists as an escape hatch:

sudo rpm-ostree install --allow-inactive vim
sudo systemctl reboot

That builds a new deployment with the package added, and the change only takes effect after the reboot. The cost arrives later. Your layered set is re-applied on top of every new base image, so a package that has vanished from the repository on the day of an update makes that update fail. Fedora's own documentation steers you toward containers for anything substantial, and toward a bootc image when you genuinely need to change the OS.

Flatcar Container Linux: no package manager at all

Flatcar is the continuation of CoreOS Container Linux and the strictest of the general-purpose options. There is no package manager to fall back on. Everything you run is a container. Provisioning is Ignition, the same as Fedora CoreOS. Updates are the two A/B /usr partitions described above, driven by update_engine, with locksmithd deciding when the reboot happens.

update_engine_client -status
update_engine_client -check_for_update

UPDATE_STATUS_UPDATED_NEED_REBOOT means the passive slot already holds the new image and only the reboot is outstanding. The default reboot strategy is reboot with a five-minute delay, so a single production VPS will restart on its own schedule unless you say otherwise. Set a window in /etc/flatcar/update.conf:

REBOOT_STRATEGY=reboot
LOCKSMITHD_REBOOT_WINDOW_START="Thu 04:00"
LOCKSMITHD_REBOOT_WINDOW_LENGTH=1h

REBOOT_STRATEGY=off leaves the reboot to you. SERVER=disabled in the same file stops the update check completely. For a cluster, REBOOT_STRATEGY=etcd-lock combined with locksmithctl set-max 4 caps how many nodes may reboot at once, so an update never takes the whole fleet down together.

Talos Linux: no shell, no SSH, no console

Talos is the narrowest of the four and the clearest about its purpose. It runs Kubernetes nodes. There is no SSH daemon, no shell and no console login. Every operation is a gRPC API call made with talosctl from your workstation, against a machine config you keep in git.

talosctl upgrade --nodes 10.20.30.40 \
  --image ghcr.io/siderolabs/installer:v1.10.6

Replace the tag with the release you are moving to. The upgrade uses an A-B scheme that retains the previous kernel and OS image, so if the new one fails to boot, Talos rolls back on its own with no intervention. Debugging looks different because there is no shell: you use talosctl logs and talosctl dmesg rather than journalctl on the box.

If your workload is not Kubernetes, Talos is the wrong answer. If it is, Talos deletes an entire category of incident, because "someone logged into a node and changed something" has nowhere to happen.

What a VPS tenant actually gives up

Ad-hoc installs on a running system. This is the big one. sudo apt install htop at 2am during an incident is not available. On bootc you get a transient overlay that dies at reboot. On Fedora CoreOS you get a layered deployment that needs a reboot. On Flatcar and Talos you get nothing.

A build pipeline you did not have before. Adding a package means editing a Containerfile, building the image, pushing it to a registry and rolling the servers. That is cheap when the pipeline exists. It is real work to stand up when it does not, and it needs a registry the servers can reach, which is another service to run or another bill to pay.

Kernel modules. The kernel comes from the image, so a module compiled against the running kernel does not survive the next update. Out-of-tree modules and DKMS (dynamic kernel module support) packages have to be built into the image, against that image's kernel. Anything needing a module the base image does not carry becomes a build problem instead of an install problem.

Vendor and provider agents. Monitoring and backup agents usually ship as a .deb or .rpm with an install script that writes into /usr and enables a unit. On a read-only system that script fails. Some vendors publish a container or document an image-mode install. Many do not. Check this before you commit, because a fleet you cannot monitor is worse than a fleet that drifts.

The image itself. Almost no VPS control panel lists Fedora CoreOS or Flatcar or Talos beside Ubuntu and Debian. You supply the disk, which is the next section.

Getting one of these onto a VPS you rent

Confirm two things about your provider first: that you have out-of-band console access, meaning VNC or a serial console, and that you can boot a rescue system. Without a console, a machine that does not come back is a support ticket instead of a five-minute fix.

If the provider accepts custom images, upload the vendor's raw or qcow2 image and the job is done. Otherwise you write the disk yourself from the rescue system. Flatcar ships a self-contained script for exactly this, and it runs from any Linux:

curl -fsSLO https://raw.githubusercontent.com/flatcar/init/flatcar-master/bin/flatcar-install
chmod +x flatcar-install
sudo ./flatcar-install -d /dev/sda -i ignition.json

Run that from the rescue system, never from the server you are replacing, because the script repartitions the target device while it works. It needs at least 8 GB of usable space on the device, and the rescue environment must provide bash, bzip2 or lbzip2, lsblk, wget, udevadm, gpg and gawk. Your ignition.json must contain an SSH key, or the installed system will have no way to let you in.

Fedora CoreOS has the same shape, and its installer runs as a container:

sudo podman run --pull=always --privileged --rm \
    -v /dev:/dev -v /run/udev:/run/udev -v .:/data -w /data \
    quay.io/coreos/coreos-installer:release \
    install /dev/vdb -i config.ign

Check the device name with lsblk before you run it. Writing to the wrong device destroys whatever was on it, and there is no confirmation prompt.

bootc offers the one path that skips rescue mode, because it converts a running Linux system in place:

sudo podman run --rm --privileged -v /dev:/dev -v /var/lib/containers:/var/lib/containers -v /:/target \
             --pid=host --security-opt label=type:unconfined_t \
             quay.io/fedora/fedora-bootc:44 \
             bootc install to-existing-root

Read the documentation for the base image you are using before you run that, and try it on a server you can throw away. After the reboot the machine runs the image, and the package set you had is gone.

Who should run an immutable server, and who should not

Want this if your servers are cattle. Many machines from one recipe. CI (continuous integration) runners that live for an hour. k3s or Kubernetes nodes that get replaced rather than repaired. Anything where the answer to a broken box is already "delete it and make another one". It also pays off when you have to prove to an auditor what is running on a machine, because the answer is an image digest rather than a package list.

Do not want this if you have one hand-tended VPS with three services on it, you install things when you need them, and you have no build pipeline. Image mode does not remove work. It moves the work from the server to the build, and charges you a registry and a pipeline for the move. If you have somewhere to put that work, you get servers that are identical and a rollback that is a reboot. If you do not, you have added moving parts to a box that was fine, and made 2am harder.

The boring middle ground still works: a normal distribution with automatic security updates, plus a rebuild you have actually practised. Picking that base is its own decision, covered in choosing which OS to run on your VPS. Image mode is only the newest round of a very old argument about how software should reach a machine, and the history of Linux distributions is largely that argument repeating itself.

FAQ

Is an immutable Linux distro really immutable?

No, and the name causes confusion. Root can still write to the disk. What actually happens is that /usr is mounted read only at runtime and replaced whole by the next image, while /etc and /var stay writable and persist across updates. Changes you make under /usr are either refused at the time or discarded at the next update, so the practical effect is that system directories only change when the image changes.

Can I run Fedora CoreOS or Flatcar on a VPS that does not offer them?

Usually yes, if the provider gives you a rescue system and console access. You boot rescue, write the distribution's disk image to the block device, then reboot. Flatcar's flatcar-install script does this from any Linux, and Fedora CoreOS ships coreos-installer as a container you can run the same way. Both need an Ignition file containing your SSH key, because there is no first-boot password prompt to fall back on. Without console access, do not attempt it: a machine that fails to come back gives you nothing to look at.

How do I install a package on an immutable server?

You add it to the image and redeploy. On bootc that is a RUN dnf -y install ... line in the Containerfile, a rebuild, a push, then sudo bootc upgrade --apply on the machine. On Fedora CoreOS you can layer it with sudo rpm-ostree install and reboot, at the cost of that package being re-applied on every future update. On Flatcar and Talos there is no package manager, so the answer is a container. For a one-off debugging tool on a bootc host, sudo bootc usr-overlay gives you a writable /usr that disappears at the next reboot.

Does image mode fix a VPS that will not boot after a kernel update?

It turns the recovery from a rescue-console job into a reboot. The previous image, kernel and userspace together, is still on disk, so sudo bootc rollback or sudo rpm-ostree rollback -r puts you back on it. Talos and Flatcar go further and roll back without being asked when the new slot fails to boot, because a boot entry only becomes the default after one successful boot. None of this prevents a bad update. It makes undoing one cheap.

Which immutable distro should I pick for a server?

Pick bootc if you want a general-purpose Linux server that you build like a container image and can install onto a machine you already have. Pick Fedora CoreOS if you want that model with the building done for you and automatic updates out of the box. Pick Flatcar if you want a minimal container host with an A/B update scheme and no package manager for anyone to reach for. Pick Talos only when the machine is a Kubernetes node, since it has no shell and runs nothing else.