Local NVMe vs network storage on a VPS

Local NVMe vs network storage on a VPS

Local NVMe vs network storage on a VPS is one decision hiding behind two product names. A local NVMe disk is flash inside the same physical machine that runs your virtual machine, reached over PCIe. A network volume is flash inside other machines, reached over the provider's network, and usually written to two or three of them before your write is acknowledged.

Both are advertised as "NVMe SSD", and both claims are honest. The flash in a storage cluster really is NVMe flash. What the spec sheet leaves out is the path between your process and that flash, and the path is the part you feel.

The two paths a write can take

On local storage the path is short. Your process calls write() and then fsync(). The guest kernel hands the request to a virtio device. The hypervisor passes it to the host block layer, which sends it over PCIe to a flash device in the same chassis. The device stores it, acknowledges, and the acknowledgement walks back up the same stack.

On network storage the path adds a network. The host runs a client for the storage system. Ceph RBD and iSCSI are the common ones, with NVMe over Fabrics (NVMe-oF, the NVMe protocol carried over a network) appearing more often now. That client works out which nodes hold your data, sends the write to them over the NIC and a switch, and waits. A cluster configured for three replicas will not acknowledge the write until enough copies are durable. Only then does the acknowledgement travel back to your process.

Nothing here is a flaw. The extra hops buy properties the short path cannot offer. But every hop costs time, and it is charged per operation.

Why the latency floor is the number that matters

Read the second path again and count the waiting. There is at least one network round trip between the host and a storage node, plus that node's own device service time, plus the wait for however many replicas the provider requires. A local write pays the device service time and very little else.

Published figures put a local NVMe write acknowledgement in the tens of microseconds, and a replicated network write in the hundreds of microseconds to a few milliseconds. Treat those as vendor and community numbers, not as a measurement of your VPS. What matters is the shape of the difference. The two designs have different floors, and no amount of parallelism lowers a floor. Parallelism raises the ceiling.

This is also why two benchmarks of the same disk can tell opposite stories. A test at queue depth 32 hides the floor, because 32 requests in flight overlap their waiting. A test at queue depth 1 exposes it, because each request waits for the one before it. Run both, and run them yourself: measuring a VPS disk with fio at queue depth 1 and again at high queue depth is the only way to learn what your machine does rather than what a product page claims.

Why databases and small writes feel it first

A database commit is a small synchronous write. PostgreSQL appends the transaction to the write-ahead log, calls fsync(), and only then reports success to the client. Until that fsync() returns, the connection sits idle and waits. So the commit rate of one connection is bounded by one divided by the commit latency. A 0.1 ms floor allows roughly ten times the single-connection commit rate that a 1 ms floor allows. That is arithmetic, and it explains why an application that felt fast on one provider feels slow on another with identical printed specs.

Concurrency softens it. PostgreSQL groups concurrent commits into a single flush, so a hundred busy connections do not pay a hundred separate waits. What stays slow is anything serial: a migration that commits per row, or a queue worker that finishes one job before starting the next.

The same pattern hits everything that calls fsync() often: etcd, Redis with appendfsync always, SQLite in its default journal mode, a mail server writing to Maildir, and a CI runner unpacking thousands of small files. Read-heavy workloads whose working set fits in RAM barely notice, because the page cache answers most reads and never reaches the disk at all.

Throughput is capped by the network, not by the device

Sequential throughput runs into a different limit on each design. A single modern NVMe device moves several gigabytes per second. A network volume cannot exceed the link it arrives over, minus protocol overhead, minus whatever every other tenant is pulling through the same switch at that moment. A 10 Gbit/s link carries about 1.25 GB/s at line rate before any overhead, so a backend built entirely from fast flash can still deliver less sequential bandwidth than one local drive.

Providers usually state this openly for network volumes, as a bandwidth cap and an IOPS (input/output operations per second) cap, often scaled by volume size, sometimes with a burst allowance that refills over time. A cap that scales with size means a 20 GB volume and a 2 TB volume behave very differently even though the page calls both of them the same product. Local disks are rarely capped that way. They are shared instead, which is a different problem.

What a noisy neighbour looks like on each

Local NVMe shares one device and one host. When another guest on your host runs a heavy random write job, your requests queue behind theirs and your latency rises. The blast radius is that host. Some providers apply per-guest I/O throttling in the hypervisor, which turns the noise into a hard ceiling you hit consistently instead of a random one you hit sometimes.

Network storage spreads your data over many nodes, so one loud tenant is diluted. In exchange you inherit cluster-wide events. A failed node triggers a recovery that rebuilds the missing copies, and that rebuild competes with client traffic across the whole cluster. Network congestion or a maintenance window on a storage rack can slow tenants who did nothing at all.

From inside the guest you can separate "my disk is busy" from "my disk is waiting" without knowing the backend. Run iostat -x 1 from the sysstat package and compare aqu-sz (average queue size) with await (average wait per request). A high await while aqu-sz stays small means the time is being spent below you, because your own workload is not deep enough to explain the queueing. cat /proc/pressure/io tells the same story as a stall percentage: a rising some avg10 value means tasks are blocked waiting on I/O. Storage noise and CPU noise often arrive together, and steal time is the matching signal on the CPU side.

What happens during a live migration

This is where network storage earns its price. When your disk lives in a cluster, moving your VPS to another host means copying CPU state and RAM. The disk does not move, because it was never on the host. Modern hypervisors do that with a pause measured in milliseconds. You usually learn about it from a status page rather than from your monitoring.

When the disk is local, the disk has to move too. The hypervisor mirrors the block device to the destination host while the machine keeps running and keeps dirtying blocks, so the copy takes at least as long as the data, and longer if you are writing during it. Some providers do this anyway. Many do not, and their maintenance notice says a reboot is required. That difference in wording is a good clue about the design underneath.

What happens when a storage node fails

Network storage survives losing a node, because the copies live elsewhere. What the tenant sees is a pause rather than data loss: I/O blocks while the cluster works out which copy is authoritative. If the pause runs long enough, the guest kernel complains. dmesg prints lines like these:

INFO: task jbd2/vda1-8:311 blocked for more than 120 seconds.
blk_update_request: I/O error, dev vda, sector 1050624
EXT4-fs (vda1): Remounting filesystem read-only

A filesystem that has gone read-only stays read-only until you remount or reboot, so a storage incident the provider handled cleanly can still leave your VPS needing a restart. Increasing the guest I/O timeout, where the device exposes one, only widens the window in which the guest waits quietly instead of failing.

Local NVMe fails in a different shape. A dead flash device is survivable when the host writes to a mirrored set, which is why RAID 10 on the host is worth asking about on any plan that advertises local disks. A dead host is not survivable the same way: your data is intact on flash that nobody can reach until the machine returns, and if it never returns, the data does not either. Local NVMe remains the right choice for plenty of workloads, provided you own the recovery path, and the difference between a snapshot and a backup is exactly what people discover too late here.

What network storage buys you

Instant snapshots are the clearest example. A snapshot in a cluster is a copy-on-write marker, so it completes in about a second whatever the volume size, because no data is copied at the moment you press the button. On a local disk, a snapshot means copying real blocks, which takes real time and real space on the same host.

Detaching a volume from one instance and attaching it to another is only possible when the volume is not welded to a chassis. Growing a volume past the largest drive the provider owns is possible for the same reason, and so is thin provisioning, which is how a panel can sell you 5 TB without anyone racking 5 TB for you today. When capacity rather than latency is your constraint, that flexibility is the entire point, and block volumes compared with object storage for cheap terabytes covers where the money actually goes.

How to find out which one you have

From inside the guest, the device name proves nothing. Almost every VPS presents its disk through virtio as /dev/vda, whatever sits behind it. cat /sys/block/vda/queue/rotational reads 0 on nearly all of them, because it reports what the hypervisor advertises. lsblk -o NAME,SIZE,ROTA,TRAN,MODEL commonly shows an empty MODEL for virtio-blk and QEMU HARDDISK for virtio-scsi, and neither string says where the blocks live.

One check does give a real answer when it applies:

ls /sys/class/nvme/
cat /sys/class/nvme/nvme0/transport

If your disk appears as /dev/nvme0n1, transport reads pcie for a device on the local PCIe bus, and tcp, rdma or fc for NVMe over Fabrics, which is network storage wearing an NVMe name. So seeing /dev/nvme0n1 is not proof of a local disk on its own. If /sys/class/nvme/ does not exist, you are on virtio and this check does not apply to you. For the wider set of kernel-side checks, see confirming an NVMe disk on Linux from inside the guest.

That leaves behaviour and paperwork, which are more dependable than device names:

• Can you detach the disk and attach it to a different instance? Then it is a network volume.
• Does a snapshot of a large disk finish in seconds? Then it is almost certainly copy-on-write in a cluster.
• Can you resize the disk while the machine runs, or order one larger than any single drive? Same conclusion.
• Does the provider promise host maintenance without a reboot? That requires the disk to stay where it is.
• Is the extra disk priced as its own line item per gigabyte per month? Providers price network volumes that way because the capacity is genuinely separable from the machine.

Then ask support in plain words. A short question gets a straight answer more often than people expect: "Liegt die Systemplatte als lokaler NVMe-Datenträger im Host, oder ist sie ein Volume aus einem verteilten Speichersystem?" Ask about the root disk and about any additional volume separately, because one VPS very often has both, a local root disk for the operating system and a network volume for bulk data.

Finally, measure. A queue depth 1 random write test says more about the design than any spec sheet, and the fio recipes for running that test correctly live in one place rather than being repeated here.

Reading a German offer that says 'Cloud' or 'VPS'

Neither word is a technical term. In practice the German market uses "Cloud" for products with an API, hourly billing, snapshots and attachable volumes, and "VPS" or "vServer" for a fixed monthly machine with a fixed disk. That correlation is real, and it is a marketing convention rather than a statement about architecture. Providers change it without telling anyone.

Read the storage wording instead of the product name. "Lokale NVMe-SSD" or "lokaler NVMe-Speicher" says the disk is in the host. "Verteiltes Speichersystem", "Netzwerkspeicher", "Blockspeicher" or "dreifach redundant gespeichert" says the write leaves the machine. The redundancy phrasing is a strong signal by itself, because a single local disk in one host is not stored three times anywhere.

Watch for the sentence that is true either way. "NVMe SSD storage" describes a Ceph cluster built from NVMe drives just as accurately as it describes a drive in your host. It names the media and says nothing about the distance. When a page only names the media, assume you have not been told yet, and ask. The same reading discipline applies to the rest of the page, which reading a cheap VPS offer line by line works through, and the difference between a storage VPS and a regular VPS covers the capacity-first plans where this question changes shape again.

Which workloads justify which

Pay for local NVMe when a small synchronous write sits on the path your users wait for. A PostgreSQL or MySQL primary with a real commit rate, etcd, a queue whose workers commit per job, a mail store, a CI runner: all of these live on the latency floor, and the floor is the thing you are buying. In exchange you accept that one host holds that copy of your data. Replicate off the box or back up off the box, and watch the disk you were given, which disk health monitoring from inside a guest covers for the parts a tenant can actually see.

Pay for network storage when the data must outlive the instance, when you need more capacity than one machine holds, or when you want a server you can move, resize and snapshot without scheduling an outage. Read replicas, application servers, backup targets and media libraries all sit here comfortably, because none of them are bounded by single-threaded commit latency.

Most real setups end up mixed: root disk on local NVMe so the operating system and the database feel fast, and a network volume attached for backups and bulk files. That arrangement produces the fewest surprises, because each kind of storage does the job it is good at. If you are still deciding on the media before you get to the topology, how NVMe compares with SATA SSD on a VPS is the earlier question to settle.

FAQ

How do I tell if my VPS disk is local NVMe or network storage?

From inside the guest the device name proves nothing, because virtio presents /dev/vda for both. If you do have /dev/nvme0n1, read /sys/class/nvme/nvme0/transport: pcie means the local PCIe bus, while tcp or rdma means NVMe over Fabrics, which is network storage. Otherwise judge by behaviour. Instant snapshots on a large disk, detaching a volume and attaching it elsewhere, online resize, or maintenance without a reboot all require network storage. Then ask support about the root disk and about any extra volume separately, since one machine often has both.

Does "NVMe" on the spec sheet mean the disk is inside my server?

No. It names the media, not the distance. A distributed storage cluster built from NVMe drives is accurately described as NVMe storage, and every write still crosses the network and waits for replica acknowledgements. Look for wording about where the data lives, such as "lokal" against "verteilt" or "redundant gespeichert", rather than for the media name.

Why is my database slow on a VPS that advertises NVMe?

Each commit calls fsync() and waits, so one connection cannot commit faster than one divided by the write acknowledgement latency. Network storage adds a network round trip and replica acknowledgements to every one of those waits. Serial work suffers most, such as a migration that commits per row or a single-threaded worker. Concurrent commits are grouped into one flush, so adding concurrency often recovers much of the throughput even while each individual commit stays slow.

Is network storage safer than local NVMe?

It is more durable against hardware failure, because the copies sit on separate machines and a lost node is rebuilt automatically. It is not a backup. A dropped table or a bad migration is replicated to every copy immediately. Local NVMe puts durability in your hands, which is worse if you do nothing about it and perfectly fine if you replicate or back up off the host.

Can I run a production database on network storage?

Yes, and many people do. Expect a higher commit latency floor and design around it: batch your writes where the application allows it, and check whether the provider caps IOPS by volume size, because a small volume can carry a small cap. If single-threaded commit rate is the thing your users feel, test both designs at queue depth 1 before you decide.