CPU steal time and noisy neighbours on a VPS

What CPU steal time actually measures

CPU steal time is the share of time your virtual CPU was ready to run, with nothing to wait for, while the hypervisor gave the physical core to a different guest. The work was queued. The core was somewhere else. Linux counts those cycles separately and reports them as st, which is how you tell "my server is busy" apart from "my server is waiting for its turn".

That difference is the whole reason the counter exists. Time your own processes spend on the CPU is reported as us (user) or sy (system). Time a task spends blocked on storage is reported as wa (I/O wait). A vCPU (virtual CPU) that is runnable, sitting on the run queue, with no I/O outstanding, and still not executing, is reported as st. Nothing inside your server can clear that state, because the scheduling decision is made one layer below you, on the host.

This follows directly from how a VPS shares one physical machine between many guests. The usual cause is a neighbour: another guest on the same node is running hot, so the host splits the cores between you. There is a second cause that gets missed. Many providers cap a shared vCPU at a fraction of a physical core, and on several hypervisors that enforced cap is accounted as steal inside the guest. So a high st reading tells you the core was not given to you. It does not always tell you who took it.

Where the steal number comes from

Your kernel cannot measure steal by itself, because it cannot see the host. The hypervisor tells it. On KVM the host writes a per-vCPU counter into a page shared with the guest, and the guest adds it up when the kernel is built with CONFIG_PARAVIRT_TIME_ACCOUNTING, which every distribution kernel is. Xen reports the same thing through its runstate area. The total reaches userspace in exactly one place:

head -1 /proc/stat

That cpu line carries ten counters, in USER_HZ ticks since boot, in this order: user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. Steal is the eighth value after the label. Every tool below, vmstat, top, mpstat and any Prometheus exporter, reads that same field and turns two samples into a percentage.

One consequence matters more than the rest. If the hypervisor never exports the counter, the field stays at zero forever, and every tool built on it reports a calm 0.0 while the host is overloaded. KVM and Xen export it. Guests on VMware and Hyper-V commonly report a flat zero. Check the platform before you trust a zero:

systemd-detect-virt

It prints the platform name, such as kvm, xen, vmware or microsoft, and none on bare metal. Inside a container it reports the runtime instead, such as lxc, docker or podman, which tells you about the container and not about the machine underneath it. On kvm a zero is real evidence that the host is treating you well. On a platform that never fills the field in, a zero is no evidence at all, and contention has to be judged by timing real work instead.

How do I check CPU steal time on a VPS?

vmstat comes from the procps package. It is present on almost every Ubuntu and Debian VPS image, and missing from some minimal container images, so install it before you depend on it.

sudo apt-get update
sudo apt-get install -y procps
vmstat --version
vmstat 1 5

vmstat --version prints a line such as vmstat from procps-ng 4.0.4. If that prints, the tool is installed and you are reading real kernel counters. vmstat 1 5 then takes one sample per second, five times.

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  0      0 3216484  98304 1284360    0    0     4    18   62  110  3  1 96  0  0  0
 2  0      0 3216232  98304 1284360    0    0     0     0  248  431  6  2 84  0  8  0

Find the st column in the cpu block on the right. Current procps-ng builds print a gu column after it for KVM guest time, so st is second from the right rather than last. Read the column by its header name, because that position has changed between releases.

Two habits keep the reading honest. The first data line is the average since boot, so ignore it and read the lines after it. And one sample is not a measurement, because steal arrives in bursts: run vmstat 1 60 and watch a full minute before you draw a conclusion.

top reports the same number on its %Cpu(s) summary line, in the field marked st:

%Cpu(s):  6.2 us,  2.1 sy,  0.0 ni, 83.9 id,  0.0 wa,  0.0 hi,  0.3 si,  7.5 st

For per-core detail, add sysstat:

sudo apt-get install -y sysstat
mpstat -P ALL 1 5

mpstat prints one row per CPU with a %steal column, which shows whether every vCPU is affected or only one. For the history a support ticket needs, keep the samples instead of reading them off the screen:

date -u | tee -a ~/steal.log
mpstat -P ALL 1 5 | tee -a ~/steal.log

Run that from cron through the hours you suspect, and the file becomes the difference between telling a provider "it felt slow last night" and showing them the exact ten minutes.

What do the steal numbers mean?

• A steady 0.0. Healthy, or the platform does not report steal at all. Confirm with systemd-detect-virt before you celebrate.
• Spikes of a few percent lasting seconds. Normal on any shared node. A neighbour's build starts, or the host runs its backups.
• 1 to 5 percent sustained on a shared plan. Expected. Shared CPU is what the price reflects.
• 5 to 10 percent sustained. A slowdown you can measure. Start recording evidence, and compare the same hours across several days.
• Above 10 percent for hours at a time. The node is oversubscribed for your workload. This is the level that justifies a support ticket or a move.

Treat those bands as a reading guide rather than a specification, because no provider publishes a steal guarantee on a shared plan. Weigh them against what you run. An overnight batch job can absorb 15 percent steal and nobody notices. A latency-sensitive service shows it in the p99 long before the average looks alarming, which is why latency-sensitive workloads such as trading bots belong on dedicated cores.

How much does steal time cost you?

The arithmetic is short. If a fraction s of your CPU time is taken, a job that needs a fixed amount of CPU takes 1 / (1 - s) times as long on the clock. For a job needing 60 seconds of CPU:

[
  {
    "steal_percent": 0,
    "wall_clock_seconds": "60.0"
  },
  {
    "steal_percent": 3,
    "wall_clock_seconds": "61.9"
  },
  {
    "steal_percent": 8,
    "wall_clock_seconds": "65.2"
  },
  {
    "steal_percent": 15,
    "wall_clock_seconds": "70.6"
  },
  {
    "steal_percent": 25,
    "wall_clock_seconds": "80.0"
  },
  {
    "steal_percent": 40,
    "wall_clock_seconds": "100.0"
  }
]

At 3 percent, an ordinary shared-plan reading, that job takes 61.9 seconds instead of 60.0. Nobody opens a ticket over that. At 8 percent it is 65.2 seconds. At 40 percent the same job needs 100.0 seconds, and a queue that used to drain starts growing instead.

Those are calculated values, not measurements. The model assumes a single runnable thread and steal spread evenly through the interval. Real services often feel worse than the curve, because a stolen slice lands in the middle of a request and the delay is then paid again by everything waiting on that request. To get your own figure rather than a formula, benchmark the VPS during a quiet hour and again during a busy one, recording st for both windows.

Is it steal, or is it something else?

Steal is easy to confuse with other symptoms. Read the counters together, on the same vmstat line.

• st high while r and us stay low: the host is not giving you the core. That is steal.
• r well above your vCPU count, with us high and st near zero: you are running more work than your own CPUs can take. Compare r with the output of nproc. This is your own oversubscription, not a neighbour.
• wa high with st near zero: tasks are blocked on storage, which is a different problem with a different fix.
• Load average high while st and us are both low: the load figure also counts uninterruptible tasks, so this usually points at a stuck device or a hung network mount rather than at CPU.

Burstable plans deserve their own note. They give you a credit balance that builds while you are idle and drains while you are busy, and when it runs out the provider holds you at a baseline rate. On some platforms that throttle is reported as steal. On others it is invisible from inside, and you simply get fewer cycles per second. Read the plan description before deciding that a neighbour is at fault.

Why a container reports no steal time

Steal is a property of the virtual machine, not of a container running inside it. A Docker container on your own VPS shares the host's /proc, so an st value read inside it is the VPS's steal, which is what you want. Container-based virtualisation sold as a VPS behaves differently. With lxcfs in place, /proc/stat inside the container is synthesized from cgroup accounting, and steal is zero by construction. A monitoring stack that scrapes only from inside can show a flat, calm zero while the physical machine underneath is starving.

Inside a container, the counter that carries the same meaning is CPU quota throttling. On cgroup v2:

cat /sys/fs/cgroup/cpu.stat

nr_throttled counts the enforcement periods in which the group hit its CPU quota, and throttled_usec totals the time it spent frozen. A rising nr_throttled means your process was runnable and not running, the same experience as steal, but caused by a limit you set yourself. Check your own limits before blaming the host, especially if you run your services in Docker on a VPS with CPU limits in the compose file. Layered virtualisation adds one more place for time to vanish, because a VM inside your VPS pays your steal plus its own scheduling delay. Keep that in mind if you run nested virtualisation on a VPS.

What to do about sustained steal

No setting inside the guest fixes steal, because the scheduler making the decision runs outside the guest. Four moves are real.

Collect evidence first. Record timestamps in UTC, the duration of each episode, how often it repeats, and whether mpstat shows one vCPU affected or all of them. A week of logged samples is worth more than a screenshot.

Open a ticket with that data. Ask two direct questions: is this node oversubscribed during these windows, and can my instance be moved. Paste the vmstat output and the exact times. Providers act on a reproducible window, and a ticket that only says the server is slow gets a reply asking for one. How much of this work you can hand over is one of the practical differences between a managed and an unmanaged VPS.

Ask for a migration. Moving a guest to a less loaded node is routine work for a provider, and it is usually a short reboot. This is the fix that costs nothing, and it solves the common case, where one node happens to hold several heavy neighbours at the same time.

Buy the contention away. A dedicated vCPU plan reserves physical cores for your instance, so the counter sits at zero and stays there. It costs more each month, and it is the honest answer for a workload that cannot absorb the variance. If that is still not enough, or you want the memory bandwidth to yourself as well, the next step is a dedicated server instead of a VPS.

While you wait for any of that, reduce how much the steal hurts. Run fewer worker threads than you have vCPUs, because threads that cannot get a core only add context switches. Move batch work to the hours when the node is quiet, which your own log now tells you. Then measure again with the same command over the same hours, so you can say whether the change worked instead of guessing.

FAQ

What is a normal CPU steal time on a VPS?

On a shared plan, brief spikes and a sustained value under about 5 percent are ordinary, because shared CPU means the host divides physical cores between guests. Sustained double digits across hours is not ordinary and is worth a ticket. On a dedicated vCPU plan the expected reading is 0.0, so anything else there is a fault to report. Judge the number against your own workload: an overnight batch job absorbs steal that a latency-sensitive API cannot.

Will a bigger plan fix high steal time?

Not by itself. More vCPUs on the same shared node means more virtual CPUs competing for the same contended physical cores, and the percentage can stay exactly where it was. What removes steal is a dedicated CPU allocation, or a move to a less loaded node. A larger share of a busy machine is still a share of a busy machine.

Why does my VPS show 0 steal time when it is clearly slow?

There are two common reasons. The hypervisor may not export the counter at all, which is usual on VMware and Hyper-V platforms, so the field stays at zero whatever the host does. Run systemd-detect-virt to see which platform you are on. Otherwise the bottleneck is elsewhere: check wa for storage waits, compare r with nproc for your own overload, and read /sys/fs/cgroup/cpu.stat inside containers for quota throttling.

Can I reduce steal time from inside my server?

You cannot change the host's scheduling from inside the guest. You can only reduce how much it hurts. Run fewer worker threads than you have vCPUs, so less work sits on the run queue waiting for a core that is not coming. Move batch jobs to the hours when the node is quieter. Cache results so fewer requests need CPU at all. The changes that actually remove steal, a migration to another node or dedicated cores, are on the provider's side of the line.