How to Check CPU Steal Time for VPS Noisy Neighbour

CPU steal time dey measure wetin

CPU steal time na the portion of time wey your virtual CPU ready to run, with nothing to wait for, but hypervisor give the physical core to another guest. The work dey queue. The core dey somewhere else. Linux dey count those cycles separately and report dem as st. Na so you fit know difference between “my server dey busy” and “my server dey wait for e turn”.

Na this difference make the counter dey exist. Time wey your own processes spend for CPU dey report as us (user) or sy (system). Time wey task spend blocked on storage dey report as wa (I/O wait). If vCPU (virtual CPU) runnable, dey sit for run queue, no I/O dey outstanding, but e still no dey execute, dem report am as st. Nothing inside your server fit clear that state, because na one layer below you, for host, dem dey make the scheduling decision.

This one follow directly from how VPS dey share one physical machine between plenty guests. The usual cause na neighbour: another guest for the same node dey run hot, so host dey split the cores between una. Another cause dey wey people dey miss. Plenty providers dey limit shared vCPU to part of one physical core, and for some hypervisors, dem dey account that enforced limit as steal inside the guest. So high st reading tell you say dem no give the core to you. E no always tell you who take am.

Wey steal number dey come from

Your kernel no fit measure steal by itself because e no fit see host. Na hypervisor dey tell am. For KVM, host dey write one counter for each vCPU inside one page wey guest share, and guest dey add the values when kernel build with CONFIG_PARAVIRT_TIME_ACCOUNTING, wey every distribution kernel get. Xen dey report the same thing through e runstate area. The total dey reach userspace for exactly one place:

head -1 /proc/stat

That cpu line carry ten counters, for USER_HZ ticks since boot, for this order: user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. Steal na the eighth value after the label. Every tool below, vmstat, top, mpstat and any Prometheus exporter, dey read that same field and turn two samples into percentage.

One consequence important pass the others. If hypervisor no ever export the counter, the field go remain zero forever, and every tool wey depend on am go report calm 0.0 while host dey overloaded. KVM and Xen dey export am. Guests for VMware and Hyper-V commonly dey report flat zero. Check the platform before you trust zero:

systemd-detect-virt

E dey print platform name, like kvm, xen, vmware or microsoft, and none for bare metal. Inside container, e dey report the runtime instead, like lxc, docker or podman. This one tells you about the container, no be the machine underneath am. For kvm, zero na real evidence say host dey treat you well. For platform wey no ever fill the field, zero no be evidence at all, and you need judge contention by timing real work instead.

How I fit check CPU steal time for a VPS?

vmstat dey come from procps package. E dey present for almost every Ubuntu and Debian VPS image, but some minimal container images no get am. So install am before you depend on am.

sudo apt-get update
sudo apt-get install -y procps
vmstat --version
vmstat 1 5

vmstat --version dey print line like vmstat from procps-ng 4.0.4. If e print, the tool don install and you dey read real kernel counters. vmstat 1 5 go then take one sample every second, five times.

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  0      0 3216484  98304 1284360    0    0     4    18   62  110  3  1 96  0  0  0
 2  0      0 3216232  98304 1284360    0    0     0     0  248  431  6  2 84  0  8  0

Find st column inside cpu block for the right. Current procps-ng builds dey print gu column after am for KVM guest time, so st dey second from the right instead of last. Read the column by the header name, because that position don change between releases.

Two habits dey help keep the reading correct. The first data line na the average since boot, so ignore am and read the lines after am. One sample no be enough measurement, because steal time fit come in bursts: run vmstat 1 60 and monitor full minute before you reach conclusion.

top dey report the same number for its %Cpu(s) summary line, inside the field marked st:

%Cpu(s):  6.2 us,  2.1 sy,  0.0 ni, 83.9 id,  0.0 wa,  0.0 hi,  0.3 si,  7.5 st

For per-core detail, add sysstat:

sudo apt-get install -y sysstat
mpstat -P ALL 1 5

mpstat dey print one row for each CPU with %steal column. E dey show whether every vCPU dey affected or na only one. For the history wey support ticket need, save the samples instead of reading dem from the screen:

date -u | tee -a ~/steal.log
mpstat -P ALL 1 5 | tee -a ~/steal.log

Run am from cron during the hours wey you suspect. The file go show the provider the exact ten minutes instead of telling dem say "e feel slow last night."

Wetin the steal numbers mean?

• A steady 0.0. E healthy, or the platform no dey report steal at all. Confirm am with systemd-detect-virt before you celebrate.
• Spikes of a few percent wey last seconds. Na normal for any shared node. Neighbour fit start build, or host fit dey run backups.
• 1 to 5 percent wey remain steady for shared plan. Na wetin person expect. Shared CPU na wetin the price dey cover.
• 5 to 10 percent wey remain steady. Na slowdown wey you fit measure. Start to record evidence, then compare the same hours across several days.
• Above 10 percent for hours at a time. The node get more workload than e fit handle for your workload. Na this level dey justify support ticket or move.

Use these bands as guide for reading the result, no be as specification, because no provider dey publish steal guarantee for shared plan. Compare dem with wetin you dey run. Overnight batch job fit absorb 15 percent steal and nobody go notice. Latency-sensitive service go show am for p99 long before average look alarming. Na why latency-sensitive workloads like trading bots suppose dey run for dedicated cores.

Steal time dey cost you how much?

The calculation short. If dem take fraction s of your CPU time, a job wey need fixed CPU time go take 1 / (1 - s) times the normal clock time. For job wey need 60 seconds of CPU:

[
  {
    "steal_percent": 0,
    "wall_clock_seconds": "60.0"
  },
  {
    "steal_percent": 3,
    "wall_clock_seconds": "61.9"
  },
  {
    "steal_percent": 8,
    "wall_clock_seconds": "65.2"
  },
  {
    "steal_percent": 15,
    "wall_clock_seconds": "70.6"
  },
  {
    "steal_percent": 25,
    "wall_clock_seconds": "80.0"
  },
  {
    "steal_percent": 40,
    "wall_clock_seconds": "100.0"
  }
]

For 3 percent, wey be normal shared-plan result, the job go take 61.9 seconds instead of 60.0. Nobody go open ticket because of that. For 8 percent, e go be 65.2 seconds. For 40 percent, the same job need 100.0 seconds, and queue wey dey clear before go start dey grow instead.

Those ones na calculated values, no be measurements. The model assume say na one runnable thread dey, and steal time spread evenly through the interval. Real services often dey feel worse pass the curve, because stolen time fit enter for middle of request, then everything wey dey wait for that request go pay the delay again. To get your own figure instead of formula, benchmark the VPS during quiet hour and again when traffic busy, while you record st for both windows.

Na e steal, or na something else?

Steal fit easy mix up with other symptoms. Read the counters together, for the same vmstat line.

• st high while r and us remain low: the host no dey give you the core. Na steal be that.
• r well above your vCPU count, with us high and st near zero: you dey run more work than your own CPUs fit handle. Compare r with the output of nproc. Na your own oversubscription be this, no be neighbour.
• wa high with st near zero: tasks dey block for storage. Na different problem, and e get different fix.
• Load average high while st and us both dey low: the load figure dey also count uninterruptible tasks. So this usually point to stuck device or hung network mount, no be CPU.

Burstable plans need their own note. Dem give you credit balance wey dey build while you idle and dey reduce while you busy. When e finish, provider go hold you for baseline rate. For some platforms, dem report this throttle as steal. For others, you no fit see am from inside, and you simply get fewer cycles per second. Read the plan description before you decide say na neighbour cause the problem.

Why container dey report say no steal time

Steal na property of virtual machine, no be container wey dey run inside am. Docker container for your own VPS dey share host /proc, so any st value wey you read inside am na the VPS steal, and na that one you want. Container-based virtualisation wey dem sell as VPS dey behave different. When lxcfs dey in place, /proc/stat inside the container na cgroup accounting dey generate am, and steal dey zero by design. Monitoring stack wey dey scrape only from inside fit show steady, calm zero while the physical machine underneath no get enough resources.

Inside container, counter wey carry the same meaning na CPU quota throttling. For cgroup v2:

cat /sys/fs/cgroup/cpu.stat

nr_throttled dey count enforcement periods wey group hit its CPU quota, while throttled_usec dey total the time wey e spend frozen. If nr_throttled dey rise, e mean say your process dey ready to run but e no dey run, the same experience as steal, but na limit wey you set by yourself cause am. Check your own limits before you blame host, especially if you run your services for Docker for VPS with CPU limits for compose file. Layered virtualisation add another place wey time fit disappear, because VM inside your VPS go pay your steal plus its own scheduling delay. Remember this if you run nested virtualisation for VPS.

Wetin to do about sustained steal

No setting inside the guest fit fix steal, because scheduler wey dey make the decision dey outside the guest. Four moves dey real.

Collect evidence first. Record timestamps for UTC, how long each episode last, how often e dey repeat, and whether mpstat show say na one vCPU dey affected or all of dem. One week of logged samples worth pass screenshot.

Open ticket with that data. Ask two direct questions: this node dey oversubscribed during these windows, and dem fit move my instance? Paste the vmstat output and the exact times. Providers dey act when dem fit reproduce the time window, and ticket wey only talk say server slow go get reply wey ask for one. How much of this work you fit hand over na one practical difference between managed and unmanaged VPS.

Ask for migration. Moving guest go node wey load no too high na normal work for provider, and e usually na short reboot. Na this fix no cost anything, and e solve the common case where one node happen to get several heavy neighbours at the same time.

Buy contention comot. Dedicated vCPU plan reserve physical cores for your instance, so counter go sit for zero and remain there. E cost more every month, and na the honest answer for workload wey no fit absorb the variance. If that still no reach, or you want make memory bandwidth belong to you alone too, the next step na dedicated server instead of VPS.

While you dey wait for any of these, reduce how much steal dey hurt. Run fewer worker threads than the number of vCPUs wey you get, because threads wey no fit get core only add context switches. Move batch work go hours when node quiet, as your own log don show you now. Then measure again with the same command across the same hours, so you fit talk whether the change work instead of guessing.

FAQ

Wetin be normal CPU steal time for VPS?

For shared plan, brief spikes and steady value under about 5 percent dey normal, because shared CPU mean say host dey divide physical cores among different guests. If e stay for double digits for hours, e no normal and e worth opening ticket. For dedicated vCPU plan, expected reading na 0.0, so any other value na fault wey you suppose report. Judge the number against your own workload: overnight batch job fit manage steal time wey latency-sensitive API no fit tolerate.

Bigger plan go fix high steal time?

No be by itself. More vCPUs for the same shared node mean more virtual CPUs dey compete for the same busy physical cores, and the percentage fit remain exactly as e be. Wetin remove steal time na dedicated CPU allocation, or move to node wey no too busy. Bigger share of busy machine still remain share of busy machine.

Why my VPS dey show 0 steal time when e clearly slow?

Two common reasons dey. Hypervisor fit no export the counter at all, and this dey common for VMware and Hyper-V platforms, so the field go remain zero no matter wetin host dey do. Run systemd-detect-virt to see which platform you dey use. If no be that, bottleneck dey somewhere else: check wa for storage waits, compare r with nproc for your own overload, and read /sys/fs/cgroup/cpu.stat inside containers to check quota throttling.

I fit reduce steal time from inside my server?

You no fit change host scheduling from inside guest. You fit only reduce how much e affect you. Run fewer worker threads than the number of vCPUs wey you get, so less work go sit for run queue dey wait for core wey no dey available. Move batch jobs to hours when node quiet pass. Cache results so fewer requests need CPU at all. The changes wey really remove steal time, like migration to another node or dedicated cores, dey for provider side.