CPU steal time for VPS: neighbour dey chop your CPU?

CPU steal time actually dey measure wetin

CPU steal time na the part of time wey your virtual CPU ready to run, with nothing to wait for, but hypervisor give the physical core to another guest. The work dey queue. The core dey somewhere else. Linux dey count those cycles separately and report dem as st. Na so you fit tell “my server dey busy” apart from “my server dey wait for e turn”.

Na this difference make the counter dey exist. Time wey your own processes spend for CPU dey report as us (user) or sy (system). Time wey task spend blocked on storage dey report as wa (I/O wait). If vCPU (virtual CPU) runnable, e dey sit for run queue, no I/O outstanding, but e still no dey execute, dem dey report am as st. Nothing inside your server fit clear this state, because na one layer below you, for host, scheduling decision dey happen.

This one follow directly from how VPS dey share one physical machine between many guests. The usual cause na neighbour: another guest for the same node dey run hot, so host dey split the cores between una. Another cause dey wey people dey miss. Plenty providers dey cap shared vCPU to part of one physical core, and for some hypervisors, dem dey account for this enforced cap as steal inside the guest. So high st reading tell you say dem no give you the core. E no always tell you who take am.

Wey steal number dey come from

Your kernel no fit measure steal by itself because e no fit see host. Hypervisor dey tell am. For KVM, host dey write one counter for each vCPU inside one page wey guest dey share, and guest dey add the values when kernel build with CONFIG_PARAVIRT_TIME_ACCOUNTING, wey every distribution kernel get. Xen dey report the same thing through e runstate area. The total dey reach userspace for exactly one place:

head -1 /proc/stat

That cpu line carry ten counters, for USER_HZ ticks since boot, for this order: user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. Steal na the eighth value after the label. Every tool below, vmstat, top, mpstat and any Prometheus exporter, dey read that same field and turn two samples into percentage.

One consequence important pass the others. If hypervisor no ever export the counter, the field go remain zero forever, and every tool wey build on top am go report calm 0.0 while host dey overloaded. KVM and Xen dey export am. Guests for VMware and Hyper-V commonly dey report flat zero. Check the platform before you trust zero:

systemd-detect-virt

E go print platform name, such as kvm, xen, vmware or microsoft, and none for bare metal. Inside container, e go report the runtime instead, such as lxc, docker or podman, and this one tell you about the container, no be the machine underneath am. For kvm, zero na real evidence say host dey treat you well. For platform wey no ever fill the field, zero no be evidence at all, and you go need judge contention by timing real work instead.

How I fit check CPU steal time for VPS?

vmstat dey come from procps package. E dey available for almost every Ubuntu and Debian VPS image, but some minimal container images no get am. So install am before you depend on am.

sudo apt-get update
sudo apt-get install -y procps
vmstat --version
vmstat 1 5

vmstat --version go print line like vmstat from procps-ng 4.0.4. If e print that, the tool don install and you dey read real kernel counters. vmstat 1 5 go then take one sample every second, five times.

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  0      0 3216484  98304 1284360    0    0     4    18   62  110  3  1 96  0  0  0
 2  0      0 3216232  98304 1284360    0    0     0     0  248  431  6  2 84  0  8  0

Find st column inside cpu block for the right side. Current procps-ng builds dey print gu column after am for KVM guest time. So st dey second from the right, instead of the last one. Read the column with the header name, because that position don change between releases.

Two habits go help keep the reading accurate. The first data line na the average since boot, so ignore am and read the lines after am. One sample no be measurement, because steal time fit come in bursts. Run vmstat 1 60 and monitor one full minute before you draw conclusion.

top dey report the same number for its %Cpu(s) summary line, inside the field marked st:

%Cpu(s):  6.2 us,  2.1 sy,  0.0 ni, 83.9 id,  0.0 wa,  0.0 hi,  0.3 si,  7.5 st

For per-core detail, add sysstat:

sudo apt-get install -y sysstat
mpstat -P ALL 1 5

mpstat go print one row for each CPU, with %steal column. This one shows whether every vCPU dey affected or na only one. For the history wey support ticket need, save the samples instead of reading dem from the screen:

date -u | tee -a ~/steal.log
mpstat -P ALL 1 5 | tee -a ~/steal.log

Run that from cron during the hours wey you suspect. The file go help you show the provider the exact ten minutes, instead of only telling dem say "e feel slow last night".

Steal numbers mean wetin?

• A steady 0.0. E healthy, or the platform no dey report steal at all. Confirm am with systemd-detect-virt before you celebrate.
• Spikes of a few percent wey last seconds. Na normal for any shared node. Neighbour build fit start, or host fit dey run backups.
• 1 to 5 percent wey dey continue for shared plan. Na wetin person fit expect. Shared CPU na wetin the price dey cover.
• 5 to 10 percent wey dey continue. Na slowdown wey you fit measure. Start to record evidence, then compare the same hours across several days.
• Above 10 percent for hours at a time. The node get more workload than e fit handle for your workload. Na this level dey justify support ticket or move.

Use these bands as guide for reading the result, no be specification, because no provider dey publish steal guarantee for shared plan. Compare dem with wetin you dey run. Overnight batch job fit handle 15 percent steal and nobody go notice. Latency-sensitive service go show am for p99 long before average look alarming. Na why latency-sensitive workloads like trading bots suppose dey on dedicated cores.

Steal time dey cost you how much?

The calculation short. If dem take fraction s of your CPU time, a job wey need fixed CPU time go take 1 / (1 - s) times longer for clock. For job wey need 60 seconds of CPU:

[
  {
    "steal_percent": 0,
    "wall_clock_seconds": "60.0"
  },
  {
    "steal_percent": 3,
    "wall_clock_seconds": "61.9"
  },
  {
    "steal_percent": 8,
    "wall_clock_seconds": "65.2"
  },
  {
    "steal_percent": 15,
    "wall_clock_seconds": "70.6"
  },
  {
    "steal_percent": 25,
    "wall_clock_seconds": "80.0"
  },
  {
    "steal_percent": 40,
    "wall_clock_seconds": "100.0"
  }
]

For 3 percent, wey be normal shared-plan reading, that job go take 61.9 seconds instead of 60.0. Nobody go open ticket because of that. For 8 percent, e go be 65.2 seconds. For 40 percent, the same job go need 100.0 seconds, and queue wey dey clear before go start dey grow instead.

Those values na calculations, no be measurements. The model assume say na one runnable thread dey, and steal dey spread evenly across the interval. Real services often dey feel worse than the curve, because stolen CPU slice fit land inside request, and everything wey dey wait for that request go pay the delay again. To get your own figure instead of formula, benchmark the VPS during quiet hour and again during busy hour, while you record st for both windows.

Na e steal, or na another thing?

Steal fit look like other symptoms. Read all the counters together, for the same vmstat line.

• st high while r and us remain low: the host no dey give you the core. Na steal be that.
• r well pass your vCPU count, with us high and st almost zero: you dey run more work than your own CPUs fit handle. Compare r with the output of nproc. Na your own oversubscription be this, no be neighbour.
• wa high with st almost zero: tasks dey wait for storage. Na different problem with different solution.
• Load average high while st and us both dey low: load figure dey also count uninterruptible tasks. So this one usually point to stuck device or hung network mount, no be CPU.

Burstable plans need separate explanation. Dem give you credit balance wey dey build when you dey idle and dey reduce when you dey busy. When e finish, provider go hold you for baseline rate. For some platforms, dem report this throttle as steal. For others, you no fit see am from inside, and you simply get fewer cycles per second. Read the plan description before you decide say na neighbour cause the problem.

Why container dey report no steal time

Steal na property of virtual machine, no be container wey dey run inside am. Docker container for your own VPS dey share host /proc, so any st value wey you read inside am na the VPS steal, and na wetin you want. Container-based virtualisation wey dem sell as VPS dey behave differently. When lxcfs dey in place, /proc/stat inside container na cgroup accounting go synthesize am, and steal na zero by construction. Monitoring stack wey dey scrape only from inside fit show flat, calm zero while physical machine underneath dey starve.

Inside container, counter wey carry the same meaning na CPU quota throttling. For cgroup v2:

cat /sys/fs/cgroup/cpu.stat

nr_throttled dey count enforcement periods wey group hit its CPU quota, while throttled_usec dey total the time wey e spend frozen. If nr_throttled dey rise, e mean say your process dey ready to run but e no dey run, na the same experience as steal, but na limit wey you set yourself cause am. Check your own limits before you blame host, especially if you run your services for Docker on a VPS with CPU limits for compose file. Layered virtualisation dey add another place wey time fit disappear, because VM wey dey inside your VPS go pay your steal plus its own scheduling delay. Keep this for mind if you run nested virtualisation on a VPS.

Wetin to do about steal wey dey continue

No setting inside the guest fit fix steal, because scheduler wey dey make the decision dey outside the guest. Kernel upgrade no go change that too: cache aware scheduling wey dem add for Linux kernel 7.2 dey rearrange your tasks across the cores wey dem actually give you, and e no fit recover the cycles wey neighbour don already take. Four moves dey work.

Collect evidence first. Record timestamps for UTC, how long each episode last, how often e dey repeat, and whether mpstat dey show say na one vCPU dey affected or all of dem. One week of logged samples better pass screenshot.

Open ticket with that data. Ask two direct questions: node dey oversubscribed during these windows, and dem fit move my instance? Paste vmstat output and the exact times. Providers dey act when you show a window wey dem fit reproduce, and ticket wey only talk say server slow go get reply wey ask for one. How much of this work you fit hand over na one practical difference between managed and unmanaged VPS.

Ask for migration. Moving guest go node wey no too load na normal work for provider, and e usually na short reboot. This na the fix wey no cost anything, and e solve the common case, where one node happen to hold several heavy neighbours at the same time.

Buy your way out of contention. Dedicated vCPU plan reserve physical cores for your instance, so counter go sit for zero and remain there. E cost more every month, and na the honest answer for workload wey no fit absorb the variation. If that still no reach, or you want make memory bandwidth dey only for you too, the next step na dedicated server instead of VPS.

While you dey wait for any of these, reduce how much the steal dey affect you. Run fewer worker threads than the number of vCPUs wey you get, because threads wey no fit get core only add context switches. Move batch work go hours when node quiet, as your own log now dey show you. Then measure again with the same command across the same hours, so you fit talk whether the change work instead of guessing.

FAQ

CPU steal time wey normal for VPS na how much?

For shared plan, short spikes and value wey dey stay under about 5 percent dey normal, because shared CPU mean say host dey divide physical cores among different guests. If e dey stay for double digits for hours, e no normal and e worth opening ticket. For dedicated vCPU plan, expected reading na 0.0, so anything else na fault wey you suppose report. Compare the number with your own workload: overnight batch job fit tolerate steal, but latency-sensitive API no fit.

Bigger plan go fix high steal time?

No be by itself. More vCPUs for the same shared node mean say more virtual CPUs dey compete for the same physical cores wey get contention, and the percentage fit remain exactly as e be. Wetin dey remove steal na dedicated CPU allocation or move to node wey no too busy. Bigger share of busy machine still be share of busy machine.

Why my VPS dey show 0 steal time when e clearly slow?

Two common reasons dey. Hypervisor fit no export the counter at all, and this dey common for VMware and Hyper-V platforms, so the field go remain zero no matter wetin host dey do. Run systemd-detect-virt to see which platform you dey use. If no be that, bottleneck dey somewhere else: check wa for storage waits, compare r with nproc to check your own overload, and read /sys/fs/cgroup/cpu.stat inside containers for quota throttling.

I fit reduce steal time from inside my server?

You no fit change how host dey schedule work from inside guest. You fit only reduce how much e dey affect you. Run fewer worker threads than the number of vCPUs wey you get, so less work go sit for run queue dey wait for core wey no dey available. Move batch jobs to hours when node dey quieter. Cache results so fewer requests go need CPU at all. The changes wey really remove steal, like migration to another node or dedicated cores, dey for provider side.