How to Measure Tokens per Second for Local LLM
GPU rental only beat per-token billing after a throughput floor. Use a concurrency sweep to measure tokens per second well before you decide.
Why tokens per second dey decide whether GPU worth am
Tokens per second na the speed wey your server dey use produce output text. Na this number dey decide whether renting GPU go cheaper pass paying API per token. GPU box billing dey based on hour, whether e busy or idle. Hosted API billing dey based on token. So GPU only go win if you maintain high enough output rate for most of the hours wey you dey pay for.
This mean say you need measurement, no be figure wey you read somewhere. This page define the four numbers wey worth recording. E then give the commands wey go produce dem and the arithmetic wey go turn dem into decision.
Why published tokens per second figure no be your own number
DigitalOcean publish throughput figures for July 2026 for one NVIDIA H200 wey dey run llama3.3-70b-instruct for FP8 (8-bit floating point) under vLLM. The figures useful, but dem no be your own numbers.
Di data wey dey behind dis chart
[
{
"config": "H200, one stream",
"tok_s": "47"
},
{
"config": "H100, saturated",
"tok_s": "236"
},
{
"config": "H200, saturated",
"tok_s": "2,036"
},
{
"config": "H200, saturated, in+out",
"tok_s": "4,071.6"
}
]Every row wey dey above come from that page, and two of dem na the low end of a range wey the page give, so read those two as minimum values. We no measure anything for this chart.
Start with the last two rows. The headline na 4,071.6 tok/s, while the output-only rate na 2,036 tok/s. The headline count input and output tokens together. That test use 1,024 input tokens against 1,024 output tokens, so almost half of the headline na output. This split matter because output na the slower part. Prefill (reading the prompt) process all input tokens in one pass. Decode (writing the answer) produce one token at a time. Total-throughput headline dey average one cheap part with one expensive part.
Now look the first row. The same H200 wey dey serve one request at a time produce 47 tok/s, so the saturated figure pass forty times higher on the same hardware. This gap dey happen because one decode step leave the GPU waiting for memory for most of the time, and concurrent requests fill that idle time. The second row, 236 tok/s, na one H100 for the same model, but KV cache hold am back (the key and value cache, wey be the per-request memory wey served conversation keep for the card). An 80 GB card fit hold fewer concurrent requests for 70B model, so e saturate for lower rate.
If you change the model or change the input-to-output ratio, every number wey dey above go change. Published figures fit set your expectation, but dem no set your budget. Na the same rule wey apply when you dey benchmark VPS honestly for disk and network.
Numbers wey matter
- Time to first token, TTFT. Na the delay between when you send request and when the first output token arrive. E include prefill time plus queue time. User dey feel this one directly.
- Output tokens per second, per stream. Na how fast one answer dey write after e don start. Once e pass roughly 20 tok/s, e don already faster than how most people dey read, so extra speed for here no dey add much.
- Saturated total output throughput. Na the total across all concurrent streams when server fully loaded. Na this number show capacity, and na e dey pay GPU bill.
- p50 and p99 TTFT under concurrency. p50 na the middle request. p99 na the value wey 99 requests out of 100 go come under. Queueing always first show for p99.
The first two dey improve when server quiet. The third dey improve when server busy. Dem dey pull against each other, and na why no single number fit describe serving box.
Fix input and output length before you measure
Throughput dey depend on the shape of the traffic. Prompt wey get 4,000 tokens with answer wey get 50 tokens na prefill-heavy work. Prompt wey get 200 tokens with answer wey get 2,000 tokens na decode-heavy work. The same server fit report very different tokens per second for these two cases, so choose one ratio, write am beside every number wey you record, and never compare different ratios. 1,024 tokens in and 1,024 tokens out na reasonable default because several vendors publish results with that ratio. If you know your real traffic, use your real traffic.
Force the output length too. Model wey reach im stop token after 60 tokens go produce shorter run wey you no fit compare with full run: TTFT go take bigger part of the run, and this go reduce output throughput as the benchmark count am, even though each request finish faster. The --ignore-eos flag for the vLLM benchmark client makes every request generate exactly the number wey you request, so the two runs remain comparable. Model choice dey change these numbers more than any flag. fit a Qwen 3 model inside one VPS GPU cover the memory side of that choice.
Measure one stream first
Start with the simplest case. E dey serve as sanity check, and e also be ceiling. Ollama dey print its own timings.
ollama run llama3.1:8b --verbose "Write 200 words about disk scheduling."The line wey you need read na eval rate, wey be output tokens per second. prompt eval rate na the prefill rate, while load duration na the time wey Ollama spend to load the model into VRAM. For the first call after cold start, load duration dey large, so total duration no too reliable. Run the command two times, then read the second result. By default, Ollama dey unload idle model after five minutes, so if you wait long between runs, you go return to cold case.
The API dey return the same fields, and e easier to script.
curl -s http://127.0.0.1:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Write 200 words about disk scheduling.",
"stream": false
}' | jq '{eval_count, eval_duration,
tok_s: (.eval_count / (.eval_duration / 1000000000))}'eval_duration dey nanoseconds, so divide am by 1,000,000,000 to get seconds. Na exactly this division Ollama own API documentation recommend for tokens per second. If the server never start yet, how to self-host an LLM with Ollama on a VPS explain the installation and the systemd unit.
TTFT need streaming request, and curl fit measure the time for you.
curl -s -o /dev/null -N \
-w 'ttfb=%{time_starttransfer}s total=%{time_total}s\n' \
http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3-8B",
"messages":[{"role":"user","content":"Write 200 words about disk scheduling."}],
"stream":true,"max_tokens":256}'time_starttransfer na the time wey the first byte of the response body reach. For streaming chat completion, that byte belong to the first server-sent event. E fit be the first content token, or role-only delta wey server send just before am. So treat the value as TTFT, with difference of about one event. E accurate enough to compare two runs for the same server.
Single-stream numbers fit make the box look better in two ways. TTFT dey at the best level wey e fit ever reach, because no request dey queue before your own. The per-stream rate dey at the best level too, because the whole card dey serve one request. Neither result tell you wetin the box fit carry.
How you fit run concurrency sweep?
Sweep dey run one fixed workload with concurrency wey dey increase, then e record wetin happen for every step. vLLM dey ship the client for this work, and e dey speak OpenAI API, so e also work against Ollama and anything else wey support OpenAI compatibility.
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:8000 \
--endpoint /v1/chat/completions \
--model Qwen/Qwen3-8B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--ignore-eos \
--num-prompts 160 \
--max-concurrency 16 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,99--max-concurrency dey limit the requests wey dey in flight, and na this variable you dey sweep. --num-prompts na the total requests wey you send, so keep am near ten times the concurrency to get stable average. The summary go print Output token throughput (tok/s): and Total token throughput (tok/s):, then Mean TTFT (ms):, Median TTFT (ms): and P99 TTFT (ms): under Time to First Token heading.
That output no get rate per stream, but na only one division remain. Mean TPOT (ms): na the average time for every output token after the first one, so 25 ms per token na 40 tokens per second for each stream. If you divide output throughput by concurrency, you go get the same answer.
Then loop am and save every run.
for C in 1 4 16 32 64 128; do
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:8000 \
--endpoint /v1/chat/completions \
--model Qwen/Qwen3-8B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--ignore-eos \
--num-prompts $(( C * 10 )) \
--max-concurrency "$C" \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,99 \
--save-result --result-filename "sweep-c$C.json"
doneHow to read the saved JSON files
Every run dey write one file, so pull the fields wey you need from all of dem at once.
for f in sweep-c*.json; do
jq -r --arg f "$f" \
'[$f, .output_throughput, .total_token_throughput,
.median_ttft_ms, .p99_ttft_ms] | @tsv' "$f"
doneoutput_throughput na output tokens per second. total_token_throughput add the input tokens back, so for 1:1 ratio e dey close to double. p99_ttft_ms for dey there even without --metric-percentiles, because that flag default na 99. The 50 for the command na wetin add p50_ttft_ms beside median_ttft_ms wey every metric named inside --percentile-metrics dey get. If you ask for percentile wey you no request, like .p90_ttft_ms, jq go print null.
Wetin concurrency sweep really dey show?
Di data wey dey behind dis chart
[
{
"label": "1 stream",
"per_stream_tok_s": 92,
"total_tok_s": 92,
"ttft_p50_ms": 48,
"ttft_p99_ms": 61
},
{
"label": "4 streams",
"per_stream_tok_s": 88,
"total_tok_s": 352,
"ttft_p50_ms": 71,
"ttft_p99_ms": 96
},
{
"label": "16 streams",
"per_stream_tok_s": 71,
"total_tok_s": 1136,
"ttft_p50_ms": 152,
"ttft_p99_ms": 244
},
{
"label": "32 streams",
"per_stream_tok_s": 54,
"total_tok_s": 1728,
"ttft_p50_ms": 287,
"ttft_p99_ms": 498
},
{
"label": "64 streams",
"per_stream_tok_s": 34,
"total_tok_s": 2176,
"ttft_p50_ms": 611,
"ttft_p99_ms": 1240
},
{
"label": "128 streams",
"per_stream_tok_s": 18,
"total_tok_s": 2304,
"ttft_p50_ms": 1490,
"ttft_p99_ms": 3820
}
]Those 6 rows na example of the shape wey sweep dey produce for small rentable GPU box, using order of magnitude wey make sense. Dem no be measurement from your server, and dem no be vendor figure. Run the loop above and replace dem with your own values.
Read the shape, because na the shape wey dey generalise. With one stream, the whole box dey produce 92 tokens per second with p99 TTFT of 61 ms. For 128 streams, total reach 2304 tokens per second, wey twenty-five times higher, while each individual stream drop to 18 tokens per second and p99 TTFT reach 3820 ms. Total throughput dey rise because batching dey turn idle memory waits to useful work. Per-stream speed dey fall because na the same compute dem dey share now.
The last doubling na the sign. From 64 to 128 streams, total only add under six percent, while p99 TTFT nearly triple. This mean say requests dey queue instead of running, either because KV cache full or because server don reach scheduling cap like vLLM's max_num_seqs. The useful operating point dey earlier: for 32 streams, the box still return 1728 tokens per second, 75 percent of its peak, at 54 tokens per second for each stream and p99 TTFT of 498 ms. Report that point as your capacity. You no fit use the peak of the curve to serve users.
Ollama and vLLM no dey measure the same thing
Run that sweep against default Ollama server, and the total go barely change. OLLAMA_NUM_PARALLEL default na 1, so one request go run while the rest dey wait. Na the queue dey make p99 TTFT climb while total output stay flat. Raise am before you measure anything.
sudo systemctl edit ollama.service[Service]
Environment="OLLAMA_NUM_PARALLEL=8"Restart with sudo systemctl restart ollama, then confirm say the model still fit. Each parallel slot get its own share of the context window, so Ollama documentation note say 2K context with 4 parallel requests dey allocate 8K. If you raise the slot count too high, the model go spill out of VRAM. Check ollama ps: if PROCESSOR column show something like 48%/52% CPU/GPU, e mean say part of the model dey for CPU, and throughput go now fall as you add concurrency instead of rising. After the parallel slots don fill, requests go queue up to OLLAMA_MAX_QUEUE, wey default na 512. After that, server go answer 503.
vLLM dey use continuous batching, so e dey admit new requests into the running batch as slots dey free up. The curve go continue climb until something finish: usually KV cache. But configured cap like max_num_seqs or CPU scheduling overhead fit make am flatten earlier. Ollama dey optimise for one model, one machine, and low setup cost. So the two engines go give different answers to the same sweep. Na this be the real subject of Ollama and vLLM compared as serving engines. Record which engine and which version produce every number.
Ways five to measure the wrong thing
- The client dey far away. If you benchmark from your laptop through internet, your round trip go enter every TTFT, so na your home connection you dey measure. Run the client for the same region where the server dey.
- The model cold. The first request go pay for weight loading, and for vLLM e fit also pay for graph capture. Send one warmup batch, then throw the result away.
- Prefix caching answer for you. vLLM dey enable automatic prefix caching by default. So if you send the same prompt again and again, you dey measure the cache instead of prefill, and TTFT go fall to small part of the real value.
--dataset-name randomavoid this because every prompt dey different. To make sure, start the server with--no-enable-prefix-caching. - The outputs short. With 32-token answers, TTFT dey dominate every request, and your tokens per second really dey describe prefill. Use
--ignore-eoswith output length wey realistic. - You report only concurrency 1. E gives the friendliest TTFT for the sheet, and na the correct cost input only if your real requests never overlap. If dem overlap, cost depend on the throughput wey you sustain at the concurrency wey your traffic actually dey produce. So measure for there too.
Turn your measured number into decision
Take the output throughput wey your sweep show for the concurrency wey your real traffic dey reach, then compare am with per-token pricing. That na saturated figure only if enough requests overlap to keep the batch full; workload wey dey send one request at a time must use the single-stream rate. Break-even na one division, once you price the input tokens wey dey come with every output token:
cost_per_million_output = price_out + price_in * (input_tokens / output_tokens)
break_even_tok_s = (price_per_hour / cost_per_million_output) * 1000000 / 3600Make we work am with DigitalOcean prices for July 2026. Their H200 dedicated inference endpoint na $4.47 per hour, while serverless Llama 3.3 70B na $0.65 per million tokens for input and $0.65 per million for output. For the 1:1 ratio wey we use above, every million output tokens bring one million input tokens with am, so the effective price na $1.30 per million output tokens. So 4.47 divided by 1.30 na about 3.44 million output tokens per hour, and if you divide by 3,600 seconds, e go be about 955 output tokens per second, or about 1,910 tokens per second when you count input and output together. Na dem get the prices. Na we do the division. When we check am on 25 October 2026, the H200 endpoint still dey cost $4.47 per hour, but DigitalOcean deprecated serverless Llama 3.3 70B on 18 August 2026 and dey direct users to Llama 4 Maverick, wey dem list at $0.25 per million input tokens and $0.87 per million output tokens. Na different model be that, so e need its own sweep before the comparison fit mean anything.
The word wey decide this na sustained. If you reach 955 output tokens per second at saturation for two hours every day, that no mean say your sustained rate na 955 output tokens per second, because you still dey pay for the other twenty-two hours. Renting the card as GPU Droplet no reduce the bar: on 25 October 2026, DigitalOcean's Droplet pricing list H200 at $4.47 per hour on demand, the same price as the endpoint, and its cheapest NVIDIA Droplets, RTX 4000 at $0.76 per hour and L40S or RTX 6000 at $1.57 per hour, too small to hold 70B model for FP8. So based on the figures above, H200 wey saturate at 2,036 output tokens per second must average about 47 percent of that peak across every hour wey you pay for, just to match $0.65 per million tokens in and out. Below that point, per-token price win. Na idle GPU hours, not slow tokens, dey usually sink self-hosting.
So your decision get two inputs. Sweep go give you the ceiling. Your traffic pattern go give you the fraction of that ceiling wey you actually collect. Multiply dem, then take the result to GPU VPS against per-token API break-even and read the answer for your volume.
FAQ
Tokens per second wey good for self-hosted LLM na how much?
Two answers dey, because this metric dey do two different jobs. If na one person dey read the output, anything wey pass roughly 20 output tokens per second per stream don already pass reading speed, so extra speed no go help. For cost, the important number na the total output throughput wey you fit sustain under your real load. “Good” mean any number wey fit clear your break-even point. If price na $0.65 per million tokens, for input and output, and the machine cost $4.47 per hour, that floor dey near 955 output tokens per second sustained at 1:1 ratio and July 2026 prices. One stream for large model no go reach am, and na why batching dey exist.
Why my Ollama throughput dey remain flat when I add concurrent requests?
OLLAMA_NUM_PARALLEL default na 1, so the server dey run one request at a time for each model and queue the remaining ones, up to OLLAMA_MAX_QUEUE (512 by default) before e return 503. Total output dey remain flat while p99 TTFT dey increase. This one show say queue dey form, not say GPU dey busy. Set the variable inside a systemd drop-in and restart. Then check ollama ps, because every parallel slot dey multiply the context wey system allocate and fit push part of the model go CPU.
I suppose measure time to first token or tokens per second?
Measure both, because dem dey move for opposite directions as load dey increase. TTFT na wetin user dey feel, while saturated output throughput na wetin your invoice dey reflect. Record p50 and p99 TTFT for every concurrency step. Then choose the highest concurrency where p99 TTFT still acceptable to you. Report the throughput for that point as your capacity, not the maximum value from the top of the curve.
Higher tokens per second always mean lower cost per token?
No. Cost per token na the hourly price divided by the tokens wey the machine actually produce for that hour. So fast server wey dey idle most of the day still get high cost per token. Utilisation na wetin decide am, not peak speed. Also watch the units: quoted total token throughput dey count input tokens, so for 1:1 input-to-output ratio, e dey near double the output rate. If the API wey you dey compare against dey bill input tokens together with output, price both sides before you compare.