How to measure tokens per second for local LLM

Why tokens per second dey decide weda GPU go pay

Tokens per second na the speed wey your server take dey produce output text, and na this number dey decide weda to rent GPU go cheaper pass to pay API per token. Dem dey bill GPU box per hour weda e dey busy or e dey idle. Dem dey bill hosted API per token. So, GPU go only pay if you fit maintain high output rate for most of the hours wey you dey pay for.

Dis mean say you need measurement, no bi just figure wey you read for somewhere. Dis page define the four numbers wey you suppose record, den e give the commands wey go produce dem and the calculation wey go help you decide.

Why di tokens per second wey dem publish no be your own number

DigitalOcean publish throughput figures for July 2026 for one NVIDIA H200 wey dey run llama3.3-70b-instruct for FP8 (8-bit floating point) under vLLM. Dem figures dey useful but dem no be your own.

[
  {
    "config": "H200, one stream",
    "tok_s": "47"
  },
  {
    "config": "H100, saturated",
    "tok_s": "236"
  },
  {
    "config": "H200, saturated",
    "tok_s": "2,036"
  },
  {
    "config": "H200, saturated, in+out",
    "tok_s": "4,071.6"
  }
]

Every row wey dey up so na wetin dem quote from dat page, and two of dem na di low end of di range wey dem give, so make you read dose two as di minimum. Nothing for dis chart na we measure.

Make we start with di last two rows. Di headline na 4,071.6 tok/s, while di output-only rate na 2,036 tok/s. Di headline count input and output tokens join. Dat test use 1,024 input tokens against 1,024 output tokens, so almost exactly half of di headline na output. Di split dey important sake of say output na di half wey dem go bill you for, and e be di slow half. Prefill (di time wey e dey read prompt) dey process all input tokens for one pass. Decode (di time wey e dey write answer) dey produce one token at a time. Total-throughput headline dey average cheap number with expensive one.

Now di first row. Di same H200 wey dey serve one request at a time dey produce 47 tok/s, so di saturated figure dey pass forty times higher for di same hardware. Dat gap dey because one decode step dey make GPU wait for memory for most of di time, and concurrent requests dey fill dat idle time. Di second row, 236 tok/s, na one H100 for di same model, wey KV cache (di key and value cache, di per-request memory wey served conversation dey keep for card) dey hold back. One 80 GB card dey hold fewer concurrent requests for 70B model, so e dey saturate lower.

If you change model or change di input to output ratio, every number wey dey up so go move. Published figures dey set your expectations, no be your budget, and na di same rule wey apply to how to do honest benchmarking for VPS for disk and network.

Di four numbers wey dey important

• Time to first token, TTFT. Di delay wey dey between wen you send request and wen di first output token show. E include prefill time plus queue time. Na dis one user dey feel direct.
• Output tokens per second, per stream. How fast one answer dey write once e don start. Once e pass like 20 tok/s, e don fast pass how most pipo dey read, so extra speed no too get benefit again.
• Saturated total output throughput. Di total sum of all concurrent streams wen di server full well-well. Dis one na di capacity number, and na dis one dey pay di GPU bill.
• p50 and p99 TTFT under concurrency. p50 na di middle request. p99 na di value wey 99 out of 100 requests no go pass. Queueing always dey show for p99 first.

Di first two dey better wen di server dey free. Di third one dey better wen di server dey busy. Dem dey pull against each other, na why no single number fit describe how one server dey perform.

Make sure say input and output lengths dey correct before you measure

Throughput dey depend on how the traffic look. If you get 4,000-token prompt wey get 50-token answer, that one na prefill-heavy work. If you get 200-token prompt wey get 2,000-token answer, that one na decode-heavy work. The same server go show different tokens per second for these two, so choose one ratio, write am near every number wey you record, and no try compare different ratios. 1,024 in and 1,024 out na good standard because many vendors dey use that ratio. If you know how your real traffic dey look, use your real traffic.

Force the output length too. If model reach e stop token after 60 tokens, the run go short and e go look fast, because TTFT go come be big part of the time. The --ignore-eos flag for the vLLM benchmark client go make every request generate exactly the amount wey you ask for, so the two runs go dey comparable. The model wey you choose go affect these numbers pass any flag: how to fit Qwen 3 model for one single VPS GPU explain the memory side of that choice.

Measure one stream first

Start with the simplest case. E be sanity check and e be ceiling. Ollama dey print im own timings.

ollama run llama3.1:8b --verbose "Write 200 words about disk scheduling."

The line wey you go read na eval rate, wey be output tokens per second. prompt eval rate na the prefill rate, and load duration na the time wey dem spend dey load the model enter VRAM. For the first call after cold start, load duration go big, so total duration fit deceive you. Run the command two times and read the second result. Ollama dey unload model wey no dey do anything after five minutes by default, so if you wait too long between runs, you go go back to cold start case.

The same fields dey come from the API, wey easy to script.

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write 200 words about disk scheduling.",
  "stream": false
}' | jq '{eval_count, eval_duration,
         tok_s: (.eval_count / (.eval_duration / 1000000000))}'

eval_duration na nanoseconds, so if you divide am by 1,000,000,000, you go get seconds. That division na exactly wetin Ollama own API documentation talk say you must do for tokens per second. If the server never start, self-hosting an LLM with Ollama on a VPS cover how to install am and the systemd unit.

TTFT need streaming request, and curl fit time that one for you.

curl -s -o /dev/null -N \
  -w 'ttfb=%{time_starttransfer}s  total=%{time_total}s\n' \
  http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-8B",
       "messages":[{"role":"user","content":"Write 200 words about disk scheduling."}],
       "stream":true,"max_tokens":256}'

time_starttransfer na the moment wey the first byte of the response body arrive. For streaming chat completion, that byte belong to the first server-sent event, wey fit be the first content token or role-only delta wey dem send just before am. So, treat the value as TTFT, give or take one event. E accurate reach to compare two runs for the same server.

Single-stream numbers dey flatter the box two times. The TTFT na the best e go ever be, because nothing dey queue ahead of you. The per-stream rate na the best e go ever be, because the whole card dey serve one request. None of dem talk wetin the box fit carry.

How you go run concurrency sweep?

One sweep dey run one fixed workload as concurrency dey increase, and e dey record wetin happen for every step. vLLM get the client wey you go use for this, and because e dey speak OpenAI API, e fit work with Ollama and any other thing wey be OpenAI-compatible.

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-8B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --ignore-eos \
  --num-prompts 160 \
  --max-concurrency 16 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,99

--max-concurrency dey limit the requests wey dey fly, and na this be the variable wey you dey sweep. --num-prompts na the total wey dem send, so make you keep am near ten times the concurrency so you go fit get stable average. The summary go print Output token throughput (tok/s): and Total token throughput (tok/s):, then Mean TTFT (ms):, Median TTFT (ms): and P99 TTFT (ms): under one Time to First Token heading.

No per-stream rate for that output, but you fit calculate am with one simple division. Mean TPOT (ms): na the mean time per output token after the first one, so 25 ms per token mean say 40 tokens per second per stream. If you divide output throughput by the concurrency, you go get the same answer.

After that, make you loop am and save every run.

for C in 1 4 16 32 64 128; do
  vllm bench serve \
    --backend openai-chat \
    --base-url http://127.0.0.1:8000 \
    --endpoint /v1/chat/completions \
    --model Qwen/Qwen3-8B \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 1024 \
    --ignore-eos \
    --num-prompts $(( C * 10 )) \
    --max-concurrency "$C" \
    --percentile-metrics ttft,tpot,itl,e2el \
    --metric-percentiles 50,99 \
    --save-result --result-filename "sweep-c$C.json"
done

for f in sweep-c*.json; do
  jq -r --arg f "$f" \
    '[$f, .output_throughput, .total_token_throughput,
      .median_ttft_ms, .p99_ttft_ms] | @tsv' "$f"
done

Wetin concurrency sweep actually dey show?

[
  {
    "label": "1 stream",
    "per_stream_tok_s": 92,
    "total_tok_s": 92,
    "ttft_p50_ms": 48,
    "ttft_p99_ms": 61
  },
  {
    "label": "4 streams",
    "per_stream_tok_s": 88,
    "total_tok_s": 352,
    "ttft_p50_ms": 71,
    "ttft_p99_ms": 96
  },
  {
    "label": "16 streams",
    "per_stream_tok_s": 71,
    "total_tok_s": 1136,
    "ttft_p50_ms": 152,
    "ttft_p99_ms": 244
  },
  {
    "label": "32 streams",
    "per_stream_tok_s": 54,
    "total_tok_s": 1728,
    "ttft_p50_ms": 287,
    "ttft_p99_ms": 498
  },
  {
    "label": "64 streams",
    "per_stream_tok_s": 34,
    "total_tok_s": 2176,
    "ttft_p50_ms": 611,
    "ttft_p99_ms": 1240
  },
  {
    "label": "128 streams",
    "per_stream_tok_s": 18,
    "total_tok_s": 2304,
    "ttft_p50_ms": 1490,
    "ttft_p99_ms": 3820
  }
]

Dem 6 rows na example of how di shape wey sweep dey produce for small GPU box wey you rent dey look, base on correct magnitude. No be measurement of your own server, and no be wetin vendor talk. Run di loop wey dey up so, come replace dem with your own numbers.

Read di shape, because na di shape dey show wetin dey happen generally. For one stream, di whole box dey produce 92 tokens per second with p99 TTFT of 61 ms. For 128 streams, di total reach 2304 tokens per second, wey be twenty-five times higher, but each individual stream come fall go 18 tokens per second and p99 TTFT reach 3820 ms. Total throughput dey rise because batching dey turn idle memory wait into work wey dey useful. Per-stream speed dey fall because di same compute power now dey shared.

Di last doubling na di main sign. To move from 64 go 128 streams add less than six percent to di total, while p99 TTFT almost triple; dis one mean say KV cache don full and requests dey queue instead of say dem dey run. Di point wey you suppose dey operate na before dat one: for 32 streams, di box still dey return 1728 tokens per second, wey be 75 percent of im peak, at 54 tokens per second per stream and p99 TTFT of 498 ms. Use dat point as your capacity. Di peak of di curve na number wey you no fit use serve users.

Ollama and vLLM no dey measure the same tin

Run dat sweep against one default Ollama server and di total go barely move. OLLAMA_NUM_PARALLEL defaults to 1, so one request dey run while di rest dey wait, and na dat queue dey make p99 TTFT climb while total output remain flat. Raise am before you measure anything.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=8"

Restart with sudo systemctl restart ollama, den confirm say di model still fit. Each parallel slot dey get im own share of di context window, so Ollama documentation talk say 2K context wit 4 parallel requests dey allocate 8K. Raise di slot count reach level wey di model go spill comot from VRAM. Check ollama ps: one PROCESSOR column wey dey read like 48%/52% CPU/GPU mean say part of di model dey on top CPU, and throughput go come fall as you dey add concurrency instead of e go up. Past di parallel slots, requests dey queue up to OLLAMA_MAX_QUEUE, 512 by default, after dat di server go answer 503.

vLLM dey use continuous batching, so e dey admit new requests into di running batch as slots dey free up, and im curve dey continue to climb until KV cache finish. Ollama dey optimize for one model, one machine, low setup cost. Di two engines therefore dey give different answers to di same sweep, wey be di real subject of Ollama and vLLM compared as serving engines. Record which engine and which version produce every number.

Five ways to measure the wrong thing

• The client dey too far. If you dey benchmark from your laptop over internet, you go add your own round trip time to every TTFT, so na your home connection you dey measure. Run the client for the same region as the server.
• The model no warm up. The first request dey pay for weight loading, and for vLLM, e fit still pay for graph capture. Send one warmup batch and throw the result comot.
• Prefix caching answer for you. vLLM dey enable automatic prefix caching by default, so if you send the same prompt over and over, na the cache you dey measure instead of prefill, and TTFT go come small pass the real value. --dataset-name random dey avoid this because every prompt different. To make sure, start the server with --no-enable-prefix-caching.
• The outputs too short. With 32-token answers, TTFT go dominate every request and your tokens per second go just dey describe prefill. Use --ignore-eos with one realistic output length.
• You report concurrency 1. Na the friendliest number for the sheet and e no get any connection to cost.

Turn your measured number into a decision

Take the saturated output throughput wey you get from your sweep, no be the single-stream rate, come compare am with per-token pricing. The break-even na simple division:

break_even_tok_s = (price_per_hour / price_per_million_output_tokens) * 1000000 / 3600

Make we calculate am with DigitalOcean July 2026 prices. Their H200 dedicated inference endpoint na $4.47 per hour, and the serverless version na $0.65 per million tokens. So, 4.47 divide by 0.65 give us 6.88 million tokens per hour, and if you divide that one by 3,600 seconds, you go get about 1,910 output tokens per second. The prices na their own. The division na our own.

The word wey go help you decide na sustained. If you hit 1,910 tokens per second for saturation for only two hours per day, that one no be 1,910 tokens per second sustained, because you go still pay for the other twenty-two hours. DigitalOcean own crossover for the cheaper $3.44 per hour GPU Droplet dey happen for 72.2 percent sustained average utilisation; if your usage dey below that, the per-token price go dey cheaper. Idle GPU hours, no be slow tokens, na wetin dey usually make self-hosting cost pass wetin e suppose be.

So, your decision get two inputs. The sweep give you the ceiling. Your traffic pattern give you the fraction of that ceiling wey you actually dey use. Multiply dem, then take the result go the GPU VPS against per-token API break-even come check the answer for your volume.

FAQ

Wetin be good tokens per second for self-hosted LLM?

Two answers dey, because this metric get two different work. For one person wey dey read wetin the AI bring out, anything wey pass like 20 output tokens per second for one stream don fast pass reading speed, so e no get extra benefit. For money matter, the number wey concern you na the total output throughput wey full ground, and "good" mean any speed wey make you recover your money. If you compare am with $0.65 per million tokens for one box wey cost $4.47 per hour, the minimum speed suppose dey near 1,910 output tokens per second wey dey steady for July 2026 prices. One stream for big model no fit reach that level, na why batching dey.

Why my Ollama throughput no dey move when I add concurrent requests?

OLLAMA_NUM_PARALLEL defaults to 1, so the server dey run one request at a time per model and dey queue the rest, up to OLLAMA_MAX_QUEUE (512 by default) before e return 503. Total output no dey change while p99 TTFT dey go up, na the sign be that say queue dey, no be say GPU busy. Set the variable for one systemd drop-in and restart, then check ollama ps, because every parallel slot dey multiply the context wey you allocate and fit push part of the model go CPU.

I suppose measure time to first token or tokens per second?

Both, because dem dey move opposite direction as load dey increase. TTFT na wetin user dey feel, and the total output throughput wey full ground na wetin your invoice go show. Record p50 and p99 TTFT for every concurrency step, then choose the highest concurrency wey p99 TTFT still dey okay for you. Report the throughput for that point as your capacity, no be the maximum wey you see for the top of the curve.

Higher tokens per second number mean say cost per token go always low?

No. Cost per token na the hourly price divide by the tokens wey the box actually produce for that hour, so fast server wey dey idle most of the day still get high cost per token. Na utilisation decide am, no be peak speed. Watch the units too: the total token throughput wey dem quote dey count input tokens, so if input to output ratio na 1:1, e dey close to double the output rate wey dem dey bill you.