Measure tokens per second on a local LLM

Why tokens per second decides whether a GPU pays

Tokens per second is the rate at which your server produces output text, and it is the number that decides whether renting a GPU is cheaper than paying an API per token. A GPU box is billed by the hour whether it is busy or idle. A hosted API is billed by the token. So the GPU only wins if you hold a high enough output rate across most of the hours you pay for.

That means you need a measurement, not a figure you read somewhere. This page defines the four numbers worth recording, then gives the commands that produce them and the arithmetic that turns them into a decision.

Why a published tokens per second figure is not your number

DigitalOcean published throughput figures in July 2026 for a single NVIDIA H200 running llama3.3-70b-instruct in FP8 (8-bit floating point) under vLLM. The figures are useful and they are not yours.

[
  {
    "config": "H200, one stream",
    "tok_s": "47"
  },
  {
    "config": "H100, saturated",
    "tok_s": "236"
  },
  {
    "config": "H200, saturated",
    "tok_s": "2,036"
  },
  {
    "config": "H200, saturated, in+out",
    "tok_s": "4,071.6"
  }
]

Every row above is quoted from that page, and two of them are the low end of a range it gives, so read those two as a floor. Nothing in this chart was measured by us.

Start with the last two rows. The headline is 4,071.6 tok/s, while the output-only rate is 2,036 tok/s. The headline counts input and output tokens together. That test used 1,024 input tokens against 1,024 output tokens, so almost exactly half of the headline is output. The split matters because output is the half you are billed on, and it is the slow half. Prefill (reading the prompt) processes all input tokens in one pass. Decode (writing the answer) produces one token at a time. A total-throughput headline averages a cheap number with an expensive one.

Now the first row. The same H200 serving one request at a time produces 47 tok/s, so the saturated figure is more than forty times higher on identical hardware. That gap exists because a single decode step leaves the GPU waiting on memory for most of its time, and concurrent requests fill that idle time. The second row, 236 tok/s, is a single H100 on the same model, held back by KV cache (the key and value cache, the per-request memory a served conversation keeps on the card). An 80 GB card holds fewer concurrent requests for a 70B model, so it saturates lower.

Change the model or change the input to output ratio and every number above moves. Published figures set your expectations, not your budget, which is the same rule that applies to benchmarking a VPS honestly on disk and network.

The four numbers that matter

• Time to first token, TTFT. The delay between sending a request and the first output token arriving. It is prefill time plus queue time. A user feels this one directly.
• Output tokens per second, per stream. How fast one answer is written once it has started. Above roughly 20 tok/s it is already faster than most people read, so extra speed here buys little.
• Saturated total output throughput. The sum across all concurrent streams with the server fully loaded. This is the capacity number, and it is the one that pays the GPU bill.
• p50 and p99 TTFT under concurrency. p50 is the middle request. p99 is the value that 99 requests in 100 come in under. Queueing always shows up in p99 first.

The first two get better when the server is quiet. The third gets better when the server is busy. They pull against each other, which is why no single number describes a serving box.

Fix the input and output lengths before you measure

Throughput depends on the shape of the traffic. A 4,000-token prompt with a 50-token answer is prefill-heavy work. A 200-token prompt with a 2,000-token answer is decode-heavy work. The same server reports very different tokens per second for those two, so pick one ratio, write it next to every number you record, and never compare across ratios. 1,024 in against 1,024 out is a reasonable default because several vendors publish at that ratio. If you know your real traffic, use your real traffic.

Force the output length too. A model that hits its stop token after 60 tokens gives a shorter run that looks faster, because TTFT is then a larger share of it. The --ignore-eos flag in the vLLM benchmark client makes every request generate exactly the requested count, so two runs stay comparable. Model choice moves these numbers more than any flag does: fitting a Qwen 3 model onto a single VPS GPU covers the memory side of that choice.

Measure one stream first

Start with the simplest case. It is a sanity check, and it is a ceiling. Ollama prints its own timings.

ollama run llama3.1:8b --verbose "Write 200 words about disk scheduling."

The line to read is eval rate, which is output tokens per second. prompt eval rate is the prefill rate, and load duration is time spent loading the model into VRAM. On the first call after a cold start load duration is large, so total duration is misleading. Run the command twice and read the second result. Ollama unloads an idle model after five minutes by default, so a long pause between runs puts you back in the cold case.

The same fields come from the API, which is easier to script.

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write 200 words about disk scheduling.",
  "stream": false
}' | jq '{eval_count, eval_duration,
         tok_s: (.eval_count / (.eval_duration / 1000000000))}'

eval_duration is nanoseconds, so dividing by 1,000,000,000 gives seconds. That division is exactly what Ollama's own API documentation prescribes for tokens per second. If the server is not up yet, self-hosting an LLM with Ollama on a VPS covers the install and the systemd unit.

TTFT needs a streaming request, and curl can time that for you.

curl -s -o /dev/null -N \
  -w 'ttfb=%{time_starttransfer}s  total=%{time_total}s\n' \
  http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-8B",
       "messages":[{"role":"user","content":"Write 200 words about disk scheduling."}],
       "stream":true,"max_tokens":256}'

time_starttransfer is the moment the first byte of the response body arrives. On a streaming chat completion that byte belongs to the first server-sent event, which is either the first content token or a role-only delta sent just before it. So treat the value as TTFT give or take one event. It is accurate enough to compare two runs on the same server.

Single-stream numbers flatter the box twice. The TTFT is the best it will ever be, because nothing is queued ahead of you. The per-stream rate is the best it will ever be, because the whole card is serving one request. Neither says what the box can carry.

How do you run a concurrency sweep?

A sweep runs one fixed workload at rising concurrency and records what happens at each step. vLLM ships the client for this, and it speaks the OpenAI API, so it also works against Ollama and anything else OpenAI-compatible.

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-8B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --ignore-eos \
  --num-prompts 160 \
  --max-concurrency 16 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,99

--max-concurrency caps the requests in flight, and it is the variable you are sweeping. --num-prompts is the total sent, so keep it near ten times the concurrency to get a stable average. The summary prints Output token throughput (tok/s): and Total token throughput (tok/s):, then Mean TTFT (ms):, Median TTFT (ms): and P99 TTFT (ms): under a Time to First Token heading.

There is no per-stream rate in that output, but it is one division away. Mean TPOT (ms): is the mean time per output token after the first, so 25 ms per token is 40 tokens per second per stream. Dividing output throughput by the concurrency gives the same answer.

Then loop it, saving each run.

for C in 1 4 16 32 64 128; do
  vllm bench serve \
    --backend openai-chat \
    --base-url http://127.0.0.1:8000 \
    --endpoint /v1/chat/completions \
    --model Qwen/Qwen3-8B \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 1024 \
    --ignore-eos \
    --num-prompts $(( C * 10 )) \
    --max-concurrency "$C" \
    --percentile-metrics ttft,tpot,itl,e2el \
    --metric-percentiles 50,99 \
    --save-result --result-filename "sweep-c$C.json"
done

for f in sweep-c*.json; do
  jq -r --arg f "$f" \
    '[$f, .output_throughput, .total_token_throughput,
      .median_ttft_ms, .p99_ttft_ms] | @tsv' "$f"
done

What does a concurrency sweep actually show?

[
  {
    "label": "1 stream",
    "per_stream_tok_s": 92,
    "total_tok_s": 92,
    "ttft_p50_ms": 48,
    "ttft_p99_ms": 61
  },
  {
    "label": "4 streams",
    "per_stream_tok_s": 88,
    "total_tok_s": 352,
    "ttft_p50_ms": 71,
    "ttft_p99_ms": 96
  },
  {
    "label": "16 streams",
    "per_stream_tok_s": 71,
    "total_tok_s": 1136,
    "ttft_p50_ms": 152,
    "ttft_p99_ms": 244
  },
  {
    "label": "32 streams",
    "per_stream_tok_s": 54,
    "total_tok_s": 1728,
    "ttft_p50_ms": 287,
    "ttft_p99_ms": 498
  },
  {
    "label": "64 streams",
    "per_stream_tok_s": 34,
    "total_tok_s": 2176,
    "ttft_p50_ms": 611,
    "ttft_p99_ms": 1240
  },
  {
    "label": "128 streams",
    "per_stream_tok_s": 18,
    "total_tok_s": 2304,
    "ttft_p50_ms": 1490,
    "ttft_p99_ms": 3820
  }
]

Those 6 rows are an illustration of the shape a sweep produces on a small rentable GPU box, at plausible orders of magnitude. They are not a measurement of your server, and they are not a vendor figure. Run the loop above and replace them with yours.

Read the shape, because the shape is what generalises. At one stream the whole box produces 92 tokens per second with a p99 TTFT of 61 ms. At 128 streams the total reaches 2304 tokens per second, twenty-five times higher, while each individual stream falls to 18 tokens per second and p99 TTFT reaches 3820 ms. Total throughput rises because batching turns idle memory waits into useful work. Per-stream speed falls because the same compute is now shared.

The last doubling is the tell. Going from 64 to 128 streams adds under six percent to the total while p99 TTFT roughly triples, which means KV cache is full and requests are queueing rather than running. The useful operating point is earlier: at 32 streams the box still returns 1728 tokens per second, 75 percent of its peak, at 54 tokens per second per stream and a p99 TTFT of 498 ms. Report that point as your capacity. The peak of the curve is a number you cannot serve users from.

Ollama and vLLM do not measure the same thing

Run that sweep against a default Ollama server and the total will barely move. OLLAMA_NUM_PARALLEL defaults to 1, so one request runs while the rest wait, and the queue is what makes p99 TTFT climb while total output stays flat. Raise it before you measure anything.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=8"

Restart with sudo systemctl restart ollama, then confirm the model still fits. Each parallel slot gets its own share of the context window, so Ollama's documentation notes that a 2K context with 4 parallel requests allocates 8K. Raise the slot count far enough and the model spills out of VRAM. Check ollama ps: a PROCESSOR column reading something like 48%/52% CPU/GPU means part of the model is on the CPU, and throughput will now fall as you add concurrency instead of rising. Past the parallel slots, requests queue up to OLLAMA_MAX_QUEUE, 512 by default, after which the server answers 503.

vLLM uses continuous batching, so it admits new requests into the running batch as slots free up, and its curve keeps climbing until KV cache runs out. Ollama optimises for one model, one machine, low setup cost. The two engines therefore give different answers to the same sweep, which is the real subject of Ollama and vLLM compared as serving engines. Record which engine and which version produced every number.

Five ways to measure the wrong thing

• The client is far away. Benchmarking from your laptop over the internet adds your round trip to every TTFT, so you measure your home connection. Run the client in the same region as the server.
• The model was cold. The first request pays weight loading, and on vLLM it can also pay graph capture. Send a warmup batch and throw the result away.
• Prefix caching answered for you. vLLM enables automatic prefix caching by default, so sending the same prompt over and over measures the cache instead of prefill, and TTFT collapses to a fraction of the real value. --dataset-name random avoids this because every prompt differs. To be sure, start the server with --no-enable-prefix-caching.
• The outputs were short. With 32-token answers, TTFT dominates each request and your tokens per second is really describing prefill. Use --ignore-eos with a realistic output length.
• You reported concurrency 1. It is the friendliest number on the sheet and it has no bearing on cost.

Turn your measured number into a decision

Take the saturated output throughput from your sweep, not the single-stream rate, and compare it with per-token pricing. The break-even is one division:

break_even_tok_s = (price_per_hour / price_per_million_output_tokens) * 1000000 / 3600

Work it through with DigitalOcean's July 2026 prices. Their H200 dedicated inference endpoint was $4.47 per hour, and the serverless equivalent was $0.65 per million tokens. So 4.47 divided by 0.65 is 6.88 million tokens per hour, and dividing by 3,600 seconds gives about 1,910 output tokens per second. The prices are theirs. The division is ours.

The word that decides this is sustained. Hitting 1,910 tokens per second at saturation for two hours a day is not 1,910 tokens per second sustained, because you pay for the other twenty-two hours as well. DigitalOcean's own crossover for the cheaper $3.44 per hour GPU Droplet lands at 72.2 percent sustained average utilisation, and below that the per-token price wins. Idle GPU hours, not slow tokens, are what usually sink self-hosting.

So your decision has two inputs. The sweep gives you the ceiling. Your traffic pattern gives you the fraction of that ceiling you actually collect. Multiply them, then take the result to the GPU VPS against per-token API break-even and read off the answer for your volume.

FAQ

What is a good tokens per second for a self-hosted LLM?

There are two answers, because the metric does two jobs. For one person reading the output, anything above roughly 20 output tokens per second per stream is already faster than reading speed, so more does not help. For cost, the number that matters is saturated total output throughput, and good means whatever clears your break-even. Against $0.65 per million tokens on a box costing $4.47 per hour, that floor sits near 1,910 output tokens per second sustained at July 2026 prices. One stream on a large model never reaches it, which is the reason batching exists.

Why does my Ollama throughput stay flat when I add concurrent requests?

OLLAMA_NUM_PARALLEL defaults to 1, so the server runs one request at a time per model and queues the rest, up to OLLAMA_MAX_QUEUE (512 by default) before returning 503. Total output stays flat while p99 TTFT climbs, which is the signature of a queue rather than a busy GPU. Set the variable in a systemd drop-in and restart, then check ollama ps, because each parallel slot multiplies the allocated context and can push part of the model onto the CPU.

Should I measure time to first token or tokens per second?

Both, since they move in opposite directions as load rises. TTFT is what a user feels, and saturated output throughput is what your invoice reflects. Record p50 and p99 TTFT at every concurrency step, then choose the highest concurrency where p99 TTFT is still acceptable to you. Report the throughput at that point as your capacity, not the maximum from the top of the curve.

Does a higher tokens per second number always mean a lower cost per token?

No. Cost per token is the hourly price divided by the tokens the box actually produced in that hour, so a fast server that idles most of the day still has a high cost per token. Utilisation decides it, not peak speed. Watch the units too: a quoted total token throughput counts input tokens, so at a 1:1 input to output ratio it is close to double the output rate you are billed on.