Prefill vs decode: why token one lags
Prefill is compute bound and owns time to first token. Decode is memory bandwidth bound and owns tokens per second. Measure them separately on your own box.
#vllm
Filtering by topic #vllm · clear
Why your self-hosted LLM stalls at 5 users
One user was fine, five crawl. How batching, KV cache limits, prefill and queue depth decide how many people your LLM server can serve at once.
Measure tokens per second on a local LLM
A rented GPU only beats per-token billing above a throughput floor. Measure tokens per second properly with a concurrency sweep, then decide.
GPU VPS vs API tokens: the break-even point
Renting a GPU by the hour beats per-token API billing only above a certain volume. Here is the formula and the monthly token count where it flips.
Ollama vs vLLM: which LLM server to run
Ollama is a convenience layer for one user, on CPU if needed. vLLM is a throughput engine for GPUs. Pick by workload, with the real commands for both.