How to self-host LiteLLM as LLM gateway
Run one OpenAI-compatible endpoint for all your providers with LiteLLM on a VPS, virtual keys, per-key budgets, fallbacks, and pinned images.
Wetin self-hosted LLM gateway dey do
LiteLLM na open source LLM gateway wey you host by yourself: one HTTP endpoint wey all your applications dey call, then e forward each request go whichever provider wey suppose answer am. LLM mean large language model. The gateway dey speak OpenAI chat completions API (application programming interface), so any client library wey already dey talk to OpenAI go work with am after two changes: the base URL and the key.
Na that one layer of indirection be the main point. Your applications no longer hold provider credentials. To change model na one line for config file on the server, instead of code change for five services. And because every call dey pass through one process, you get place to set budget and keep record of wetin dem spend.
This na wetin you go get once e dey run:
- One endpoint. Applications go target
https://gateway.example.com/v1and ask for model name wey you invent, likebulkorstrong. - Virtual keys. Each application go get e own key, with e own model allowlist and e own spend ceiling. You fit revoke one without touching the others.
- Fallbacks. If call fail or prompt too big, system go retry am against different model automatically.
- A logged record. Every request go write one row wey carry the cost, so question like "which app spend that" go get answer.
Why you go run the gateway by yourself
Managed router get the same shape, but another person process dey for middle of every request. If you run am by yourself, your provider keys and prompt text go remain for machine wey you control. The cost dey real: now na you go operate the component wey every application depend on. The last section of this guide dey about this cost, because na the part wey most write-ups dey leave out.
Wetin you need
- One VPS (virtual private server) wey dey run Ubuntu 24.04, with Docker and Compose plugin installed.
- One domain name wey point to am, if machines outside the box go reach the gateway through TLS (transport layer security).
- At least one provider API key.
The gateway no dey run inference. E dey forward requests and stream answers back, so the CPU load dey follow request volume, no be model size. One 1 vCPU box fit carry some internal applications without problem. Na database dey grow, because the gateway dey write one spend row for every request.
Write config.yaml first
The config file dey decide which models client fit ask for. Four top-level sections matter: model_list, litellm_settings, router_settings and general_settings.
model_list:
- model_name: bulk
litellm_params:
model: anthropic/claude-haiku-4-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: strong
litellm_params:
model: anthropic/claude-sonnet-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: strong
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
num_retries: 2
request_timeout: 120
json_logs: true
set_verbose: false
router_settings:
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"bulk": ["strong"]}]
context_window_fallbacks: [{"bulk": ["strong"]}]
general_settings:
background_health_checks: true
health_check_interval: 300model_name na the name wey your clients send. litellm_params.model na the real model, wey you write as provider/model. Name your models after the work dem do, no be after the vendor. Application wey dey ask for bulk go continue to work when you decide next month say bulk suppose be another model.
api_key: os.environ/ANTHROPIC_API_KEY dey tell LiteLLM make e read that variable when e dey run. The literal key no dey appear for the file. This one matter because na config.yaml be the file wey you commit.
Two entries get the same name strong, and na intentional. When more than one deployment carry the same model_name, the router treats dem as interchangeable and spreads requests for that name across dem. Na so strong fit survive when one provider get bad hour: once one deployment enter cooldown (below), new requests for strong go the other one.
num_retries: 2 retries retryable error up to two times. Those retries fit choose another available deployment inside the same model group: LiteLLM architecture docs, wey dem check on 2026-10-29, describe retries inside the group and cross-group fallbacks as separate steps. So retry for strong fit land for the other provider without enabling weighted failover. Fallback go only run after those retries finish.
allowed_fails and cooldown_time na router options, so dem dey under router_settings. The proxy builds its router from that section, and for v1.95.0, cooldown_time wey you write under litellm_settings no dey reach the router. Because of that, cooldowns quietly remain for the 5 second default. cooldown_time: 30 sets how long deployment go stay out of rotation after e cool down. The configuration reference describe allowed_fails as the failure count wey deployment must pass inside cooldown_time before e cool down. For the v1.95.0 source, allowed_fails: 3 na the default, and with that default the router dey use the built-in triggers; failure count only take effect when you set another number. Those triggers include immediate cooldown for 429 response when a group get more than one deployment, more than 50% failures for the current minute after at least 5 requests for that kind group, and errors wey router no go retry, like 401 from bad key. Another trigger dey cool down one deployment when every request for the current minute fail and traffic reach SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD, wey default to 1000 requests. So for strong, wey get two deployments, provider wey dey return 500s in one burst go stop being tried for every request, but one stray 500 every few minutes no go trigger the cooldown. bulk get only one deployment, so the failure-rate trigger no apply to am: burst of 500s wey stay below 1000 requests per minute leave am for rotation, and na num_retries followed by fallback to strong dey protect its callers.
fallbacks and context_window_fallbacks get different triggers, and the second one na the useful one wey people dey skip.
fallbacksgo run when the primary call fail.context_window_fallbacksgo run when provider reject the request because e pass that model context window, so oversized prompt go the model wey get enough space instead of returning error to the caller.
There is also content_policy_fallbacks, for provider wey refuse request because of content policy. Set am only if you get sensible place to send those calls.
Deploy LiteLLM on a VPS with Docker Compose
Make directory wey go hold three files: config.yaml, docker-compose.yml and .env. The upstream quickstart dey pull latest tag. Pin release tag instead, so upgrade go happen only when you edit the file, and rollback go be one line. Tag by itself no guarantee say contents go remain identical, because image tags fit change and publisher fit point one tag to new image. If docker compose up -d next month suppose give you exactly the same gateway wey e give you today, append image digest too, for this format: ghcr.io/berriai/litellm:v1.95.0@sha256:<digest>; docker buildx imagetools inspect ghcr.io/berriai/litellm:v1.95.0 go print am.
services:
litellm:
image: ghcr.io/berriai/litellm:v1.95.0
restart: unless-stopped
command: ["--config", "/app/config.yaml", "--num_workers", "1"]
ports:
- "127.0.0.1:4000:4000"
volumes:
- ./config.yaml:/app/config.yaml:ro
env_file: .env
depends_on:
db:
condition: service_healthy
db:
image: postgres:16
restart: unless-stopped
environment:
POSTGRES_USER: litellm
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
POSTGRES_DB: litellm
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 5s
timeout: 5s
retries: 10
volumes:
- postgres_data:/var/lib/postgresql/data
volumes:
postgres_data:Compose dey read .env twice here. First, e use am to substitute ${POSTGRES_PASSWORD} inside the compose file itself. Second, e use env_file to pass every variable enter the container.
v1.95.0 na the current release for August 2026. Check the project's releases page and pin the version wey current when you deploy. Every release publishes signature, so you fit check the image before you trust am:
cosign verify --key https://raw.githubusercontent.com/BerriAI/litellm/v1.95.0/cosign.pub ghcr.io/berriai/litellm:v1.95.0The port line na 127.0.0.1:4000:4000, wey publish the port for loopback interface only. If you write 4000:4000 instead, your gateway go reachable from the whole internet, because Docker add im own rules for the iptables FORWARD chain and dem dey evaluate before ufw rules, so ufw deny 4000 no go stop am. Na this be the most common way self-hosted gateway dey end up open: see how Docker dey publish container port straight past ufw. Traffic from outside go enter through the reverse proxy instead.
Provider keys no dey inside image
The .env file dey hold every secret. E dey pass as environment variable when runtime start, so dem no dey bake am inside image, and dem no dey commit am.
LITELLM_MASTER_KEY=sk-REPLACE_ME
LITELLM_SALT_KEY=sk-REPLACE_ME_TOO
POSTGRES_PASSWORD=REPLACE_ME_AS_WELL
DATABASE_URL=postgresql://litellm:REPLACE_ME_AS_WELL@db:5432/litellm
STORE_MODEL_IN_DB=True
LITELLM_MODE=PRODUCTION
LITELLM_LOG=ERROR
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...Generate the two LiteLLM keys with real randomness, then lock the file down:
printf 'sk-%s\n' "$(openssl rand -hex 32)"
chmod 600 .envLITELLM_MASTER_KEY na the admin credential. E dey authenticate the management API, and na the password for Admin UI at /ui. No application suppose ever hold am.
LITELLM_SALT_KEY dey encrypt provider credentials wey dey stored for database. Set am once and leave am. If you change am later, credentials wey already dey stored no go decrypt. The gateway go start normally, then every call to those providers go fail for authentication.
STORE_MODEL_IN_DB=True let you add and edit models from Admin UI without touching config.yaml. E convenient, but e split your source of truth into two places. Decide which one dey authoritative and write the decision down beside the config.
The same reason wey make keys no suppose dey inside config file na the same reason wey make dem no suppose dey inside tools wey you hand to agent. Keep provider secrets out of AI agents explain that pattern, while env files and secrets for Docker Compose explain the mechanics.
Bring am up and monitor the first boot:
docker compose up -d
docker compose logs -f litellmCheck say e dey really work
Two unauthenticated probes and one authenticated probe dey available, and different reasons fit make dem fail.
curl -s http://127.0.0.1:4000/health/liveliness
curl -s http://127.0.0.1:4000/health/readiness/health/liveliness no need auth and e dey answer "I'm alive!" as long as the process dey run. /health/readiness too no need auth. E dey return JSON object wey get "status": "healthy" and db field, or 503 when database no dey reachable. Point your monitoring to readiness, because liveliness fit still stay green for gateway wey no fit look up even one virtual key.
The authenticated check na the one wey dey talk to providers:
curl -s http://127.0.0.1:4000/health \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"E dey answer with healthy_endpoints and unhealthy_endpoints arrays. If model dey for unhealthy_endpoints with authentication error, e mean say provider key for .env wrong or e no dey there. Na this failure you wan find now. Because background_health_checks: true set, proxy dey run these probes by itself every health_check_interval seconds, and /health dey return the last result. So, polling am no dey send test request to your providers every time.
Virtual keys and per-key budgets
Every application dey get im own key, wey dem mint against the master key.
curl -s http://127.0.0.1:4000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d '{
"key_alias": "nightly-summariser",
"models": ["bulk"],
"max_budget": 5,
"budget_duration": "30d",
"rpm_limit": 60,
"tpm_limit": 200000
}'The response get one key field wey start with sk-. Na that string the application dey receive, and na only that thing the application ever gets.
modelsna allowlist of wetin this key fit request. The key above fit ask forbulkonly.max_budget: 5withbudget_duration: "30d"na five US dollars per calendar month. LiteLLM dey reset30dbudget on the 1st day of every month for midnight, for UTC unless you settimezoneunderlitellm_settings. E no dey use rolling 30 days. So after the key spend five dollars, e go stop working until the next 1st.rpm_limitandtpm_limitdey limit requests per minute and tokens per minute for this key alone.key_aliasna wetin you go recognise for the spend log six weeks later. Always set am.
When the budget don finish, the call go fail with error of type budget_exceeded and message wey look like this:
Budget has been exceeded! Key=<key> Current cost: <spend>, Max budget: <max_budget>The status code depend on the release. For the pinned v1.95.0, e na HTTP 429. From v1.104.0, the default na HTTP 422, and if you set budget_exceeded_status_code: 429 under litellm_settings, e go restore the earlier code. Genuine rpm_limit and tpm_limit hits still return 429 for both releases. Check which one you dey get after every upgrade.
The status code na wetin dey make this confusing, because neither code talk "budget". A client library fit report 429 as rate limit and retry am with backoff against budget wey no go return until the reset. E fit report 422 as request wey e no fit process. Log the response body together with the status code, otherwise budget exhaustion go look like another problem every time.
Inspect and adjust keys through the same management API:
curl -s "http://127.0.0.1:4000/key/info?key=sk-..." \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
curl -s -X POST http://127.0.0.1:4000/key/update \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d '{"key": "sk-...", "max_budget": 25}'Budget wey gateway enforce still dey apply even when na the agent itself cause the problem. Na why e be the backbone of cost control for AI agents on a VPS.
Send bulk work go cheap model
Point client go the gateway. Base URL, key, model name:
curl -s http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer sk-<the virtual key>" \
-H 'Content-Type: application/json' \
-d '{
"model": "bulk",
"messages": [{"role": "user", "content": "Say hello in five words."}]
}'Any OpenAI client library dey work the same way, though option names dey differ by language. For Python client, set base_url to https://gateway.example.com/v1 and api_key to the virtual key. The JavaScript and TypeScript client calls the same two options baseURL and apiKey.
The routing policy from config.yaml now dey apply without caller knowing. Request for bulk go the cheap model. If that call fail after its retries, the request go retry against strong. If prompt too long for bulk, context_window_fallbacks go send am to strong instead of returning error. Bulk work like classification pass or summarising backlog go run cheap by default, and na only the hard requests go cost more.
Na here gateway also show why e useful for agents wey dey use tools. An MCP (model context protocol) server for the same VPS and the agent wey dey drive am fit both point to one endpoint, so the model behind dem fit change without redeploying either one.
How you go know say fallback happen?
Na this failure mode dey cost money, because nothing go look broken. Successful fallback go return HTTP 200 with normal response body. Your cheap model fit dey down for one day, while the expensive one quietly dey serve every call, and the first evidence fit be the invoice.
The evidence dey inside response headers. Request dem:
curl -s -D - -o /dev/null http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer sk-<the virtual key>" \
-H 'Content-Type: application/json' \
-d '{"model":"bulk","messages":[{"role":"user","content":"ping"}]}' \
| grep -i '^x-litellm'x-litellm-model-groupna the model name wey the request route under, likebulk.x-litellm-model-idna the deployment ID wey answer the request, and you fit find am with/v1/model/info?litellm_model_id=.... When that ID no be one of yourbulkdeployments, fallback don happen.x-litellm-attempted-fallbacksandx-litellm-attempted-retriesdey count dem. For healthy call, none of dem suppose show count above 0.x-litellm-response-costna the cost of that one call for US dollars.x-litellm-call-idna the identifier wey you go use find the same call for your logs.
Record x-litellm-attempted-fallbacks for every request and set alert whenever e pass 0. That one number na the difference between routing policy wey dey work and routing policy wey don silently change to "always use the expensive model".
The complete version of this na tracing, and e need im own setup: self-hosted Langfuse for tracing agent calls. LiteLLM ships the callback, so wiring am na two lines plus credentials.
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://langfuse.example.comFor Langfuse v3 and later, LiteLLM Langfuse page now recommend im langfuse_otel preset; the langfuse callback above still dey documented. Set failure_callback together with success_callback. If you skip am, the only traces wey you go keep na the ones where nothing go wrong. Separately from all this, LiteLLM dey write one spend row per request into Postgres, and Admin UI for /ui dey read that table. E dey grow as traffic increase, so monitor am if disk small.
Put the gateway behind a reverse proxy
Make sure nothing outside the box fit reach port 4000. Terminate TLS for nginx or Caddy, then forward traffic go the loopback address.
location / {
proxy_pass http://127.0.0.1:4000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_read_timeout 600s;
}Two of those lines na the ones people dey leave out. proxy_buffering off matter because streaming completion na series of server-sent events. When buffering dey on, nginx go collect chunks for buffers before e forward dem, and this go delay delivery to the client. proxy_read_timeout 600s matter because nginx default of 60 seconds na the longest time e go wait between two reads from the gateway, not limit for the whole response. Non-streamed completion no send anything until generation finish. So, if generation long pass 60 seconds without any byte arriving, client go receive 504, while error log go record upstream timed out (110: Connection timed out) while reading response header from upstream. Streamed completion wey dey continue send chunks fit run much longer than 60 seconds. E go only trigger the timeout if the model stay quiet for that long.
For the certificate, Certbot with Let's Encrypt for nginx na the short path. If the box already dey serve several containers, Traefik in front of multiple Compose apps go handle routing and certificates for one place.
The gateway don turn to one point wey fit fail
Make you honest about wetin you don build. Every application wey you own now depend on one container for one VPS. When e dey down, nothing fit call any model, including providers wey dey perfectly healthy. Four things follow from this.
- Bad config fit bring everything down at once.
restart: unless-stoppeddey restart crash, and e go keep restarting container wey no fit parse config.yaml. Readdocker compose logs litellmafter every config change, and make config changes when you get time to monitor dem. - Postgres dey inside request path when cache miss happen. Virtual key lookup dey use am when key no dey cached; spend recording dey happen asynchronously after response.
/health/readinesswey return 503 na warning say gateway dey run but e no fit do either one. - Use the cores first, then add instances. The project production guidance of one worker per instance (
--num_workers 1) na for Kubernetes or another pod autoscaler wey dey add instances for you. For one VM or bare container like this one, e mean the opposite: set worker count to the machine vCPU count. For the 1 vCPU machine above, na still--num_workers 1; for bigger machine, increase am, and remember say every worker get im own memory. A second gateway behind load balancer removes the single container from the setup. E no remove the database. - Back up wetin you no fit regenerate. Na
config.yamland.env, together withpg_dumpof the database. If you loseLITELLM_SALT_KEY, the encrypted provider credentials inside that dump become useless. So the env file and the dump suppose dey inside the same backup job: restic backups to storage wey dey outside the machine.
To upgrade, edit image tag and run docker compose up -d. LiteLLM dey run prisma migrate deploy on startup by default, so the new container go migrate database schema for im first boot. Take the dump before you change the tag, because putting the old image back no go undo migration wey don already run.
Read release notes for every version wey you skip, no be only the version wey you land on. Moving from v1.95.0 to v1.104.0 or later crosses several breaking changes: the budget status code above, a proxy wey refuse to start when master key unset, empty, or publicly known, and a proxy wey exits when database setup fail during startup. Under restart: unless-stopped, this one go show as restart loop. If you dey come from v1.102.x or earlier, the v1.104.0 notes also ask you to build two LiteLLM_SpendLogs indexes by yourself.
FAQ
LiteLLM dey add latency wey noticeable to every call?
The project publish 8 ms for the 95th percentile at 1000 requests per second, as e talk for the README for November 2026. Treat am as vendor figure. The thing wey really change your latency na the network distance between your applications and the gateway, because you don add one round trip to every call. Run the gateway for the same region with the applications wey dey call am, then measure your own overhead with the x-litellm-overhead-duration-ms header for real response.
Why streaming stop to work after I put nginx for front?
Na because nginx dey buffer upstream responses by default, and streaming completion na series of server-sent events. When proxy_buffering dey on, nginx dey collect chunks for buffers before e forward dem, so delivery to client dey delay. Set proxy_buffering off; for the location block. Increase proxy_read_timeout for the same block too. By default, nginx go give up after 60 seconds without data from the gateway. Long non-streamed generation fit easily reach this limit, and client go receive 504.
Wetin go happen when virtual key finish e budget?
The call go fail with error of type budget_exceeded and message wey look like Budget has been exceeded! Key=<key> Current cost: <spend>, Max budget: <max_budget>. For the pinned v1.95.0, the status na HTTP 429; from v1.104.0, the default na HTTP 422 instead. No one of the codes talk budget. Na this be the trap: client library fit report 429 as rate limit and retry am, so people go start investigate throttling instead of reading the message. Log the response body together with the status code. Confirm the key's real position with /key/info?key=sk-... against the master key, and increase the ceiling with /key/update if the budget too low.
Gateway fit route go local model as well as hosted ones?
Yes, and e na be one more entry for model_list. Use the ollama_chat/ prefix with an api_base, for example model: ollama_chat/llama3.1 together with api_base: http://ollama:11434. For Compose's default networking, localhost inside container means that container, so use the Compose service name or the host's address for the Docker network, never 127.0.0.1. Host networking, where container dey share the host's network stack, na the exception. Setting up the local model na separate job: see how to self-host an LLM with Ollama for VPS.