Self-host LiteLLM as your LLM gateway

What a self-hosted LLM gateway does

LiteLLM is an open source LLM gateway you host yourself: one HTTP endpoint that all of your applications call, which then forwards each request to whichever provider should answer it. LLM means large language model. The gateway speaks the OpenAI chat completions API (application programming interface), so any client library that already talks to OpenAI works against it after two changes: the base URL and the key.

That one layer of indirection is the point. Your applications stop holding provider credentials. Swapping a model becomes one line in a config file on the server instead of a code change in five services. And because every call passes through one process, you have somewhere to put a budget and somewhere to keep a record of what was spent.

Here is what you have once it is running:

• One endpoint. Applications target https://gateway.example.com/v1 and ask for a model name you invented, such as bulk or strong.
• Virtual keys. Each application gets its own key with its own model allowlist and its own spend ceiling. You revoke one without touching the others.
• Fallbacks. A failed call or an oversized prompt is retried against a different model automatically.
• A logged record. Every request writes a row carrying its cost, so "which app spent that" has an answer.

Why run the gateway yourself

A managed router is the same shape with someone else's process sitting in the middle of every request. Running it yourself keeps your provider keys and your prompt text on a box you control. The cost is real: you now operate the component that every application depends on. The last section of this guide is about that cost, because it is the part most write-ups leave out.

What you need

• A VPS (virtual private server) running Ubuntu 24.04, with Docker and the Compose plugin installed.
• A domain name pointing at it, if machines outside the box will reach the gateway over TLS (transport layer security).
• At least one provider API key.

The gateway runs no inference. It forwards requests and streams answers back, so its CPU load tracks request volume rather than model size. A 1 vCPU box carries a handful of internal applications without complaint. What grows is the database, because the gateway writes a spend row per request.

Write config.yaml first

The config file decides which models a client may ask for. Four top level sections matter: model_list, litellm_settings, router_settings and general_settings.

model_list:
  - model_name: bulk
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: strong
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: strong
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  num_retries: 2
  request_timeout: 120
  allowed_fails: 3
  cooldown_time: 30
  json_logs: true
  set_verbose: false

router_settings:
  fallbacks: [{"bulk": ["strong"]}]
  context_window_fallbacks: [{"bulk": ["strong"]}]

general_settings:
  background_health_checks: true
  health_check_interval: 300

model_name is the name your clients send. litellm_params.model is the real model, written as provider/model. Name your models after the job rather than after the vendor. An application that asks for bulk keeps working when you decide next month that bulk should be a different model.

api_key: os.environ/ANTHROPIC_API_KEY tells LiteLLM to read that variable at run time. The literal key never appears in the file, which matters because config.yaml is the file you commit.

Two entries share the name strong, on purpose. When more than one deployment carries the same model_name, the router treats them as interchangeable and tries the other one when the first fails. That is how strong survives one provider having a bad hour.

num_retries: 2 retries the same deployment on a retryable error. A fallback only fires after those retries are used up. allowed_fails: 3 with cooldown_time: 30 pulls a deployment out of rotation for 30 seconds once it has failed 3 times, so a provider returning 500s stops being tried on every single request.

fallbacks and context_window_fallbacks have different triggers, and the second one is the useful one people skip.

• fallbacks fires when the primary call fails.
• context_window_fallbacks fires when the provider rejects the request for being longer than that model's context window, so an oversized prompt goes to a model with room for it instead of returning an error to the caller.

There is also content_policy_fallbacks, for a provider refusing on content policy grounds. Set it only if you have somewhere sensible to send those calls.

Deploy LiteLLM on a VPS with Docker Compose

Make a directory holding three files: config.yaml, docker-compose.yml and .env. The upstream quickstart pulls the latest tag. Pin a release tag instead, so docker compose up -d next month gives you the same gateway it gave you today, and so a rollback is one line.

services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.95.0
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--num_workers", "1"]
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file: .env
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Compose reads .env twice here. Once to substitute ${POSTGRES_PASSWORD} inside the compose file itself, and once through env_file to pass every variable into the container.

v1.95.0 was the current release in August 2026. Check the project's releases page and pin whatever is current when you deploy. Each release publishes a signature, so you can check the image before you trust it:

cosign verify --key https://raw.githubusercontent.com/BerriAI/litellm/v1.95.0/cosign.pub ghcr.io/berriai/litellm:v1.95.0

The port line is 127.0.0.1:4000:4000, which publishes the port on the loopback interface only. Write 4000:4000 instead and your gateway is reachable from the whole internet, because Docker adds its own rules in the iptables FORWARD chain and those are evaluated before ufw's, so ufw deny 4000 does not stop it. This is the most common way a self-hosted gateway ends up open: see how Docker publishes a container port straight past ufw. Traffic from outside arrives through the reverse proxy instead.

Keep provider keys out of the image

The .env file holds every secret. It is passed in as environment at run time, so it is never baked into the image, and it is never committed.

LITELLM_MASTER_KEY=sk-REPLACE_ME
LITELLM_SALT_KEY=sk-REPLACE_ME_TOO
POSTGRES_PASSWORD=REPLACE_ME_AS_WELL
DATABASE_URL=postgresql://litellm:REPLACE_ME_AS_WELL@db:5432/litellm
STORE_MODEL_IN_DB=True
LITELLM_MODE=PRODUCTION
LITELLM_LOG=ERROR
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...

Generate the two LiteLLM keys with real randomness, then lock the file down:

printf 'sk-%s\n' "$(openssl rand -hex 32)"
chmod 600 .env

LITELLM_MASTER_KEY is the admin credential. It authenticates the management API and it is the password for the Admin UI at /ui. No application should ever hold it.

LITELLM_SALT_KEY encrypts provider credentials stored in the database. Set it once and leave it. Change it later and the credentials already stored cannot be decrypted, so the gateway starts up normally and then every call to those providers fails on authentication.

STORE_MODEL_IN_DB=True lets you add and edit models from the Admin UI without touching config.yaml. That is convenient, and it splits your source of truth in two. Decide which one is authoritative and write the decision down next to the config.

The reasoning that keeps keys out of the config file is the same reasoning that keeps them out of the tools you hand to an agent. Keeping provider secrets out of AI agents covers that pattern, and env files and secrets in Docker Compose covers the mechanics.

Bring it up and watch the first boot:

docker compose up -d
docker compose logs -f litellm

Check that it is actually working

There are two unauthenticated probes and one authenticated one, and they fail for different reasons.

curl -s http://127.0.0.1:4000/health/liveliness
curl -s http://127.0.0.1:4000/health/readiness

/health/liveliness needs no auth and answers "I'm alive!" while the process is running. /health/readiness also needs no auth. It returns a JSON object with "status": "healthy" and a db field, or a 503 when the database is unreachable. Point your monitoring at readiness, because liveliness stays green on a gateway that cannot look up a single virtual key.

The authenticated check is the one that talks to providers:

curl -s http://127.0.0.1:4000/health \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

It answers with healthy_endpoints and unhealthy_endpoints arrays. A model sitting in unhealthy_endpoints with an authentication error means the provider key in .env is wrong or missing, which is the failure you want to find now. Because background_health_checks: true is set, the proxy runs these probes every health_check_interval seconds on its own and /health returns the last result, so polling it does not send a test request to your providers each time.

Virtual keys and per-key budgets

Every application gets its own key, minted against the master key.

curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "key_alias": "nightly-summariser",
    "models": ["bulk"],
    "max_budget": 5,
    "budget_duration": "30d",
    "rpm_limit": 60,
    "tpm_limit": 200000
  }'

The response carries a key field starting with sk-. That string is what the application gets, and it is the only thing the application ever gets.

• models is an allowlist of what this key may request. The key above can ask for bulk and nothing else.
• max_budget: 5 with budget_duration: "30d" is five US dollars per rolling 30 days, then the key stops working.
• rpm_limit and tpm_limit cap requests per minute and tokens per minute for this key alone.
• key_alias is what you will recognise in the spend log six weeks later. Always set it.

When the budget is gone, the call fails with HTTP 401 and a body of this shape:

ExceededBudget: Current spend for token: 7.2e-05; Max Budget for Token: 2e-07

The status code is what makes this confusing. A client library reports 401 as an authentication problem, so the developer reading the stack trace starts checking whether the key is valid. Log the response body next to the status code, or budget exhaustion looks like a broken credential every single time.

Inspect and adjust keys through the same management API:

curl -s "http://127.0.0.1:4000/key/info?key=sk-..." \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

curl -s -X POST http://127.0.0.1:4000/key/update \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"key": "sk-...", "max_budget": 25}'

A budget enforced at the gateway holds even when the thing that has gone wrong is the agent itself, which is why it is the backbone of cost control for AI agents on a VPS.

Send bulk work to a cheap model

Point a client at the gateway. Base URL, key, model name:

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<the virtual key>" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bulk",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

Any OpenAI client library behaves the same way: set base_url to https://gateway.example.com/v1 and api_key to the virtual key.

The routing policy from config.yaml now applies without the caller knowing. A request for bulk goes to the cheap model. If that call fails after its retries, the request is retried against strong. If the prompt is too long for bulk, context_window_fallbacks sends it to strong instead of returning an error. Bulk work such as a classification pass or a summarising backlog runs cheap by default, and only the hard requests cost more.

This is also where a gateway earns its place with tool-using agents. An MCP (model context protocol) server on the same VPS and the agent driving it can both point at one endpoint, so the model behind them changes without redeploying either.

How do you know a fallback happened?

This is the failure mode that costs money, because nothing looks broken. A successful fallback returns HTTP 200 with an ordinary response body. Your cheap model can be down for a day, every call quietly served by the expensive one, and the first evidence is the invoice.

The evidence does exist, in response headers. Ask for them:

curl -s -D - -o /dev/null http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<the virtual key>" \
  -H 'Content-Type: application/json' \
  -d '{"model":"bulk","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^x-litellm'

• x-litellm-model-group is what the client asked for. x-litellm-model-id is the deployment that answered. When those two disagree, a fallback happened.
• x-litellm-attempted-fallbacks and x-litellm-attempted-retries count them. On a healthy call both are 0.
• x-litellm-response-cost is the cost of that one call in US dollars.
• x-litellm-call-id is the identifier you use to find the same call in your logs.

Record x-litellm-attempted-fallbacks on every request and alert when it stops being 0. That one number is the difference between a routing policy that works and a routing policy that has silently become "always use the expensive model".

The full version of this is tracing, and it deserves its own setup: self-hosted Langfuse for tracing agent calls. LiteLLM ships the callback, so wiring it is two lines plus credentials.

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://langfuse.example.com

Set failure_callback as well as success_callback. Skip it and the only traces you keep are the ones where nothing went wrong. Separately from any of this, LiteLLM writes a spend row per request into Postgres and the Admin UI at /ui reads that table. It grows with traffic, so watch it on a small disk.

Put the gateway behind a reverse proxy

Nothing outside the box should reach port 4000. Terminate TLS in nginx or Caddy and forward to the loopback address.

location / {
    proxy_pass http://127.0.0.1:4000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_read_timeout 600s;
}

Two of those lines are the ones people leave out. proxy_buffering off matters because a streaming completion is a series of server-sent events, and with buffering on nginx holds the chunks until the response ends, so the client sits in silence and then receives everything at once. proxy_read_timeout 600s matters because a long generation runs past nginx's 60 second default, and when it does the client gets a 504 while the error log records upstream timed out (110: Connection timed out) while reading response header from upstream.

For the certificate, Certbot with Let's Encrypt on nginx is the short path. If the box already serves several containers, Traefik in front of multiple Compose apps handles routing and certificates in one place.

The gateway is now a single point of failure

Be honest about what you have built. Every application you own now depends on one container on one VPS. While it is down, nothing can call any model, including the providers that are perfectly healthy. Four things follow from that.

• A bad config takes everything down at once. restart: unless-stopped restarts a crash, and it restarts a container that cannot parse config.yaml, over and over. Read docker compose logs litellm after every config change, and make config changes when you have time to watch them.
• Postgres sits in the request path. Virtual key lookup and spend recording both use it. /health/readiness returning 503 is your warning that the gateway is running but can do neither.
• Scale by adding instances, not by making one bigger. The project's own guidance is one worker per instance (--num_workers 1) with several instances sharing one database. Two small gateways behind a load balancer remove the single container from the story. They do not remove the database.
• Back up what you cannot regenerate. That is config.yaml and .env, together with a pg_dump of the database. Losing LITELLM_SALT_KEY makes the encrypted provider credentials inside that dump useless, so the env file and the dump belong in the same backup job: restic backups to off-box storage.

Upgrading is editing the image tag and running docker compose up -d. LiteLLM runs prisma migrate deploy on startup by default, so the new container migrates the database schema on its first boot. Take the dump before you change the tag, because putting the old image back does not undo a migration that has already run.

FAQ

Does LiteLLM add noticeable latency to every call?

The project publishes 8 ms at the 95th percentile at 1000 requests per second, as stated in its README in August 2026. Treat that as a vendor figure. The number that actually moves your latency is the network distance between your applications and the gateway, because you have added one round trip to every call. Run the gateway in the same region as the applications that call it, then measure your own overhead with the x-litellm-overhead-duration-ms header on a real response.

Why did streaming stop working after I put nginx in front?

Because nginx buffers upstream responses by default and a streaming completion is a series of server-sent events. With proxy_buffering on, nginx collects the chunks and releases them only when the response finishes, so the client waits in silence and then receives the whole answer at once. Set proxy_buffering off; in the location block. Raise proxy_read_timeout in the same block, because a long generation otherwise exceeds nginx's 60 second default and the client gets a 504.

What happens when a virtual key runs out of budget?

The call fails with HTTP 401 and a body of the form ExceededBudget: Current spend for token: 7.2e-05; Max Budget for Token: 2e-07. The 401 is the trap: a client library reports it as an authentication failure, so people start checking whether the key is valid instead of reading the message. Log the response body alongside the status code. Confirm the key's real position with /key/info?key=sk-... against the master key, and raise the ceiling with /key/update if the budget was set too low.

Can the gateway route to a local model as well as hosted ones?

Yes, and it is one more entry in model_list. Use the ollama_chat/ prefix with an api_base, for example model: ollama_chat/llama3.1 alongside api_base: http://ollama:11434. From inside a container, localhost means that container, so use the Compose service name or the host's address on the Docker network, never 127.0.0.1. Standing the local model up is a separate job: see self-hosting an LLM with Ollama on a VPS.