How to self-host LiteLLM as LLM gateway

Wetin self-hosted LLM gateway dey do

LiteLLM na open source LLM gateway wey you host by yourself: one HTTP endpoint wey all your applications dey call, then e forward each request go whichever provider wey suppose answer am. LLM mean large language model. The gateway dey speak OpenAI chat completions API (application programming interface), so any client library wey already dey talk to OpenAI go work with am after two changes: the base URL and the key.

Na that one layer of indirection be the main point. Your applications no longer hold provider credentials. To change model na one line for config file on the server, instead of code change for five services. And because every call dey pass through one process, you get place to set budget and keep record of wetin dem spend.

This na wetin you go get once e dey run:

• One endpoint. Applications go target https://gateway.example.com/v1 and ask for model name wey you invent, like bulk or strong.
• Virtual keys. Each application go get e own key, with e own model allowlist and e own spend ceiling. You fit revoke one without touching the others.
• Fallbacks. If call fail or prompt too big, system go retry am against different model automatically.
• A logged record. Every request go write one row wey carry the cost, so question like "which app spend that" go get answer.

Why you go run the gateway by yourself

Managed router get the same shape, but another person process dey for middle of every request. If you run am by yourself, your provider keys and prompt text go remain for machine wey you control. The cost dey real: now na you go operate the component wey every application depend on. The last section of this guide dey about this cost, because na the part wey most write-ups dey leave out.

Wetin you need

• One VPS (virtual private server) wey dey run Ubuntu 24.04, with Docker and Compose plugin installed.
• One domain name wey point to am, if machines outside the box go reach the gateway through TLS (transport layer security).
• At least one provider API key.

The gateway no dey run inference. E dey forward requests and stream answers back, so the CPU load dey follow request volume, no be model size. One 1 vCPU box fit carry some internal applications without problem. Na database dey grow, because the gateway dey write one spend row for every request.

Write config.yaml first

config file go decide which models client fit ask for. Four top-level sections matter: model_list, litellm_settings, router_settings and general_settings.

model_list:
  - model_name: bulk
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: strong
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: strong
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  num_retries: 2
  request_timeout: 120
  allowed_fails: 3
  cooldown_time: 30
  json_logs: true
  set_verbose: false

router_settings:
  fallbacks: [{"bulk": ["strong"]}]
  context_window_fallbacks: [{"bulk": ["strong"]}]

general_settings:
  background_health_checks: true
  health_check_interval: 300

model_name na the name wey your clients go send. litellm_params.model na the real model, written as provider/model. Name your models after the work wey dem do, no be after the vendor. Application wey ask for bulk go continue to work when you decide next month say bulk suppose be another model.

api_key: os.environ/ANTHROPIC_API_KEY tells LiteLLM make e read that variable when e dey run. The literal key no dey appear for the file. This matter because na config.yaml be the file wey you commit.

Two entries get the same name strong, and na intentional. When more than one deployment get the same model_name, router treats dem as interchangeable and tries the other one if the first fail. Na so strong fit survive when one provider get bad period.

num_retries: 2 retries the same deployment when retryable error happen. Fallback only starts after those retries finish. allowed_fails: 3 together with cooldown_time: 30 removes a deployment from rotation for 30 seconds after e fail 3 times. This means provider wey dey return 500s no go dey tried for every request.

fallbacks and context_window_fallbacks get different triggers, and the second one na the useful one wey people dey skip.

• fallbacks fires when the primary call fail.
• context_window_fallbacks fires when provider reject the request because e pass that model context window, so oversized prompt go go model wey get enough space instead of returning error to the caller.

There is also content_policy_fallbacks, for when provider refuse request because of content policy. Set am only if you get sensible place to send those calls.

LiteLLM deploy for VPS with Docker Compose

Make one directory wey go hold three files: config.yaml, docker-compose.yml and .env. The upstream quickstart dey pull latest tag. Pin one release tag instead, so docker compose up -d next month still give you the same gateway wey e give you today, and rollback go be one line.

services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.95.0
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--num_workers", "1"]
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file: .env
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Compose dey read .env two times for here. One time na to substitute ${POSTGRES_PASSWORD} inside the compose file itself. The second time na through env_file to pass every variable enter the container.

v1.95.0 na the current release for August 2026. Check the project's releases page and pin the release wey dey current when you deploy. Every release publishes one signature, so you fit check the image before you trust am:

cosign verify --key https://raw.githubusercontent.com/BerriAI/litellm/v1.95.0/cosign.pub ghcr.io/berriai/litellm:v1.95.0

The port line na 127.0.0.1:4000:4000, and e dey publish the port for loopback interface only. If you write 4000:4000 instead, your gateway go reachable from the whole internet. This na because Docker dey add e own rules for the iptables FORWARD chain, and dem dey evaluate those rules before ufw rules, so ufw deny 4000 no go stop am. This na the most common way self-hosted gateway dey end up open: see how Docker dey publish container port straight past ufw. Traffic from outside go enter through the reverse proxy instead.

Provider keys no dey inside image

The .env file dey hold every secret. E dey pass as environment variable when runtime start, so dem no dey bake am inside image, and dem no dey commit am.

LITELLM_MASTER_KEY=sk-REPLACE_ME
LITELLM_SALT_KEY=sk-REPLACE_ME_TOO
POSTGRES_PASSWORD=REPLACE_ME_AS_WELL
DATABASE_URL=postgresql://litellm:REPLACE_ME_AS_WELL@db:5432/litellm
STORE_MODEL_IN_DB=True
LITELLM_MODE=PRODUCTION
LITELLM_LOG=ERROR
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...

Generate the two LiteLLM keys with real randomness, then lock the file down:

printf 'sk-%s\n' "$(openssl rand -hex 32)"
chmod 600 .env

LITELLM_MASTER_KEY na the admin credential. E dey authenticate the management API, and na the password for Admin UI at /ui. No application suppose ever hold am.

LITELLM_SALT_KEY dey encrypt provider credentials wey dey stored for database. Set am once and leave am. If you change am later, credentials wey already dey stored no go decrypt. The gateway go start normally, then every call to those providers go fail for authentication.

STORE_MODEL_IN_DB=True let you add and edit models from Admin UI without touching config.yaml. E convenient, but e split your source of truth into two places. Decide which one dey authoritative and write the decision down beside the config.

The same reason wey make keys no suppose dey inside config file na the same reason wey make dem no suppose dey inside tools wey you hand to agent. Keep provider secrets out of AI agents explain that pattern, while env files and secrets for Docker Compose explain the mechanics.

Bring am up and monitor the first boot:

docker compose up -d
docker compose logs -f litellm

Check say e dey really work

Two unauthenticated probes and one authenticated probe dey available, and different reasons fit make dem fail.

curl -s http://127.0.0.1:4000/health/liveliness
curl -s http://127.0.0.1:4000/health/readiness

/health/liveliness no need auth and e dey answer "I'm alive!" as long as the process dey run. /health/readiness too no need auth. E dey return JSON object wey get "status": "healthy" and db field, or 503 when database no dey reachable. Point your monitoring to readiness, because liveliness fit still stay green for gateway wey no fit look up even one virtual key.

The authenticated check na the one wey dey talk to providers:

curl -s http://127.0.0.1:4000/health \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

E dey answer with healthy_endpoints and unhealthy_endpoints arrays. If model dey for unhealthy_endpoints with authentication error, e mean say provider key for .env wrong or e no dey there. Na this failure you wan find now. Because background_health_checks: true set, proxy dey run these probes by itself every health_check_interval seconds, and /health dey return the last result. So, polling am no dey send test request to your providers every time.

Virtual keys and per-key budgets

Every application dey get im own key, wey you mint against the master key.

curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "key_alias": "nightly-summariser",
    "models": ["bulk"],
    "max_budget": 5,
    "budget_duration": "30d",
    "rpm_limit": 60,
    "tpm_limit": 200000
  }'

The response dey carry a key field wey start with sk-. Na that string the application go get, and na only that thing the application ever go get.

• models na allowlist of wetin this key fit request. The key above fit ask for bulk and nothing else.
• max_budget: 5 with budget_duration: "30d" na five US dollars per rolling 30 days, then the key go stop working.
• rpm_limit and tpm_limit dey limit requests per minute and tokens per minute for this key alone.
• key_alias na wetin you go recognise for spend log six weeks later. Always set am.

When the budget finish, the call go fail with HTTP 401 and body wey get this shape:

ExceededBudget: Current spend for token: 7.2e-05; Max Budget for Token: 2e-07

Na the status code dey make this confusing. Client library dey report 401 as authentication problem, so the developer wey dey read the stack trace go start check whether the key valid. Log the response body together with the status code, otherwise budget exhaustion go look like broken credential every time.

Inspect and adjust keys through the same management API:

curl -s "http://127.0.0.1:4000/key/info?key=sk-..." \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

curl -s -X POST http://127.0.0.1:4000/key/update \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"key": "sk-...", "max_budget": 25}'

Budget wey gateway enforce go still hold even when na the agent itself cause the problem. Na why e be the backbone of cost control for AI agents on a VPS.

Send bulk work go cheap model

Point client go the gateway. Base URL, key, model name:

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<the virtual key>" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bulk",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

Any OpenAI client library dey behave the same way: set base_url to https://gateway.example.com/v1 and api_key to the virtual key.

The routing policy from config.yaml don apply now without caller knowing. Request for bulk go the cheap model. If that call fail after its retries, dem go retry the request against strong. If prompt too long for bulk, context_window_fallbacks go send am to strong instead of returning error. Bulk work like classification pass or summarising backlog go run cheap by default, and na only hard requests go cost more.

Na here too gateway show why e useful with tool-using agents. An MCP (model context protocol) server for the same VPS and the agent wey dey control am fit both point to one endpoint, so the model behind dem fit change without redeploying either one.

How you go know say fallback happen?

Na this failure mode dey cost money, because nothing look broken. Successful fallback dey return HTTP 200 with ordinary response body. Your cheap model fit down for one day, and the expensive one go quietly serve every call. The first evidence fit be the invoice.

The evidence dey response headers. Ask for dem:

curl -s -D - -o /dev/null http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<the virtual key>" \
  -H 'Content-Type: application/json' \
  -d '{"model":"bulk","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^x-litellm'

• x-litellm-model-group na wetin the client ask for. x-litellm-model-id na the deployment wey answer. When the two no match, fallback don happen.
• x-litellm-attempted-fallbacks and x-litellm-attempted-retries dey count dem. For healthy call, both go be 0.
• x-litellm-response-cost na the cost of that one call for US dollars.
• x-litellm-call-id na the identifier wey you go use find the same call for your logs.

Record x-litellm-attempted-fallbacks for every request, and set alert when e stop to be 0. That one number na the difference between routing policy wey dey work and routing policy wey don quietly change to "always use the expensive model".

The complete version of this na tracing, and e need im own setup: self-hosted Langfuse for tracing agent calls. LiteLLM ships the callback, so wiring am na two lines plus credentials.

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://langfuse.example.com

Set failure_callback together with success_callback. If you skip am, na only traces where nothing go wrong you go keep. Separate from all this, LiteLLM dey write one spend row per request into Postgres, and Admin UI for /ui dey read that table. The table go grow as traffic increase, so monitor am if disk small.

Put the gateway behind a reverse proxy

Make sure nothing outside the box fit reach port 4000. Terminate TLS for nginx or Caddy, then forward traffic to the loopback address.

location / {
    proxy_pass http://127.0.0.1:4000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_read_timeout 600s;
}

Na two of those lines people dey usually leave out. proxy_buffering off matter because streaming completion na series of server-sent events. When buffering dey on, nginx go hold the chunks until response finish. The client go remain silent, then receive everything at once. proxy_read_timeout 600s matter because long generation fit pass nginx 60 second default. When that happen, client go receive 504, while error log go record upstream timed out (110: Connection timed out) while reading response header from upstream.

For the certificate, Certbot with Let's Encrypt for nginx na the short path. If the box already dey serve several containers, Traefik in front of multiple Compose apps go handle routing and certificates for one place.

The gateway don turn single point of failure

Make we talk true about wetin you don build. Every application wey you get now depend on one container for one VPS. While e dey down, nothing fit call any model, including providers wey dey perfectly healthy. Four things follow from this.

• Bad config fit bring everything down at once. restart: unless-stopped dey restart crash, and e go keep restarting container wey no fit parse config.yaml. Read docker compose logs litellm after every config change, and make config changes when you get time to monitor dem.
• Postgres dey inside the request path. Virtual key lookup and spend recording both use am. If /health/readiness return 503, na warning say gateway dey run but e no fit do either one.
• Scale by adding instances, no be by making one bigger. The project own guidance na one worker per instance (--num_workers 1), with several instances sharing one database. Two small gateways behind load balancer remove the single container from the setup. But dem no remove the database.
• Back up wetin you no fit regenerate. Na config.yaml and .env, together with pg_dump of the database. If you lose LITELLM_SALT_KEY, the encrypted provider credentials inside that dump go become useless. So the env file and the dump suppose dey inside the same backup job: restic backups to off-box storage.

To upgrade, edit the image tag and run docker compose up -d. LiteLLM dey run prisma migrate deploy for startup by default, so the new container go migrate the database schema for e first boot. Take the dump before you change the tag, because putting the old image back no go undo migration wey don already run.

FAQ

LiteLLM dey add latency wey noticeable to every call?

The project publish 8 ms for the 95th percentile at 1000 requests per second, as e dey stated for the README for August 2026. Treat am as vendor figure. The thing wey actually dey change your latency na the network distance between your applications and the gateway, because you don add one round trip to every call. Run the gateway for the same region with the applications wey dey call am, then measure your own overhead with the x-litellm-overhead-duration-ms header for real response.

Why streaming stop to work after I put nginx for front?

Because nginx dey buffer upstream responses by default, and streaming completion na series of server-sent events. When proxy_buffering dey on, nginx dey collect the chunks and release dem only when the response finish. So the client go wait without any response, then receive the whole answer at once. Set proxy_buffering off; for the location block. Increase proxy_read_timeout for the same block, because long generation fit otherwise pass nginx default of 60 seconds and the client go receive 504.

Wetin happen when virtual key finish its budget?

The call fail with HTTP 401 and body wey get form of ExceededBudget: Current spend for token: 7.2e-05; Max Budget for Token: 2e-07. The 401 na the trap: client library fit report am as authentication failure, so people go start check whether the key valid instead of reading the message. Log the response body together with the status code. Confirm the key real position with /key/info?key=sk-... against the master key, and increase the limit with /key/update if the budget too low.

Gateway fit route to local model as well as hosted ones?

Yes, and na one more entry for model_list. Use the ollama_chat/ prefix with an api_base, for example model: ollama_chat/llama3.1 together with api_base: http://ollama:11434. From inside container, localhost mean that container, so use the Compose service name or the host address for the Docker network, never 127.0.0.1. Setting up the local model na separate job: see self-hosting an LLM with Ollama on a VPS.