Build a RAG pipeline on your own VPS

What a self-hosted RAG pipeline looks like

A RAG pipeline (retrieval augmented generation) has five stages: chunk the documents, embed the chunks, store the vectors, retrieve the closest ones for a question, and send those chunks to a language model that writes the answer. On a VPS you already rent, the first four run on the box. PostgreSQL with the pgvector extension holds the vectors, and a small embedding model served by Ollama turns text into them. Only the last stage has to leave the box.

That split is the argument of this guide. Chunking is plain CPU work. Embedding is a 137 million parameter model that sits in a few hundred megabytes of RAM. Storage is a Postgres table whose size you can work out with arithmetic before you write a single row. For a corpus in the hundreds of thousands of chunks, all of that runs on an ordinary VPS. Generation is different, because it costs something on every question, forever.

Which parts of a RAG pipeline actually cost money

DigitalOcean's end to end RAG tutorial reaches for a managed vector database and a hosted embedding model, and its cost section is qualitative: cache repeated queries, keep the number of retrieved chunks small, rerank before generation. That advice is correct. It also skips the option that changes the arithmetic, which is running the embedding model on the server you are already paying for.

Count tokens rather than dollars, because token counts do not go stale when a price list changes. Take a corpus of 100,000 chunks at 400 tokens each, 10,000 questions asked of it, 8 chunks sent to the model per answer, a 100 token question and instruction block, and 400 token answers.

[
  {
    "label": "Embed the corpus (once)",
    "tokens_millions": 40,
    "tokens_per_question": "4,000"
  },
  {
    "label": "Embed each question",
    "tokens_millions": 0.2,
    "tokens_per_question": "20"
  },
  {
    "label": "Generation input",
    "tokens_millions": 33,
    "tokens_per_question": "3,300"
  },
  {
    "label": "Generation output",
    "tokens_millions": 4,
    "tokens_per_question": "400"
  }
]

Embedding the whole corpus is 40 million tokens, and it happens once. Spread over those 10,000 questions that is 4,000 tokens per question. Ask a hundred thousand questions and it falls to 400. Generation never falls. It costs 3,300 tokens in and 400 tokens out on every question you will ever answer.

So the money follows the stage that repeats. Own the embedding step, because you pay for it once and the VPS is running anyway. Buy the generation step, because that is where a better model is worth real money. Caching matters for the same reason: a cache hit skips the only stage whose cost never amortises. The difference between a KV cache and a prompt cache decides which half of that you can reuse, and a RAG prompt has a stable instruction block followed by a changing chunk block, which is the shape that benefits most.

Chunking: why fixed size with overlap is the right default

A chunk is the unit you retrieve, so its size decides everything downstream. It has to be small enough that its embedding is about one thing, because an embedding is a single point in space, so a chunk covering four topics lands between them and is close to none of them. It has to be large enough to answer a question by itself, because the language model sees the chunk and not the document around it.

Start at 300 words with 50 words of overlap. English runs at roughly 1.3 tokens per word, so 300 words is about 400 tokens. The overlap exists because a sentence that falls on a boundary is otherwise split in half, and neither half answers the question.

Split on structure first where the documents have structure. Break on headings, then on paragraphs, and apply the fixed size rule only inside a section that is still too long. A chunk that starts in the middle of a sentence reads badly in the final answer, because the model quotes back what you handed it.

Do not tune chunking before you can measure it. Fixed size with overlap is deterministic and cheap to run again, which makes it a baseline you can beat. Build the scoring query further down first, then change one thing at a time.

Embedding on the same box, and what it costs in RAM and latency

curl -fsSL https://ollama.com/install.sh | sh
ollama pull nomic-embed-text

nomic-embed-text is 137 million parameters and a 274 MB download as of August 2026. Check what it returns before you design a table around it.

curl -s http://127.0.0.1:11434/api/embed \
  -d '{"model": "nomic-embed-text", "input": "search_document: hello"}' |
  python3 -c 'import json,sys; print(len(json.load(sys.stdin)["embeddings"][0]))'

That prints 768. Your column type has to match that number exactly.

Two settings on this model catch people out.

The task prefix is not optional. Nomic's model card says the input "must include a task instruction prefix". Documents are embedded with search_document: in front, questions with search_query: . Leave them out and nothing fails: you get vectors back, retrieval quality drops, and no log line anywhere tells you why.

Long input is truncated quietly. The /api/embed endpoint takes a truncate field and it defaults to true, and the model as packaged by Ollama advertises a 2K context. A chunk longer than that is cut at the limit and embedded anyway, so its tail is unsearchable. Send "truncate": false while you are testing, so an oversized chunk fails instead of passing.

Batch the requests, and keep the model resident.

curl -s http://127.0.0.1:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["search_document: first chunk", "search_document: second chunk"],
  "keep_alive": "30m"
}' > /dev/null

input accepts a list, and one request carrying 32 chunks beats 32 requests, because the HTTP round trip and the model lookup happen once instead of 32 times. keep_alive controls how long the model stays in memory after a request, and the default is 5 minutes. When it expires, the next request pays the load time again.

Measure the two numbers that matter on your own box. They depend on your vCPU count, so no published figure will match.

ollama ps
time curl -s http://127.0.0.1:11434/api/embed \
  -d '{"model":"nomic-embed-text","input":"search_document: ... one real chunk ..."}' > /dev/null

ollama ps prints the resident size of the loaded model, which is RAM you have committed for as long as keep_alive holds it. The time output divided by the batch size is your seconds per chunk. Multiply by the chunk count to get the one time indexing cost. On a CPU only plan, expect a 100,000 chunk corpus to take hours rather than minutes. That is fine, because it happens once and it can run overnight under nice -n 19. If hours is not fine, the real question is whether renting a GPU pays for itself, and that is a break even calculation against API tokens rather than a preference.

If the box already serves a chat model, the embedding model is a second resident model and the RAM adds up. Running Ollama on a VPS covers sizing the generation side, and what a self-hosted model does under concurrent users covers what happens when several people ask at once. The embedding model is small enough to sit beside either.

python3 -m venv ~/rag
~/rag/bin/pip install "psycopg[binary]" pgvector

import json, urllib.request
import psycopg
from pgvector.psycopg import register_vector
from pgvector import Vector

OLLAMA = "http://127.0.0.1:11434/api/embed"
MODEL = "nomic-embed-text"

def embed(texts, prefix="search_document: "):
    payload = {"model": MODEL,
               "input": [prefix + t for t in texts],
               "truncate": False,
               "keep_alive": "30m"}
    req = urllib.request.Request(OLLAMA, data=json.dumps(payload).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["embeddings"]

def split(text, size=300, overlap=50):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

with psycopg.connect("dbname=rag user=rag") as conn:
    register_vector(conn)
    for doc_id, text in documents():          # your loader
        pieces = split(text)
        for start in range(0, len(pieces), 32):
            batch = pieces[start:start + 32]
            vectors = embed(batch)
            with conn.cursor() as cur:
                cur.executemany(
                    "INSERT INTO chunks (doc_id, seq, body, embedding)"
                    " VALUES (%s, %s, %s, %s)",
                    [(doc_id, start + i, body, Vector(vec))
                     for i, (body, vec) in enumerate(zip(batch, vectors))])
        conn.commit()

Storage: the pgvector schema, and how big it gets

Ubuntu 24.04 ships postgresql-16-pgvector at version 0.6.0, which is older than the halfvec type. Use the PostgreSQL project's own repository for a current build.

sudo apt update && sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-17 postgresql-17-pgvector

The number in the package name has to match your server's major version. Then create the role, the database and the extension.

sudo -u postgres createuser --pwprompt rag
sudo -u postgres createdb --owner rag rag
sudo -u postgres psql -d rag -c 'CREATE EXTENSION vector;'

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  doc_id    text NOT NULL,
  seq       int  NOT NULL,
  body      text NOT NULL,
  embedding vector(768) NOT NULL,
  fts       tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED
);

CREATE INDEX chunks_fts ON chunks USING gin (fts);

vector(768) has to match the model's output. Insert a 1024 dimension vector into that column and Postgres refuses it with expected 768 dimensions, not 1024, which is the clearest error message in this whole pipeline. The generated fts column costs nothing to maintain and buys you keyword search later.

The storage is arithmetic. pgvector documents a vector as 4 * dimensions + 8 bytes and a halfvec as 2 * dimensions + 8. The dimension counts below are each model's published output size.

[
  {
    "label": "384 (all-minilm)",
    "bytes_per_vector": "1,544",
    "vector_mib_per_100k": 147,
    "halfvec_mib_per_100k": 74
  },
  {
    "label": "768 (nomic-embed-text)",
    "bytes_per_vector": "3,080",
    "vector_mib_per_100k": 294,
    "halfvec_mib_per_100k": 147
  },
  {
    "label": "1024 (mxbai-embed-large)",
    "bytes_per_vector": "4,104",
    "vector_mib_per_100k": 391,
    "halfvec_mib_per_100k": 196
  },
  {
    "label": "1536 (hosted API model)",
    "bytes_per_vector": "6,152",
    "vector_mib_per_100k": 587,
    "halfvec_mib_per_100k": 294
  }
]

At 768 dimensions each vector is 3,080 bytes, so 100,000 chunks hold 294 MiB of vector data. The same corpus embedded by a 1536 dimension hosted model needs 587 MiB, and the index over it grows in proportion. Half precision halves both: halfvec(768) stores that corpus in 147 MiB. Whether it costs you any recall is a question the scoring query below answers in one run.

Those figures cover the vector column alone. Text, row overhead and indexes sit on top, so measure the real table.

SELECT pg_size_pretty(pg_total_relation_size('chunks')) AS total,
       pg_size_pretty(pg_relation_size('chunks'))       AS heap,
       count(*) AS n_rows
FROM chunks;

If you would rather have the same extension with an API and user accounts around it, a self-hosted Supabase stack is Postgres with pgvector already enabled, and every query in this guide works there unchanged.

Indexing: the HNSW settings that matter

Below a few thousand rows, skip the index. Exact search reads every row, it is fast enough at that size, and its recall is perfect. Add the index when the sequential scan stops being fast enough, and understand the trade: an approximate index returns approximately the right neighbours.

SET maintenance_work_mem = '2GB';
SET max_parallel_maintenance_workers = 3;
CREATE INDEX chunks_embedding ON chunks
  USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);

m = 16 and ef_construction = 64 are pgvector's defaults. Raising them improves recall and costs build time and index size. Use vector_cosine_ops with the <=> operator unless you know your model emits unit length vectors, because cosine distance ignores vector length while inner product does not.

Watch the build. When the graph outgrows maintenance_work_mem, pgvector says so:

NOTICE:  hnsw graph no longer fits into maintenance_work_mem after 100000 tuples
DETAIL:  Building will take significantly more time.

That is not an error and the build still finishes, but it drops onto a much slower path. Raise maintenance_work_mem in the session that builds the index and leave the server default alone, because that setting is per maintenance operation and a high global value is a way to run the box out of memory. Follow a long build from a second session.

SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%"
FROM pg_stat_progress_create_index;

Then compare the finished index against the memory in the box.

SELECT pg_size_pretty(pg_relation_size('chunks_embedding'));
SHOW shared_buffers;

An HNSW search walks a graph, so it touches pages scattered across the index instead of reading a range. An index that does not fit in memory turns every query into disk reads, and the slow tail is what users notice. That is the one sizing rule for the server: the index plus the rows you actually serve should fit in RAM. free -m and the size above are the two numbers to compare.

At query time, hnsw.ef_search is the recall dial and it defaults to 40.

BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT id, body FROM chunks ORDER BY embedding <=> $1 LIMIT 8;
COMMIT;

A higher value searches more of the graph, finds better neighbours and costs latency. It is a session setting, so you can raise it for one query without touching the index.

If a query is not using the index at all, the plan shows it.

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM chunks ORDER BY embedding <=> $1 LIMIT 8;

A sequential scan here often comes down to storage. A 768 dimension vector is 3,080 bytes, which is more than Postgres keeps inline, so the value moves to the TOAST table (the out of line store for oversized values). pgvector's own note is that the planner does not count out of line storage in its cost estimates, which can make a serial scan look cheaper than it is. ALTER TABLE chunks ALTER COLUMN embedding SET STORAGE PLAIN; keeps vectors inline. It applies to rows written after the change, so existing rows need a table rewrite.

Retrieval: one query, two signals

Vector search finds text that means the same thing as the question. It is weak on exact strings: a part number, an error code, a surname. Keyword search is the opposite, and Postgres already does it. Combine them in one query instead of running a second system.

Reciprocal rank fusion is the simplest combiner that works. Every result takes 1 / (60 + rank) from each list it appears in, and the two scores add. It needs no score normalisation, because it reads positions rather than distances.

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY distance) AS rank
  FROM (SELECT id, embedding <=> $1 AS distance
        FROM chunks ORDER BY embedding <=> $1 LIMIT 40) s
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY score DESC) AS rank
  FROM (SELECT c.id, ts_rank_cd(c.fts, q) AS score
        FROM chunks c, websearch_to_tsquery('english', $2) q
        WHERE c.fts @@ q
        ORDER BY score DESC LIMIT 40) k
)
SELECT c.id, c.body,
       coalesce(1.0 / (60 + s.rank), 0) + coalesce(1.0 / (60 + k.rank), 0) AS rrf
FROM (SELECT id FROM semantic UNION SELECT id FROM keyword) u
JOIN chunks c ON c.id = u.id
LEFT JOIN semantic s ON s.id = u.id
LEFT JOIN keyword  k ON k.id = u.id
ORDER BY rrf DESC
LIMIT 8;

$1 is the question's embedding from the same model, built with the search_query: prefix. $2 is the question as text. Both are bound from your application. websearch_to_tsquery accepts a real user question without choking on punctuation, which to_tsquery does not. One more thing to know: adding a WHERE filter on top of an HNSW scan can return fewer rows than you asked for, because the index is searched first and the filter runs after. SET hnsw.iterative_scan = relaxed_order; makes pgvector keep scanning until it has enough rows.

How do you tell whether retrieval is any good?

This is the step nearly every RAG guide skips, and it is the only one that tells you whether the other choices helped. It does not need an evaluation framework. It needs 30 questions and the id of the chunk that answers each one.

Write them by hand. Take questions people really ask about this corpus, run each one, read what comes back, and record the id of the chunk that should have won. Thirty questions will not resolve small differences. It will catch the differences that matter, because those are large.

CREATE TABLE gold (
  id        bigserial PRIMARY KEY,
  question  text   NOT NULL,
  chunk_id  bigint NOT NULL REFERENCES chunks(id),
  embedding vector(768) NOT NULL
);

Embed each question with the search_query: prefix, store it, then score the whole set in one query.

WITH hits AS (
  SELECT g.id,
         min(r.rank) FILTER (WHERE r.id = g.chunk_id) AS hit_rank
  FROM gold g
  CROSS JOIN LATERAL (
    SELECT top.id, row_number() OVER (ORDER BY top.distance) AS rank
    FROM (SELECT c.id, c.embedding <=> g.embedding AS distance
          FROM chunks c
          ORDER BY c.embedding <=> g.embedding
          LIMIT 10) top
  ) r
  GROUP BY g.id
)
SELECT count(*)         AS questions,
       count(hit_rank)  AS found_in_top_10,
       round(avg(coalesce(1.0 / hit_rank, 0)), 3) AS mrr
FROM hits;

found_in_top_10 divided by questions is recall at 10: how often the answer was inside the window you send to the model. MRR (mean reciprocal rank) averages 1 divided by the position of the right chunk and counts a miss as zero, so it rewards ranking the answer first rather than eighth. Both numbers move when you change the chunk size, swap the embedding model or add keyword search, and now you can see which way they moved.

Protect recall at 10 above everything else, because the generator cannot use a chunk it never received. When recall at 10 is 0.9 and the answers are still wrong, the fault is in the prompt or in the model, not in retrieval. That one split saves days of guessing.

Check the index separately. Approximate search costs recall, and pgvector shows you how much: run the same query with exact search and compare the ids.

BEGIN;
SET LOCAL enable_indexscan = off; -- use exact search
SELECT id FROM chunks ORDER BY embedding <=> $1 LIMIT 10;
COMMIT;

Nine ids of ten in common means ef_search is fine. Four of ten means raise it.

Reranking and generation: where an API earns its money

A reranker is a different kind of model. It reads the question and one chunk together and scores that pair, which beats comparing two embeddings computed independently, and it is far too slow to run across a whole corpus. That is exactly why it belongs here. It sees the 40 candidates that retrieval returned, not the 100,000 chunks in the table, so a hosted reranking API charges for 40 short pairs per question and drops the worst false positives before they reach the expensive stage.

Generation is the recurring bill, and two levers move it. Send fewer chunks, using recall at 10 to find out how few you can send without losing answers. Keep the front of the prompt stable byte for byte so a provider's prompt cache can hit it, and put the retrieved chunks after that stable part. Cache finished answers by question as well, because the cheapest generated token is the one you generated last week.

Sizing the box, and when this stops being enough

Every sizing rule here is something you measure rather than estimate.

• RAM is the binding constraint: the resident model size from ollama ps, plus the HNSW index size, plus shared_buffers, with headroom left for connections and the page cache.
• Disk wants twice pg_total_relation_size('chunks'), because rebuilding an index holds both copies at once.
• CPU decides reindex time, at your measured seconds per chunk multiplied by the chunk count.
• Reindexing happens more often than you expect, because changing the embedding model invalidates every vector already stored.

This design stops being enough at a point you can see coming. When the HNSW index no longer fits in the RAM you can buy, query latency turns into disk seeks and no setting recovers it. When one table serves many tenants and every query filters by tenant, partitioning the table becomes the fix, and that is real work. When indexing writes and user queries fight over the same box, move the embedding worker to a second server before you move the database. Until one of those happens, Postgres with pgvector on the VPS you already rent is a production answer, and the numbers above tell you how far away the limit sits.

FAQ

Can I run a RAG pipeline on one VPS, or do I need a vector database?

One VPS is enough for corpora in the hundreds of thousands of chunks. At 768 dimensions, 100,000 chunks is 294 MiB of vector data plus the text and the HNSW index, which fits in the RAM of an ordinary plan. The limit is memory rather than row count, because an HNSW search jumps around the index, so latency degrades once the index stops fitting in RAM. Compare pg_relation_size on the index with free -m and you will know where you stand.

Do I need a GPU to embed my documents?

Not if you embed once and query afterwards. A 137 million parameter model such as nomic-embed-text runs on CPU, and a full pass over a large corpus takes hours you can spend overnight. A GPU starts to matter when documents arrive continuously, or when you want to run generation on the same box. Time one batch against /api/embed on your own server and multiply by your chunk count, because vCPU counts differ too much for a published figure to help.

Why does my vector query use a sequential scan instead of the HNSW index?

Read the plan with EXPLAIN (ANALYZE, BUFFERS). The common cause is storage: pgvector notes that the planner does not count out of line storage in its cost estimates, which makes a serial scan look cheaper than it is, and a 768 dimension vector is 3,080 bytes so it lives in the TOAST table by default. ALTER TABLE chunks ALTER COLUMN embedding SET STORAGE PLAIN; keeps new rows inline. The other two causes are an operator that does not match the index, since an index built with vector_cosine_ops is only used by <=>, and a query with no ORDER BY ... LIMIT, since an approximate index only serves ordered nearest neighbour queries.

How do I know whether my retrieval is any good?

Build a gold set of 30 questions, each paired with the id of the chunk that answers it, and store the question embeddings alongside them. Then measure recall at 10, which is how often the right chunk appears in the top 10, and MRR, which rewards ranking it first. Those two numbers are what tell you whether a change to chunk size, embedding model or rank fusion helped. Without them you are changing settings and trusting your impression of a handful of answers.